The argument of the trigonometric functions in your formulas is not the absolute angle of a point but the relative angle by which the point $(x',y')$ is rotated with respect to the point $(x,y)$. Thus you don't need to know where the angle zero is in absolute terms; all you need to know is whether a positive angle represents a clockwise or a counterclockwise rotation. This you can find out e.g. by rotating the point $(1,0)$ by $\pi/2$ (or $90^\circ$); the result is $(0,1)$, so the rotation is counterclockwise.
If you do ever need to know where the angle $0$ is in absolute terms, it's usually taken to be along the positive $x$ axis, with positive angles between $0$ and $\pi/2$ representing points in the first quadrant, i.e. with positive $x$ and $y$ coordinates.
By the way, I disagree with your claim that "in programming, zero degrees is at point $A$". I think Java counts as programming, and it uses the mathematical convention, in which zero degrees corresponds to $B$; see e.g. the API for java.awt.Graphics.drawArc.