I have two points $(x_1,y_1)$ and $(x_2,y_2)$. Using dot product I have calculated the angle between the two. Let's call this angle $A$. Now, I want to rotate $(x_2,y2)$ around $(x1,y1)$ such that resulting angle between the $(x_1,y_1)$ and $(x'_2, y'_2)$ becomes $5^\circ$. So, I continue like below. Find angle of rotation which will $A' =(A -5)$. So new points will be
$$x'_2 = \cos(A')(x_2 - x_1) - \sin(A') (y_2 - y_1) + x_1$$ $$y'_2 = \sin(A')(x_2 - x_1) + \cos(A') (y_2 - y_1) + y_1$$
Is this correct?
if it's correct then the problem is that if I recalculate the angle using dot product between $(x_1,y_1)$ and $(x'_2,y'_2)$ then the resulting angle is not $5^\circ$. How can his be. My understanding of rotation and resulting new angle is wrong?