Linear transformation of 3D point to origin. I am trying to figure out how to solve for $\delta a,\delta b $. Where M is a 3D point that I am trying to move to the origin.
$$R=\begin{bmatrix}
\cos(a) & 0 & \sin(a) \\
\sin(b)\sin(a) & \cos(b) & -\sin(b)\cos(a) \\
-\sin(a)\cos(b) & \sin(b) & \cos(b)\cos(a) \\\end{bmatrix}$$
$$M=\begin{bmatrix}M_x & M_y& M_z \end{bmatrix}^T$$
$$M = R(\delta a,\delta b).\begin{array}{ccc} [0 & 0 & ||M||] \end{array} ^T$$
When solving for $\delta a,\delta b $ the answer is supposed to be:
$$ \delta a = -\arctan(M_x/M_z)\\
\delta b = -\arctan(M_y/\sqrt{M_x^2+M_z^2}) $$
When I try solving I get:
$$ M_x = -\sin( \delta a) \cdot \sqrt{M_x^2+M_y^2+M_z^2}\\
M_y = -\sin( \delta b) \cdot \cos(\delta a)\cdot\sqrt{M_x^2+M_y^2+M_z^2}\\
M_z = \cos( \delta b) \cdot \cos(\delta a)\cdot\sqrt{M_x^2+M_y^2+M_z^2}\\$$
If I try to solve furthe the equation get messy and not like the answer. I think I am making a mistake with my product of my matrix and vector.
 A: Your equations for $M_x,M_y,M_z$ are not correct according to the given answers, as seen by the equations for $\delta a,\delta b$ follow.
First of all, let $r:=\Vert M\Vert =\sqrt{M_x^2+M_y^2+M_z^2}$. Then
$$\begin{array}{} 
M_x = -r\sin(\delta a)\\
M_y = -r\sin(\delta b)\cos(\delta a)\\
M_z = r\cos(\delta b)\cos(\delta a)\\
\end{array}$$
From this, we get the following properties:
$$\frac{M_y}{M_z}=\frac{-r\sin(\delta b)\cos(\delta a)}{
r\cos(\delta b)\cos(\delta a)} = -\frac{\sin(\delta b)}{\cos(\delta b)} = -\tan(\delta b)$$
$$M_y^2+M_z^2 = \left(-r\sin(\delta b)\cos(\delta a)\right)^2+\left(r\cos(\delta b)\cos(\delta a)\right)^2=r^2\cos^2(\delta a)\left(\sin^2(\delta b)+\cos^2(\delta b)\right) = r^2\cos^2(\delta a)$$
$$\frac{M_x}{\sqrt{M_y^2+M_z^2}}=\frac{-r\sin(\delta a)}{\sqrt{r^2\cos^2(\delta a)}}=\frac{-\sin(\delta a)}{\cos(\delta a)} = -\tan(\delta a)$$
From these, we see that your equations give the answers
$$
\delta a=-\arctan\left(\frac{M_x}{\sqrt{M_y^2+M_z^2}}\right) \\
\delta b=-\arctan\left(\frac{M_y}{M_z}\right)$$
This is a similar form to your given answers, but the elements are all mixed up.

Working backwards from the given solution, we can deduce what the equations for $M_x,M_y,M_z$ are supposed to be, assuming they have a similar form to the expressions you found. From there we can try to find out what went wrong in your calculations.
$$\frac{M_x}{M_z} = -\tan(\delta a)=-\frac{\sin(\delta a)}{\cos(\delta a)} \\
\Rightarrow M_x = -k\sin(\delta a),\quad M_z=k\cos(\delta a)$$
$$\frac{M_y}{\sqrt{M_x^2+M_z^2}}=\frac{M_y}{k}=-\tan(\delta b)=-\frac{\sin(\delta b)}{\cos(\delta b)} \\
\Rightarrow M_y = -p\sin(\delta b),\quad k=p\cos(\delta b)$$
$$r = \sqrt{M_x^2+M_y^2+M_z^2}=\sqrt{(-p\cos(\delta b)\sin(\delta a))^2 + (-p\sin(\delta b))^2+(p\cos(\delta b)\cos(\delta a))^2} = p$$
So finally we get
$$\begin{array}{} 
M_x = -r\cos(\delta b)\sin(\delta a)\\
M_y = -r\sin(\delta b)\\
M_z = r\cos(\delta b)\cos(\delta a)\\
\end{array}$$
And it appears that your equations were wrong in that $M_x$ and $M_y$ were switched, as were $\delta a$ and $\delta b$.

Now, I am not entirely sure from the text of your question what you are trying to do involving the rotation matrix and vector. My assumption is that you are trying to find the rotation matrix $R$ (parametrized by $\delta a$ and $\delta b$) that moves an arbitrary point $M$ to the positive $z$-axis by rotating about the origin. (This transformed point is $M'=[0\quad0\quad r]^T$.)
To do this, it appears that you may have used the equation $M=RM'$ to obtain your equations. The equations you listed are equivalent to this matrix equation except for the sign of the "lone sine" equation for $M_x$. However, this matrix equation actually represents the transformation from $M'$ to $M$, the inverse operation of what you are looking for.
The equation you want to solve is $M' = RM$. Using some properties of rotation matricies,
$$RM = M'\\
M = R^{-1}M'\\
M = R^TM'$$
and we get a different set of element equations:
$$\left[\begin{matrix}M_x\\ M_y\\ M_z\end{matrix}\right]=\left[\begin{matrix}\cos(a) & \sin(b)\sin(a) & -\sin(a)\cos(b) \\
0 & \cos(b) & \sin(b) \\
\sin(a) & -\sin(b)\cos(a) & \cos(b)\cos(a) \end{matrix}\right]\left[\begin{matrix}0\\ 0\\ r\end{matrix}\right]$$
$$\Rightarrow \begin{align}&M_x = -r\sin(a)\cos(b)\\
&M_y = r\sin(b)\\&M_z = r\cos(b)\cos(a) \end{align}$$
This is much closer to the equations implied by the given answers. In fact, the only difference is the sign on the "lone sine", as was the case in your multiplication. This leads me to suspect one of the following has occurred:

*

*The rotation matrix was improperly transcribed, with all entries with $\sin(\delta b)$ given the wrong sign.

*The signs of $\delta a$ and/or $\delta b$ are used inconsistently, possibly with some transformation (other than unity) between them and $a$ and $b$ as seen in the matrix.

*The sign of the answer given for $\delta b$ is improperly transcribed.

Adjusting the rotation matrix or $\delta b$ formula as I have described should give you the transformation you are looking for.

Side note:
Using $\arctan\left(\frac{y}{x}\right)$ for finding angles (about which no other information is known) is a bit dubious, especially when one or more of $x,y$ may be zero. I recommend using ${\rm atan2}(y,x)$, the four-quadrant arctan, in all arctangent calculations.
