Rotation matrix from plane A to B I have two planes defined by two orthogonal vectors. Plane A is defined by vectors $A_x, A_y$ and B, $B_x, B_y$. I want to find the rotation matrix that would transform a point which has coordinates of $(x, y, z)$ to coordinates $(x', y', z')$ where original point have $(x'', y'')$ as components on Plane A and the transformed point would have $(x'', y'')$ on Plane B.
 A: In 3 dimensions, there are infinitely many linear transformations that mat one plane onto another. We want to find one such transformation, defined by the matrix R, with the following properties:
$$R\mathbf{a}_x=\mathbf{b}_x\ ,\ R\mathbf{a}_y=\mathbf{b}_y$$
We can arbitrarily chose a third vector so that each set is a complete, linearly independent basis:
$$R\mathbf{a}_z=\mathbf{b}_z$$ 
where $\mathbf{a}_z$ and $\mathbf{b}_z$ are not contained within their respective planes. Let A be a matrix whose columns are $\mathbf{a}_x$, $\mathbf{a}_y$, and $\mathbf{a}_z$, and likewise with B. Then we have:
$$RA=B$$
Right multiplying by the inverse of A gives:
$$R=BA^{-1}$$
We can check to see if this gives the desired result:
$$(x,y,z)=x''\mathbf{a}_x+y''\mathbf{a}_y$$
$$(x',y',z')=R(x,y,z)=Rx''\mathbf{a}_x+Ry''\mathbf{a}_y$$
$$=x''R\mathbf{a}_x+y''R\mathbf{a}_y$$
$$=x''\mathbf{b}_x+y''\mathbf{b}_y$$
The choice of $\mathbf{a}_z$ and $\mathbf{b}_z$ does not affect what happens to points on the plane. It only determines how the "off-plane" component is transformed.
If R is to be a rotation matrix, A and B need to be orthogonal matrices, or some matrix multiple thereof. The conditions $||\mathbf{a}_x||=||\mathbf{b}_x||\ ,\ ||\mathbf{b}_y||=||\mathbf{a}_y||$, and $\mathbf{a}_x\cdot\mathbf{a}_y=\mathbf{b}_x\cdot\mathbf{b}_y$ must be met in order for this to be the case. Choosing $\mathbf{a}_z=\mathbf{a}_y\times\mathbf{a}_y$ and $\mathbf{b}_z=\mathbf{b}_y\times\mathbf{b}_y$ will then give the rotation matrix.
To see why this is, we need to use the property that the dot product is invariant under rotation. That is, for any vectors $\mathbf{u}$ and $\mathbf{v}$, $\mathbf{u}\cdot\mathbf{v}=(R\mathbf{u})\cdot(R\mathbf{v})$. From this we get 3 equalities:
$$\mathbf{a}_x\cdot\mathbf{a}_y=(R\mathbf{a}_x)\cdot(R\mathbf{a}_y)=\mathbf{b}_x\cdot\mathbf{b}_y$$
$$||\mathbf{a}_x||^2=\mathbf{a}_x\cdot\mathbf{a}_x=(R\mathbf{a}_x)\cdot(R\mathbf{a}_x)=\mathbf{b}_x\cdot\mathbf{b}_x=||\mathbf{b}_x||^2$$
$$||\mathbf{a}_y||^2=\mathbf{a}_y\cdot\mathbf{a}_y=(R\mathbf{a}_y)\cdot(R\mathbf{a}_y)=\mathbf{b}_y\cdot\mathbf{b}_y=||\mathbf{b}_y||^2$$
When choosing $\mathbf{a}_z$ and $\mathbf{b}_z$, we we will have 3 mode sets of dot products that need to be equal. It can be shown that $||\mathbf{a}_x\times\mathbf{a}_y||=||\mathbf{b}_x\times\mathbf{b}_y||$, and all of the other dot products are conveniently $0$.
Hope this helps.
