Least squares solution to overdetermined $AX=B$ where each matrix is a rotation

I can't seem to find anything that would help me with this particular problem. I have a bunch of measurements of a matrix $$A$$ and corresponding matrix $$B$$, which I know are related by a third rotation $$X$$ ($$3\times 3$$ matrices). How do I arrive at a least squares solution to $$X$$ where $$X$$ is constrained to be a rotation?

I thought maybe SVD could be employed somehow, but the approach is not obvious to me.

• When you say in your title that each matrix is a rotation, is that just a typo, or do you mean that, say, $A$ and $B$ are semi-orthogonal with $X$ a proper rotation matrix? – Cade Reinberger May 23 at 23:25
• If they are all measurements of the same matrices $A$ and $B$, why wouldn't you compute the means $\bar A$ and $\bar B$ and then solve $X = \bar A^{-1}\bar B$? If you have measurements of different matrices $A_1X\approx B_1,A_2X\approx B_2,\dots$, see orthogonal Procrustes problem. In particular, you want to minimize $\|X^T [A_1^T, A_2^T, \dots] - [B_1^T, B_2^T, \dots]\|_F$. – Rahul May 24 at 5:15