Solution to $AX=XB$ for $3\times3$ rotation matrices. Given the $3\times3$ rotation matrices $A$ and $B$, find the rotation matrix $X$ that satisfies $$AX=XB.$$
 A: This equation commonly arises in the hand-eye calibration problem (robotics), where one want to find a transformation matrix between a camera mounted on the end-effector and the end-effector itself.


*

*In case no noise is presented in A and B, a minimum of two equations $A_i X=X B_i$ is required to determine a unique solution. Shiu and Ahmad 1989

*Unfortunately this is not true in practice. Instead, a more common approach is to find a "best-fit" solution from a set of equalities $A_i X=X B_i$.
There are many solutions has been proposed for this including [F. Park 1994-Robot Sensor Calibration: Solving AX = XB on the Euclidean Group], [K. Daniilidis-The dual quaternion approach to hand-eye calibration], [K. H. Strobl and G. Hirzinger. Optimal Hand-Eye Calibration], etc.

A: If $A,B$ are 3x3 real matrices representing rotations ("3D rotation matrices"), the following are equivalent:
(1) $tr(A) = tr(B)$
(2) There exists real orthogonal $X$ such that $A = XBX^{-1}$.
(3) There exists rotation matrix $X$ such that $A = XBX^{-1}$.
Since rotation matrices are those orthogonal matrices with determinant 1, it's evident that (3) implies (2), and by replacing $X$ by $-X$ if necessary, also (2) implies (3).
Since similar matrices have equal traces, (2) implies (1).  It remains only to show (1) implies (2), which we will do in a constructive fashion.
Setting aside the trivial case of zero rotation (identity matrix), every rotation in three dimensions is characterized by an axis of points fixed by the rotation and an angle by which points rotate around that axis.  In the case of a special orthogonal linear transformation (rotation represented by a 3x3 matrix), the axis passes through the origin and may be identified by a unit vector $u$ that generates it.  If we adopt a convention that positive angles of rotation are counterclockwise about the axis pointed in direction $u$, then changing the sign of the angle is equivalent to changing instead the vector $u$ into $-u$ (and keeping the same angle as before).
As the Wikipedia article points out, if rotation matrix $A$ has angle of rotation $\theta$ about an axis, then the trace of $A$ is $1 + 2 \cos \theta$.  For example, by our convention the matrix which represents counterclockwise rotation by $\theta$ about the positive x-axis is:
$$R_x(\theta) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & \cos \theta & -\sin \theta \\
0 & \sin \theta & \cos \theta \end{pmatrix} $$
and the trace is the sum of diagonal entries.
If $A$ is a rotation matrix with $tr(A) = 1 + 2 \cos \theta$, let $u$ be a unit vector that generates the axis of rotation, i.e. $Au = u$.  Extend that to an orthonormal basis $\{u,v,w\}$ for $\mathbb{R}^3$, and obtain an orthogonal matrix $P$ using these basis vectors as columns.  Then similar rotation matrix $P^{-1} A P$ is either $R_x(\theta)$ or $R_x(-\theta)$ in the notation above.  If the latter, replace $w$ by $-w$, so that:
$$R_x(\theta) = P^{-1} A P$$
Do the same with rotation matrix $B$ also having $tr(B)  = 1 + 2 \cos \theta$:
$$R_x(\theta) = Q^{-1} B Q$$
Now $X = P Q^{-1}$ satisfies (2), so proving that (1) implies (2) as desired.
A: Well, this is equivalent to $A = XBX^{-1}$, so it's only possible if $A$ and $B$ are equivalent.
If they are, you can solve this by diagonalizing in $\mathbb{C}$ and finding both base changes $T$, $S$. Then: $A = T^{-1}DT = T^{-1}SBS^{-1}T$ if $B = S^{-1}DS$. And $X = T^{-1}S$.
Edit: This only gives a unitary $X$. Maybe this can be fixed.
A: Genericaly, there is an infinity of solutions in $SO_3$. Let $A=Rot(\Delta,\alpha)$ with $\alpha\notin \pi\mathbb{Z}$; since $A=XBX^{-1}$, $A,B$ are othogonally similar, that is equivalent to say that $B$ can be written in the form $B=Rot(\Delta_1,\alpha)$. Let $P\in SO_3$ be fixed s.t. $B=P^TAP$.
Then $A(XP^T)=(XP^T)A$ and $XP^T\in SO_3$ commute with $A$, that is equivalent to $XP^T\in U=\{Rot(\Delta,\beta);\beta\in [0,2\pi)\}$. Finally, the set of solutions is $X\in UP$.
For $P$, it suffices to choose any rotation that sends $\Delta_1$ on $\Delta$; for example, the one with axis $u\wedge u_1$ where $u,u_1$ are unitary vectors on $\Delta,\Delta_1$.
