How to solve $AX-XB=0$ for $X$? Suppose $A, B, X$ are all $4\times4$ transform matrices, like
$$\begin{bmatrix}
R & T\\
0 & 1
\end{bmatrix}$$
where $R$ is a rotation matrix, i.e., orthogonal and $\det(\cdot)=1$, $T$ is a 3D translation vector.
Can the equation $AX-XB=0$ be solved for $X$ and is the solution unique?
 A: Furthermore, I find that my problem is very similar to the Hand/Eye Calibration in machine vision. So, I think I've found the solution. Thanks, guys. Hand_eye_calibration_problem
A: Let $A = \begin{bmatrix} R_a && T_a \\ 0 && 1 \end{bmatrix}$ ,$B = \begin{bmatrix} R_b && T_b\\ 0 && 1 \end{bmatrix}$, $X = \begin{bmatrix} R_x && T_x \\ 0 && 1 \end{bmatrix}$
then
$AX = \begin{bmatrix} R_a R_x && R_a T_x + T_a \\ 0 && 1 \end{bmatrix}$
and
$X B = \begin{bmatrix} R_x R_b && R_x T_b + T_x \\ 0 && 1 \end{bmatrix}$
Thus, we want
$ R_a R_x = R_x R_b \hspace{24pt} (1) $
and
$ R_a T_x + T_a  = R_x T_b + T_x \hspace{24pt} (2)$
Equation $(1)$ implies
$ R_b = (R_x)^{-1} R_a (R_x) = (R_x)^T R_a (R_x) \hspace{24pt}(3)$
Thus $R_a $ must be similar to $R_b$ , i.e. sharing the same eigenvalues.  For this to happen, the two rotation matrices $R_a$ and $R_b$ must be rotations by the same angle about possibly different axes.
If $R_a$ and $R_b$ satisfy this condition, then the matrix $R_x$ can be found, and it is not unique.
Next, equation $(2)$ implies
$ (R_a - I) T_x   = R_x T_b - T_a \hspace{24pt} (4)$
The matrix $(R_a - I)$ is singular and the eigenvector corresponding to its zero eigenvalue is the axis of rotation of $R_a$.  Thus, if $T_a$ and $T_b$ are zero, then $T_x$ can be any vector along the axis of $R_a$.  Otherwise, in the general case, it is only required that $(R_x T_b - T_a)$ be in the column space of $(R_a - I)$.  Again, the solution is not unique.
Final remark on how to obtain $R_x$.  Matrices $R_a$ and $R_b$ are rotation matrices about axes $a$ and $b$ by the same angle, then a unitary (rotation) matrix $U_a$, can be found such that
$ R_a = U_a R_z U_a^{-1} = U_a R_z U_a^T \hspace{24pt} (5) $
where $R_z = \begin{bmatrix} \cos \theta && -\sin \theta && 0 \\ \sin \theta && \cos \theta && 0 \\ 0 && 0 && 1 \end{bmatrix} $
Matrix $U_a$ is not unique, and can be any rotation matrix having as its third column the unit vector $a$.  Similarly,
$R_b =  U_b R_z U_b^{-1} = U_b R_z U_b^T \hspace{24pt} (6) $
Using $(5)$ , $R_z = U_a^T R_a U_a $, hence
$R_b = U_b U_a^T R_a U_a U_b^T \hspace{24pt} (7)$
Thus $R_x = U_a U_b^T $
