solution of $B = XAX^{-1}$ in terms of $AX = BX$ system of linear equations I am looking for general solution of the form $B = XAX^{-1}$
with the following constraints
$X^{-1}=X'$
where $X$ is unknown matrix and $A,\,B,\, X$, are $3\times 3$ matrices.
The problem appears peculiar and thus I wish to simplify it further to more practical problem
X is rotation matrix ($R$), and $A$ is plane equation matrix, B is resulted homography in the sense of
$B =XAX^{-1} = XAX^{T}= R(I-tn^T/d)R^{T}$ ,
where $t$ is translation vector, $n$ is plane normal vector, and $d$ is signed distance, all three are known
My dirty solution so far is as follows :
Finding the Jordan Canonical form:
$MBM^{-1}=J=NAN^{-1}$,  
and then $X$ is obviously   
$X = N^{-1}M$
but this works only if the canonical forms are the same, otherwise cannot be utilized for solution.
The second dirty trick is to solve a system of Linear equations
in terms of 
$AX = BX$ 
but i cannot figure out how to program this in the software (e.g. Matlab linsolve).
Could you help me to clarify how to write system of linear equations
in the shape of 9 rows?
The trick is how to notate them on the "right side" to use in ready to use functions for solving SLE.
Thank, you
 A: Case 1. $A,B$ are not orthogonally similar; no solutions.
Case 2. $A,B$ are orthogonally similar; $A=PBP^T$, where $P\in O(n)$. Let $C(A)=\{X\in M_n(\mathbb{R});AX=XA\}$. Then, it is not difficult to show 
$\textbf{Proposition}$ $\{X\in O(n);A=XBX^T\}=(C(A)\cap O(n))P$.
It remains to obtain $C(A)\cap O(n)$. When $n=3$, $C(A)$ is a vector space of dimension $3,5$ or $9$ (when $A$ is a scalar matrix).
Generically (randomly choose $A$), $A$ is cyclic 
https://en.wikipedia.org/wiki/Cyclic_subspace
and $C(A)=\{aI+bA+cA^2;a,b,c\in \mathbb{R}\}$. Generically, the eigenspaces of $A$ are not orthogonal and, consequently, $C(A)\cap O(n)=\pm I_n$ and $X=\pm P$.
EDIT. $A,B$ are orthogonally similar iff the couples $(A,A^T)$ and $(B,B^T)$ are similar. Then, practically, one studies the linear system
$(S)$ $AX=XB,A^TX=XB^T$.
If $A,B$ are not orthog. similar, then the sole solution is $X=0$.
If $A,B$ are orthog. similar, then, for a generic $A$, $(S)$ admits a vector space of solutions of dimension $1$: $X=uX_0$. It remains to calculate $u$ so that $||uX_0||_2=1$.
