Solve a linear mapping from an equation with two square matrices Given two square matrices A and B with sizes $n\times n$ and $m\times m$, I want to obtain the solution of a linear mapping with size $n\times m$ from the following equation:
$M^TAM=B$.
Particularly, if $A$ and $B$ are definite, I can use the Cholesky decomposition and get,
$M^TL_A^TL_AM=L_B^TL_B$
Then, we have $L_AM=L_B$ and obtain the solution of M as follows:
$M=L_A^{-1}L_B$
More generally, I wonder if $M$ is solvable when $A$ and $B$ are two general square matrices.
 A: In addition to the conditions mentioned in the question, suppose that $A,B$ are symmetric. Let $n_0,n_+,n_-$ denote indices of inertia.
A solution $M$ will exist if and only if $n_+(B) \leq n_+(A)$ and $n_-(B) \leq n_-(A)$.
Suppose that $s = n_+(B) \leq n_+(A) = p$ and $t = n_-(B) \leq n_-(A) = q$. By Sylvester's law of inertia, there exist invertible matrices $P,Q$ such that
$$
P^TAP = \pmatrix{I_p & 0 & 0\\0 & -I_q & 0\\0 & 0 & 0}, \quad Q^TBQ = \pmatrix{I_s & 0 & 0\\0 & -I_t & 0\\0 & 0 & 0}.
$$
There exists a $(p + q)\times (s + t)$ submatrix $J$ of the size-$(p+q)$ identity matrix such that
$$
J^T \pmatrix{I_p & 0\\0 & -I_q} J = \pmatrix{I_s & 0\\0 & -I_t}.
$$
By padding $J$ with zeros, we can produce a matrix $K$ such that
$$
K^T[P^TAP]K = Q^TBQ.
$$
It follows that
$$
B = Q^{-T}K^TP^T A PKQ^{-1} = [PKQ^{-1}]^T A [PKQ^{-1}],
$$
which means that we can take $M = PKQ^{-1}$.
Conversely, suppose that $M^TAM = B$. Let $U_+$ denote the maximal subspace of $\Bbb R^n$ such that for all non-zero $x \in U_+$, we have $x^TAx > 0$. Notably, $\dim(U_+) = n_+(A)$. Let $V_+$ denote the maximal subspace of $\Bbb R^m$ such that for all non-zero $y \in V_+$, we have $y^TBy > 0$.
For all $y$ in $V_+$, it holds that $(My)^TA(My) > 0$. Notably, this implies that
$M|_{V_+}$ (the restriction of $M$ to $V_+$) has a trivial kernel. We can see that the image of $M|_{V_+}$ is a subspace of $\Bbb R^n$ such that all non-zero elements $x$ of this subspace satisfy $x^TAx > 0$. However, by the definition of $U_+$, this means that the dimension of the image of $M|_{V_+}$ is less than or equal to $\dim(U_+) = n_+(A)$. Because $M|_{V_+}$ is injective, this means that $\dim(U_+) \geq \dim(V_+)$, which is to say that $n_+(B) \leq n_+(A)$.
By a similar argument (or by applying the same argument to $-A$ and $-B$), we find that $n_-(B) \leq n_-(A)$.
