Linear least-squares with matrices rather than vectors I have two coordinate frames, each represented by a 4-by-4 matrix ($A$ and $B$), where this is the pose (orientation and translation) in homogeneous coordinates. I now want to find a third matrix $T$, which is the transformation between $A$ and $B$:
$$TA = B$$
To do this, I have taken several measurements of $A$ and $B$, and then want to solve $T$ with a least-squares approach. Now, I now how to solve the following equation using SVD linear least-squares:
$$Mx = y$$
But in the case I am familiar with, $M$ is a matrix, and $x$ and $y$ are vectors. However, in my case, both $A$ and $B$ are matrices, rather than vectors. So, how can I solve $TA = B$ using a similar linear least-squares solution? What is the equivalent of $x$, $y$ and $M$?
 A: The usual formulation of the vector problem is to minimize
$$
\sum_{i=1}^n\|Mx_i-y_i\|_2^2
$$
Now you need to instead minimize
$$
\sum_{i=1}^n\|TX_i-Y_i\|_F^2=\sum_{i=1}^n\mathrm{trace}((TX_i-Y_i)^\top(TX_i-Y_i))
$$
By expanding the sum in terms of the constant, linear and quadratic term you get the following
$$
\mathrm{trace}(T(\sum_{i=1}^nA_iA_i^\top)T^\top - 2T(\sum_{i=1}^nA_iB_i^\top) + \sum_{i=1}B_iB_i^\top)
$$
Now take the derivative with respect to T and you get
$$
2T(\sum_{i=1}^nA_iA_i^\top)-2(\sum_{i=1}^nA_iB_i^\top)^\top.
$$
Setting the derivative equal to zero and solving gives
$$
T = (\sum_{i=1}^nB_iA_i^\top)(\sum_{i=1}^nA_iA_i^\top)^{-1}.
$$
I'm not sure about the correspondence to the vector problem.
There is an additional difficulty in the case when $\sum_{i=1}^nA_iA_i^\top$ is not invertible.  In that case you need to replace the inverse with the pseudo inverse.
A: Let $x=\begin{pmatrix}x_{1}\\
x_{2}\\
\vdots\\
x_{n}
\end{pmatrix}$ and similarly $y$ be the stacked measurement vectors, actually block matrices as $x_{i},y_{i}\in M_{4}(\mathbb{R})$ are
square matrices. We look for the best approximation in a normed space
minimizing the map $\|xA-y\|^{2}$ with respect to $A\in M_{4}(\mathbb{R})$.
For that we find the map derivative, (or the gradient since we are
given an inner product), the following map linear in $H\in M_{4}(\mathbb{R})$,
$$\left.\frac{d}{dt}\right|_{0}\|x(A+tH)-y\|^{2}=\left.\frac{d}{dt}\right|_{0}\langle x(A+tH)-y,x(A+tH)-y\rangle\\=2\left\langle xA-y,xH\right\rangle \\=2\left\langle x^{t}xA-x^{t}y,H\right\rangle. $$
The gradient is then $2(x^{t}xA-x^{t}y)$. This must be the zero map
at the best approximation, hence $$A=(x^{t}x)^{-1}x^{t}y$$ which actually looks the same as the usual least squares formula. I think the inverse of $x^{t}x\in M_{4}(\mathbb{R})$ exists as long as at least
one $x_{i}$ is invertible.
