How to find an orthogonal transformation between two matrices I wonder if there is a way to solve this problem:$$\arg \min_{\alpha, U} \|A - \alpha UB\|_\mathcal F^2 \\ \text{s.t. } \alpha \in \mathbb R, U \in \mathbb R^{n \times n} \text{ and } U^\top U = I$$
where $A,B$ are $n \times m$ matrices, $I$ is the $n \times n$ identity matrix.
 A: Note: I assume that $U$ is square.
Finding an optimal $U$ for a fixed $\alpha \in \Bbb R$ amounts to the orthogonal Procrustes problem. In particular, the minimum
$$
\operatorname{min}_{U \in \Bbb R^{n \times n}} \|A - U\alpha B\| \quad \text{s.t.} \quad U^\top U = I
$$
is given by $\sum_{i=1}^n [\sigma_i(\alpha BA^\top)-1]^2 = \sum_{i=1}^n [(|\alpha|\cdot \sigma_i(BA^\top))-1]^2$.
Interestingly, there are only two possible minimizers $U$. Let $B = V \Sigma W^\top$ be a singular value decomposition, and let $U_* = VW^\top$. $\alpha B = [\operatorname{sgn}(\alpha) V][|\alpha|\cdot \Sigma]W^\top$ is also a singular value decomposition. It follows that for a given $\alpha$, the minimizer is either $U_*$ in the case that $\alpha > 0$ or $U_-$ in the case that $\alpha<0$. Because the minimal distance for a given $\alpha$ depends only on $|\alpha|$, we can necessarily find a minimizing pair $\alpha, U$ with $\alpha > 0$, so that the minimizing $U$ is correspondingly $U_*$.
To find the minimizing (positive) value of $\alpha$, we need to compute the minimizer of
\begin{align}
f(\alpha) &= \sum_{i=1}^n [(\alpha\cdot \sigma_i(BA^\top))-1]^2 = 
\left(\sum_{i=1}^n \sigma_i^2(BA^\top)\right)\alpha^2 
- 2\left(\sum_{i=1}^n \sigma_i(BA^\top)\right)\alpha + n
\\ & = 
\|BA^\top\|_F^2\alpha^2 
- 2\left(\sum_{i=1}^n \sigma_i(BA^\top)\right)\alpha + n.
\end{align}
This is a quadratic function, so its minimizer is easily found as
$$
\alpha_* = \frac{\sum_{i=1}^n \sigma_i(BA^\top)}{\sum_{i=1}^n \sigma_i^2(BA^\top)}
= \frac{\sum_{i=1}^n \sigma_i(BA^\top)}{\|BA^\top\|_F^2}.
$$
The corresponding minimum is equal to
$$
f(\alpha_*) = n - \frac{\left(\sum_{i=1}^n \sigma_i(BA^\top)\right)^2}{\|BA^\top\|_F^2}
=
n - \frac{\left(\sum_{i=1}^n \sigma_i(BA^\top)\right)^2}{\sum_{i=1}^n \sigma_i^2(BA^\top)}.
$$
