Conditions on matrix expression What conditions are needed on the matrices A and B so that the following is true:
$A^{T}BA=B$. Clearly, if B is invertible, the determinant of A must be 1 or -1. What other conditions are needed?
 A: This can be seen as a special case of the matrix equation $AXB = C$, which is equivalent to $(B^T \otimes A) \operatorname{vec} X = \operatorname{vec} C$, where $B^T \otimes A$ is a Kronecker product $\operatorname{vec} X$ is the vectorization of $X$ (the columns stacked on top of each other).
The equations $AXB = C$ and $(B^T \otimes A) \operatorname{vec} X = \operatorname{vec} C$ can be seen to be equivalent by looking at the columns of $AXB$. Take the $k$:th column, $(AXB)_k$, let $B$ and $X$ have columns $b_k$ and $x_k$ respectively:
$$(AXB)_k = AXb_k = A \sum_{i=1}^n x_i b_{i,k} = (A b_{1,k}, \dots, Ab_{n,k}) (x_1, \dots, x_n)^T$$
Now, vectorize this (put the columns of top of each other), and you get the result.
In this case we have $A^TBA = B$ and we get $(A^T \otimes A^T) \operatorname{vec}B = \operatorname{vec}B$, so $B$ has to be an eigenvector of $(A^T \otimes A^T) = (A \otimes A)^T$ with eigenvalue 1.
Now, all the eigenvalues of a Kronecker product are products of the eigenvalues of the constituent matrices. Thus, $A^T$ and therefore $A$ has to have at least two eigenvalues $\lambda, \mu$ such that $\lambda\mu = 1$ ($\lambda = \mu$ is ofc ok).
Let $(\lambda_i, \mu_i)$ for $i = 1, \dots, k$ be all the ordered pairs of eigenvalues to $A$ such that $\lambda_i \mu_i = 1$ and let $(x_i, y_i)$ be the corresponding eigenvectors to $A^T$. Then the following is true:
$$\operatorname{vec}B = \sum_{i=1}^k \alpha_i x_i \otimes y_i$$
and this gives all the $\operatorname{vec}B$ that satisfies the equation. We can also write this on matrix form, $x_i$ forming columns in a matrix and $y_i$ forming rows in another matrix:
$$B = \sum_{i=1}^k \alpha_i x_iy_i^T = (x_1, x_2, \dots, x_k)\operatorname{diag}(\alpha_1, \dots, \alpha_n) (y_1, y_2, \dots, y_k)^T$$
We can here see that $B$ at most can have the same rank as the number of pairs of eigenvalues $(\lambda_i, \mu_i)$.
$$\operatorname{rank} B \leq \operatorname{rank} A$$
To guarantee a solution, the existence of an eigenvalue $\lambda$ such that $\lambda^2 = 1$ is enough. For $B$ to be invertible, $A$ has to have eigenvalues whose absolute values are all 1, so the absolute value of the determinant has to be 1 (converse to the result stated in the question!).
The idea seems to be also pretty easy to verify in the original equation $A^TBA = B$.
It is tempting to compare this with the SVD of $B$, but there is a couple of things in the way. $\alpha_i$ can be taken negatively. In some cases the minus sign can be but in one of the vectors instead, but since it's possible that an eigenvector can repeat, this might prove problematic. Second, the eigenvector-sets $\{x_i\}$ and $\{y_i\}$ are not necessarily orthogonal sets, which they need to be.
