# Congruence and diagonalizations

How does one find matrix $M\in M_3(\mathbb R)$ such that $M^TAM=I$ where $$A=\left(\begin{array}{rrr}2&0&0\\ 0&3&-1\\ 0&-1&3\end{array}\right)$$ and simultaneously, $M^TBM$ is some diagonal matrix where $$B=\left(\begin{array}{rrr}1&3&-3\\ 3&3&1\\ -3&1&3\end{array}\right)$$?

Thanks.

-

As noted in my answer to your related question, there is a theorem regarding simultaneous diagonalization by congruence: if $A,B$ are real symmetric and $A$ is nonsingular, then they are simultaneously congruent to diagonal matrices if and only if $C=A^{−1}B$ is diagonalizable by similarity transform (Horn and Johnson, Matrix Analysis, Theorem 4.5.15.).
When all eigenvalues of $C$ are distinct, the matrix for diagonalizing $C$ via similarity transform is by itself the matrix for simultaneous diagonalizing $A$ and $B$ by congruence. Unfortunately, in your problem, $C$ has repeated eigenvalues and hence some additional manual work is needed. Specifically, we first try to diagonalize $C$ and obtain $$C = A^{-1}B = \begin{pmatrix} 1/2&3/2&-3/2\\ 3/4&5/4&3/4\\ -3/4&3/4&5/4 \end{pmatrix} =M_1\ D\ M_1^{-1},$$ where $$M_1 = \begin{pmatrix} 2&1&0\\ -1&1&1\\ 1&0&1 \end{pmatrix},\quad D=\textrm{diag}(-1,2,2).$$ So, our first candidate for $M$ is $M_1$. However, observe that $D$ has repeated eigenvalues. So there is a chance that $M_1^\top AM_1$ or $M_1^\top BM_1$ are not diagonal. Indeed, $$M_1^\top AM_1 = \begin{pmatrix} 16&0&0\\ 0&5&2\\ 0&2&4 \end{pmatrix}, \quad M_1^\top BM_1 = \begin{pmatrix} -16&0&0\\ 0&10&4\\ 0&4&8 \end{pmatrix}.$$ So we need to further diagonalize $\begin{pmatrix}5&2\\2&4\end{pmatrix}$ and $\begin{pmatrix}10&4\\4&8\end{pmatrix}$ simultaneously via congruence. Since the two matrices are multiples of each other and they are symmetric, it suffices to find a matrix $M_2$ that orthogonally diagonalizes either one of them. It is not difficult to work out such a matrix: for $$\lambda=\frac{9+\sqrt{17}}{2},\quad M_2=\frac{1}{\sqrt{\lambda+4}}\begin{pmatrix}\lambda-4&-2\\2&\lambda-4\end{pmatrix},$$ we have $$\begin{pmatrix}5&2\\2&4\end{pmatrix} = M_2\begin{pmatrix}\lambda&0\\0&9-\lambda\end{pmatrix}M_2^\top.$$ Now we may set $$M = M_1\begin{pmatrix}1\\&M_2\end{pmatrix}.$$
For this particular problem, since $A$ is positive definite, there is a simpler solution -- find the square root of $A$, then orthogonally diagonalize $(\sqrt{A}^\top)^{-1}B\sqrt{A}^{-1}$. In other words, if $\sqrt{A}$ is a matrix such that $\sqrt{A}^\top\sqrt{A}=A$, and $Q$ is an orthogonal matrix such that $Q^\top(\sqrt{A}^\top)^{-1}B\sqrt{A}^{-1}Q$ is diagonal, then you may set $M=\sqrt{A}^{-1}Q$. In this case you will get $M^\top AM=I$.