How to solve quadratic matrix equations of the form $A^T B A=C$? I want to solve the following matrix equation
$$A^T  
\begin{pmatrix}
 10 & 0 & 0 \\
 0 & 20 & 0 \\
 0 & 0 & 25 \\
\end{pmatrix} A =
\begin{pmatrix}
 \frac{35}{2} & \frac{5 \sqrt{3}}{2} & 0 \\
 \frac{5 \sqrt{3}}{2} & \frac{25}{2} & 0 \\
 0 & 0 & 25 \\
\end{pmatrix}$$
In the above formula, $A^T$ is the transpose of matrix A.
At present, I don't have a good way. I only know that the reference answer of matrix A is 
$$\left(\begin{array}{ccc}
 \frac{1}{2} & -\frac{\sqrt{3}}{2} & 0 \\
 \frac{\sqrt{3}}{2} & \frac{1}{2} & 0 \\
 0 & 0 & 1 \end{array}\right)$$
 A: Call the matrix $$C'=\begin{bmatrix}\frac{35}{2}&\frac{5\sqrt 3}{2}\\\frac{5\sqrt 3}{2}&\frac{25}{2}\end{bmatrix}=\begin{bmatrix}a&b\\b&d\end{bmatrix}$$ Apply the 'diagonalization of a symmetric matrix by congruence' formula (which comes from 'completing the square' in the corresponding quadratic form). If $$E=\begin{bmatrix}1&-\frac{b}{a}\\0&1\end{bmatrix}$$ then $$E^TC'E=\begin{bmatrix}a&0\\0&\frac{\det C'}{a}\end{bmatrix}.$$ Thus $$E=\begin{bmatrix}1&\frac{-\sqrt 3}{7}\\0&1\end{bmatrix}$$
and $$E^TC'E=\begin{bmatrix}\frac{35}{2}&0\\0&\frac{80}{7}\end{bmatrix}.$$ Let
 $$F=\begin{bmatrix}\frac{2}{\sqrt 7}&0\\0&\frac{\sqrt 7}{2}\end{bmatrix}$$ Then 
$$F^TE^TC'EF=\begin{bmatrix}10&0\\0&20\end{bmatrix}.$$ Let G be the 3 x 3 matrix whose upper left 2 x 2 block is $EF$ and whose element in row 3 and column 3 is 1 and whose other elements are 0. Then $$G^TCG=\begin{bmatrix}10&0&0\\0&20&0\\0&0&25\end{bmatrix}.$$ Thus $A=G^{-1}$ 
A: The matrix $C$ is real symmetric so that you can diagonalize it as
$$C=P^TDP$$ where $P$ is orthogonal (hence its transpose is its inverse).
If $D$ matches the given $B$, you have the solution. If it does not, there is no solution, as the Eigenvalues are uniquely defined.
A: In our case we have two given symmetric matrices $B,C$,
$$
\begin{aligned}
B 
&= 
\begin{bmatrix}
10 &&\\&20&\\&&25
\end{bmatrix}=B_1^2 =B_1^TB_1=B_1B_1^T\ ,\text{ where }
\\
B_1
&= 
\begin{bmatrix}
\sqrt{10} &&\\&2\sqrt 5&\\&&5
\end{bmatrix}=B_1^\ ,
\\[3mm]
C &=
\begin{bmatrix}
 \frac{35}{2} & \frac{5 \sqrt{3}}{2} & 0 \\
 \frac{5 \sqrt{3}}{2} & \frac{25}{2} & 0 \\
 0 & 0 & 25 
\end{bmatrix}=C_1^2=C_1^TC_1=C_1C_1^T\ ,\text{ where }
\\
C_1
&= 
\begin{bmatrix}
a & b&\\b&c&\\&&5
\end{bmatrix}=C_1^T\ ,\text{ where its entries $a,b,c$ are}
\\
a &= \frac{\sqrt{10}}4(3\sqrt 2+1)\ ,\\
b &= \frac{\sqrt{30}}4(\sqrt 2-1)\ ,\\
c &= \frac{\sqrt{10}}4(\sqrt 2+3)\ .
\end{aligned}
$$
We  have to find one or all "unknown" matrices $A$, so that the matrix equation holds:
$$A^TBA=C\ .
$$
Equivalently, after successive transformations:
$$
\begin{aligned}
A^TBA &=C\ ,\\
A^TB_1^TB_1A &=C_1^TC_1\ ,\\
(C_1^{-1})^TA^TB_1^TB_1A(C_1^{-1}) &=I\ ,\\
(B_1AC_1^{-1})^T\;(B_1AC_1^{-1}) &=I\ ,\\
B_1AC_1^{-1}&\in SO(3)\ .
\end{aligned}
$$
Here, $SO(3)$ is the special orthogonal group of all matrices $X$ (with real entries, well, i suppose that that "complex" in the title is subjectively a synonym for "difficult"), such that $X^TX=I=XX^T=$. For each such $X$ we get a solution $A$ by isolating $A$ in $B_1AC_1^{-1}=X$. (So there are infinitely many solutions.) To have one solution, take the obvious $X=I$ as a particular solution.

Note: The diagonalization (if possible or not) is a process to put a linear transformation in a suitable form by using a base change. In our case, it is a quadratic form to be transformed by a base change. This is different.
https://en.wikipedia.org/wiki/Quadratic_form#Equivalence_of_forms
Note that the first object, the linear map given by a matrix $A$, transforms via a base change by $S$ as 
$$A\to S^{-1}AS\ ,$$
the second object, the quadratic form given by a symmetric $A$, transforms as $$A\to S^TAS\ .$$
