# How do I find a rotation that brings this equation of a cone to standard axes?

I'm really bad at this alternate coordinate stuff because I missed the last bit of classes and that's when we did change of bases. I know for a vector in standard basis $$\bar{x}$$ we can find a representation for the same thing in a base $$B$$, denoted $$[\bar{x}]_B$$ by finding $$\bar{C}[\bar{x}]_b = \bar{x}$$. So far practice problems have been simple manipulations of matrices, so I'm not sure how to approach this problem:
A conic in the xy coordinate system is $$5x^2 -2\sqrt{3}xy+7y^2 = 16$$. Find a rotation of coordinates that bring it to standard form. What is the matrix of this rotation? What is the cosine of the angle of the rotation? (this part I can probably do after getting started).
A general rotation matrix has the form $$R=\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix},$$ while the equation of the conic can be written as $$\begin{bmatrix}x&y\end{bmatrix} \begin{bmatrix}5&-\sqrt3\\-\sqrt3&7\end{bmatrix} \begin{bmatrix}x\\y\end{bmatrix}=17.$$ Substiuting $$(x,y)^T=R^{-1}(x',y')^T$$ into the left-hand side of this equation and using $$R^{-1}=R^T$$ gives $$\begin{bmatrix}x'&y'\end{bmatrix} R\begin{bmatrix}5&-\sqrt3\\-\sqrt3&7\end{bmatrix} R^T \begin{bmatrix}x'\\y'\end{bmatrix} = \begin{bmatrix}x'&y'\end{bmatrix} \begin{bmatrix}6-\cos2\theta+\sqrt3\sin2\theta & -\sqrt3\cos2\theta-\sin2\theta \\ -\sqrt3\cos2\theta-\sin2\theta & 6+\cos2\theta-\sqrt3\sin2\theta\end{bmatrix} \begin{bmatrix}x'\\y'\end{bmatrix}.$$ We want the $$x'y'$$ term to vanish, so we must have $$\sqrt3\cos2\theta+\sin2\theta=0.$$ Solve for $$\theta$$. Note that the rows of the resulting rotation matrix will be eigenvectors of the symmetric matrix associated with the conic, but there’s no need to go through the usual eigenvector/eigenvalue computation to solve this problem.
One straightforward way of doing this is to find a rotation matrix, $$\ U\$$, such that the matrix $$D=U^\top\pmatrix{5&-\sqrt{3}\\-\sqrt{3}&7}U$$ is diagonal. The equation of your conic can be written as $$\pmatrix{x&y} \pmatrix{5&-\sqrt{3}\\-\sqrt{3}&7} \pmatrix{x\\y}=16\ ,$$ so if you define new coordinates $$\ x', y'\$$ by $$\pmatrix{x',y'}=U^{-1} \pmatrix{x\\y}\ ,$$ —that is, with respect to axes subjected to the rotation determined by $$\ U\$$—then the equation in the new coordinates is \begin{align} 16&= \pmatrix{x'&y'}U^\top \pmatrix{5&-\sqrt{3}\\-\sqrt{3}&7} U\pmatrix{x'\\y'}\\ &= \pmatrix{x'&y'}D \pmatrix{x'\\y'}\\ &= d_1x'^2+d_2y'^2\ , \end{align} where $$\ d_1, d_2\$$ are the (diagonal) entries of $$\ D\$$.
The columns of $$\ U\$$ need to be chosen as normalised eigenvectors of the matrix $$\ \pmatrix{5&-\sqrt{3}\\-\sqrt{3}&7}\$$, and $$\ d_2, d_2\$$ will then be the corresponding eigenvalues.