Confusion about orthogonal basis of a quadratic form. Let $q(x)=x_1^2+4x_1x_2+6x_1x_3+3x_2^2+8x_2x_3+5x_3^2$
1)Write the matrix $A$ of $q$ in the standard basis of $\mathbb{R}^3$
2)Find the diagonal matrix $A'$ congruent to A.
3)Find the orthogonal basis of the previous quadratic form.
Solution
1)
$
A = \begin{pmatrix} 
1 & 2 & 3 \\
2 & 3 & 4 \\
3 & 4 & 5
\end{pmatrix}$ Since $A$ is symmetric.
2)
Since $a_{11}=1$ ( different than zero) , $q$ can be written as : 
$q(x)=\frac{1}{a_{11}}(x_1+2x_2+3x_3)^2+q'(x)$
Solving for $q'(x)$ We get :
$q'(x)=-x_2^2-4x_2x_3-4x_3^2=-(x_2+2x_3)^2$
Hence : $q(x)=(x_1+2x_2+3x_3)^2-(x_2+2x_3)^2$
Changing variables :
Let :
$x'_1=x_1+2x_2+3x_3$ 
$x'_2=x_2+2x_3$ 
$x'_3=x_3$ 
Thus : $
A' = \begin{pmatrix} 
1 & 0& 0\\
0 & -1 & 0\\
0& 0& 0
\end{pmatrix}$ 
3)
We need to find $x_1 , x_2 ,x_3$ function of $x'_1 , x'_2 , x'_3$
$x_1=x'_1-2x'_2+x'_3$
$x_2=x'_2-2x'_3$
$x_3=x'_3$
And $
P = \begin{pmatrix} 
1 & -2 & 1 \\
0 & 1 & -2 \\
0 & 0 & 1
\end{pmatrix}$
(We can verify that $^{t}PAP=A'$)
And the orthogonal Basis would be $B'=${$v_1=(1,0,0),v_2=(-2,1,0),v_3=(1,-2,1)$}
(Please point out any calculation mistakes)
However I'm not quite sure what this basis denotes , is this the basis in which the congruent matrix $A'$ of $A$ is diagonal ?
Shouldn't this basis contain only 2 vectors since the rank of $A$ is 2 ? 
Thanks in advance.
 A: Conceptually, the issue is that the same quadratic form can have different diagonalizations if we don't require the change of bases to be an orthogonal matrix (this is what often happens if you diagonalize by "completing the squares"; there is more than one way to do this, for example by starting not with $a_{11}$ but with $a_{22}$ or $a_{33}$; starting not with $a_{11}$ as you did  you get an upper triangular change of basis, which is not orthogonal (unless it's is diagonal, which will only happen if A itself is diagonal)). 
In order to find an orthonormal basis in which A becomes diagonal you need to find a basis of unit eigenvectors. Wolfram gives eigenvectors $(1/10 (-5 + \sqrt{105}), 1/20 (5 + \sqrt{105}) , 1)$, $(1/10 (-5 - \sqrt{105}) , 1/20 (5 - \sqrt{105}) , 1)$, and $(1,-2 ,1)$ with some rather unpleasant lengths, so the orthonormal basis is rather cumbersome. But depending on what one means, the three eigenvectors themselves can be considered to be an "orthogonal bases" (the form will look diagonal in this orthogonal basis).
