Using Lagrange's diagonalization on degenerate linear forms 
Let $A=\begin{pmatrix}1 & 2 & 3\\
2 & 3 & 4\\
3 & 4 & 5
\end{pmatrix}$ be a real matrix.
  Find an invertible matrix $P\in M_{3}(\mathbb{R})$ such that $P^TAP$ is diagonal whose elements on the diagonal are all in the set $\{-1, 0, 1\}$

I've been trying to solve this with Lagrange's squares method, but I'm stuck at the end where I seem to get "less" squares then required. I assume that this is because $A$ is degenerate, but I'm unable to complete my form to a basis that would satisfy the requirements as a result, and would love a general method for this.
These are my calculations so far:
Let $\begin{pmatrix}x_{1}\\
x_{2}\\
x_{3}
\end{pmatrix}\in\mathbb{R}^{3}$.
Then 
$$\begin{aligned}\begin{pmatrix}x_{1}\\
x_{2}\\
x_{3}
\end{pmatrix}^{T}\begin{pmatrix}1 & 2 & 3\\
2 & 3 & 4\\
3 & 4 & 5
\end{pmatrix}\begin{pmatrix}x_{1}\\
x_{2}\\
x_{3}
\end{pmatrix}= & x_{1}^{2}+4x_{1}x_{2}+6x_{1}x_{3}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\
= & x_{1}^{2}+2x_{1}\left(2x_{2}+3x_{3}\right)+\left(2x_{2}+3x_{3}\right)^{2}\\
 & -\left(2x_{2}+3x_{3}\right)^{2}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\
= & \left(x_{1}+\left(2x_{2}+3x_{3}\right)\right)^{2}-4x_{2}^{2}-12x_{2}x_{3}-9x_{3}^{2}+3x_{2}^{2}+5x_{3}^{2}+8x_{2}x_{3}\\
= & \left(x_{1}+2x_{2}+3x_{3}\right)^{2}-x_{2}^{2}-4x_{2}x_{3}-4x_{3}^{2}\\
= & \left(x_{1}+2x_{2}+3x_{3}\right)^{2}-\left(x_{2}+2x_{3}\right)^{2}
\end{aligned}
 $$
and you can see I have only two "elements" for the basis, and I'm stuck..
 A: Denote the quadratic form by
$$ q \left( \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} \right) = (x_1, x_2, x_3) \cdot A \cdot \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = (x_1 + 2x_2 + 3x_3)^2 - (x_2 + 2x_3)^2. $$
We see that if we make the change of variables
$$ u_1 = x_2 + 2x_2 + 3x_3, \,\, u_2 = x_2 + 2x_3, u_3 = r_{31} x_1 + r_{32} x_2 + r_{33} x_3 $$
the quadratic form will become diagonal. You can choose the scalars $r_{31}, r_{32}, r_{33}$ in any way you like as long as you get a legitimite change of variables - that is, the corresponding map is invertible.
More formally, consider the matrix $R$ given by
$$ R = \begin{pmatrix} 1 & 2 & 3 \\ 0 & 1 & 2 \\ r_{31} & r_{32} & r_{33} \end{pmatrix}, \,\, R \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix}. $$
If we choose $r_{31}, r_{32}, r_{33}$ so that $R$ will be invertible, then
$$ q \left( R^{-1} \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix} \right) = (u_1, u_2, u_3) \left( R^{-1} \right)^T A R^{-1} \begin{pmatrix} u_1 \\ u_2 \\ u_3 \end{pmatrix} = u_1^2 - u_2^2 $$
and so the $P$ you are looking for can be taken to be $R^{-1}$. The simplest choice (that will also make computing $R^{-1}$ easier) is to take $r_{31} = r_{32} = 0$ and $r_{33} = 1$.
A: This can be done entirely with matrices. As it happens, the diagonal matrix we find first has diagonal elements that are already $0, \pm 1.$ If that had not been the case, we could adjust with a final tweak matrix, diagonal with some square roots as elements. For the direction of your question, you just need the matrix I call $P.$ Sorry, I called the original matrix $M,$ so it comes out $P^T M P = D.$ Often people want $Q^T D Q = M,$ which is why I found $Q = P^{-1}.$ 
Anyway,
=================================
? m = [ 1,2,3; 2,3,4; 3,4,5]
%1 = 
[1 2 3]

[2 3 4]

[3 4 5]

? m - mattranspose(m)
%2 = 
[0 0 0]

[0 0 0]

[0 0 0]

? p1 = [ 1,-2,-3; 0,1,0; 0,0,1]
%3 = 
[1 -2 -3]

[0 1 0]

[0 0 1]

? m1 = mattranspose(p1) * m * p1 
%4 = 
[1 0 0]

[0 -1 -2]

[0 -2 -4]

? p2 = [ 1,0,0; 0,1,-2; 0,0,1]
%5 = 
[1 0 0]

[0 1 -2]

[0 0 1]

? d = mattranspose(p2) * m1 * p2 
%6 = 
[1 0 0]

[0 -1 0]

[0 0 0]

? p = p1 * p2
%7 = 
[1 -2 1]

[0 1 -2]

[0 0 1]

? q = matadjoint(p)
%8 = 
[1 2 3]

[0 1 2]

[0 0 1]

?  mattranspose(p) * m * p 
%9 = 
[1 0 0]

[0 -1 0]

[0 0 0]

?  mattranspose(q) * d * q 
%10 = 
[1 2 3]

[2 3 4]

[3 4 5]

===================================
I and others discuss this method, including several typeset examples, at
reference for linear algebra books that teach reverse Hermite method for symmetric matrices
Bilinear Form Diagonalisation 
Given a $4\times 4$ symmetric matrix, is there an efficient way to find its eigenvalues and diagonalize it? 
Find the transitional matrix that would transform this form to a diagonal form.
Writing an expression as a sum of squares
Determining matrix $A$ and $B$, rectangular matrix
Method of completing squares with 3 variables
