Given the quadratic form $$Q(x) = \alpha\alpha_1\alpha_2 + 2\alpha^2\alpha_1\alpha_3$$ on $\mathbb{R}^2$ where $x = (\alpha_1,\alpha_2,\alpha_3)$ in some basis I want to find the signature of $Q$ dependent on $\alpha$ and the diagonal basis of $Q$.

I was trying to rewrite $Q$ into a sum of squares like this: $$Q(x) = (\alpha\alpha_1 + \alpha\alpha_3 + \frac{1}{2}\alpha_2)^2 - \alpha^2\alpha_1^2 - \alpha^2\alpha_3^2 - \frac{1}{4}\alpha_2^2 - \alpha\alpha_3\alpha_2$$ which yields $$(\alpha\alpha_1 + \alpha\alpha_3 + \frac{1}{2}\alpha_2)^2 - (\alpha\alpha_1)^2 - (\alpha\alpha_3 + \frac{1}{2}\alpha_2)^2 = Q(x)$$

Since now I have $Q$ represented as a sum of squares, I see the signature being $(0,1,2)$ if $\alpha \neq 0$ and $(1,1,1)$ if $\alpha = 0$. Also the terms in parentheses must be the coordinates of $x$ in the diagonal basis of $Q$: $$\beta_1 = \alpha\alpha_1 + \alpha\alpha_3 + \frac{1}{2}\alpha_2, \beta_2 = \alpha\alpha_1, \beta_3 = \alpha\alpha_3 + \frac{1}{2}\alpha_2$$

which corresponds to a matrix \begin{pmatrix}\alpha & \frac{1}{2} & \alpha\\ \alpha & 0 & 0\\ 0 & \frac{1}{2} & \alpha \end{pmatrix}

To get the diagonal basis, I need to invert this matrix (get alphas represented by betas), but it is clearly singular. What am I doing wrong?

-

$$Q(x) = \alpha \alpha_1 \alpha_2 + 2 \alpha^2 \alpha_1 \alpha_3$$

be a quadratic form on $\mathbb{R}^3$ in some basis. We know what the signature is $(3,0,0)$ if $\alpha = 0$, so let us assume that $\alpha \neq 0$.

Something went wrong in your diagonalization process because you managed to diagonalize to a non-singular quadratic form. The quadratic form

$$(\alpha\alpha_1 + \alpha\alpha_3 + \frac{1}{2}\alpha_2)^2 - (\alpha\alpha_1)^2 - (\alpha\alpha_3 + \frac{1}{2}\alpha_2)^2$$

you got has signature $(0,1,2)$. You may want to try expanding and then comparing the result with $Q(x)$.

The Gram matrix

$$\begin{pmatrix} 0 & \frac{\alpha}{2} & \alpha^2 \\ \frac{\alpha}{2} & 0 & 0 \\ \alpha^2 & 0 & 0 \end{pmatrix}$$

of $Q(x)$ is singular because its second and third rows are linearly dependent. Let us first consider the upper left $2 \times 2$ submatrix which corresponds to the form $\alpha \alpha_1 \alpha_2$. This form represents all real numbers, so it must have signature $(0,1,1)$. Thus the first two entries of the diagonalized matrix must be $1$ and $-1$. Since the original Gram matrix is singular, the final element on the diagonal must be $0$. Putting everything together, we see that the signature of $Q(x)$ is $(1,1,1)$.

If we carry out the diagonalization by hand, we will find that indeed

$$\begin{pmatrix} 1 & 1 & 0 \\ \frac{1}{a} & -\frac{1}{a} & -2 \\ 0 & 0 & \frac{1}{a} \end{pmatrix}^{\top} \begin{pmatrix} 0 & \frac{\alpha}{2} & \alpha^2 \\ \frac{\alpha}{2} & 0 & 0 \\ \alpha^2 & 0 & 0 \end{pmatrix} \begin{pmatrix} 1 & 1 & 0 \\ \frac{1}{a} & -\frac{1}{a} & -2 \\ 0 & 0 & \frac{1}{a} \end{pmatrix} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 0 \end{pmatrix}$$

The change-of-basis matrix is non-singular.

-