Prove that $3 \times 3$ matrices under certain conditions form a group Let $S = \{M \in M_3(\mathbb{Z})\mid M^T \Omega M = \Omega \}$, where 
$$\Omega = \begin{bmatrix}
1 & 0 & 0\\
0 & 1 & 0 \\
0 & 0 & -1 
\end{bmatrix}$$
I'm trying to prove that $S$ is a group under matrix-multiplication. 
Proof:


*

*Show it is closed under multiplication


Let $M,N \in S$. This means $M^T \Omega M = \Omega$ and $N^T \Omega N = \Omega$. 
I have to show that $(MN)^T \Omega (MN) = \Omega$. This would be equivalent to $N^TM^T \Omega MN = \Omega$ which is the same as $N^T \Omega N = \Omega$.
So it is closed under matrix multiplication.
Now, I struggle to show that there exists an inverse. I will have to prove that all of these matrices in $S$ are invertible and that their inverse is in $S$ as well. 
Can you give me an idea on how to do this?
 A: If $M^{\top}\Omega M=\Omega$ then $M$ is invertible in $M_3(\Bbb{Z})$ because
$$\det(M^{\top}\Omega M)=\det(\Omega)=-1,$$
and hence $\det(M)=\pm1$. It follows that
$$M^{-\top}\Omega M^{-1}=M^{-\top}(M^{\top}\Omega M)M^{-1}=(M^{-\top}M^{\top})\Omega(MM^{-1})=\Omega,$$
so $M^{-1}\in S$. You may also want to mention that $I\in S$, even though this follows from the above.
A: Since $M^T\Omega M=\Omega$,$$\det(M^T\Omega M)=\det\Omega=-1.$$But$$\det(M^T\Omega M)=-\det\nolimits^2M.$$So, $\det^2M=1$ and therefore $\det M=\pm1$. So, $M$ is invertible in $M_n(\mathbb Z)$.
A: Meanwhile, from Magnus, given integers with $\alpha \delta - \beta \gamma = 1$ and $\alpha + \beta + \gamma + \delta$ even, we get 
$$
M = 
\left(
\begin{array}{ccc}
\frac{1}{2} \left( \alpha^2 - \beta^2 - \gamma^2 + \delta^2 \right) & \alpha \beta - \gamma \delta & \frac{1}{2} \left( \alpha^2 + \beta^2 - \gamma^2 - \delta^2 \right) \\
\alpha \gamma - \beta \delta &\alpha \delta + \beta \gamma  &\alpha \gamma + \beta \delta \\
\frac{1}{2} \left( \alpha^2 - \beta^2 + \gamma^2 - \delta^2 \right) & \alpha \beta + \gamma \delta & \frac{1}{2} \left( \alpha^2 + \beta^2 + \gamma^2 + \delta^2 \right) \\
\end{array}
\right)
$$
This gives all such $M$ with positive determinant. 
For example, $\alpha = 9, \beta = 4, \gamma = 2, \delta = 1,$ I get
$$
M = 
\left(
\begin{array}{ccc}
31 & 34 & 46  \\
14 &17&22 \\
 34& 38 & 51 \\
\end{array}
\right)
$$
If I then take column vector $v$ with elements $(3,4,5)$ which is a Pythagorean triple, I get a column vector with elements $459,220,509$
This should be a Pythagorean triple as well. $459^2 + 220^2 = 259081.$ Then $509^2 = 259081.$ Hooray!
ADDED: the indefinite form with a really, really nice integer automorphism group has Hessian matrix
$$
\left(
\begin{array}{ccc}
0 & 0 & -1  \\
0 &2& 0 \\
 -1& 0 & 0 \\
\end{array}
\right)
$$
A: Note that $M \in M_3(\mathbb{Z})\mid M^T \Omega M = \Omega \Longrightarrow \operatorname{det}(M)=\pm1$, and thence:
\begin{alignat}{1}
S =\{M \in \operatorname{SL}_3^\pm(\mathbb{Z})\mid M^T \Omega M = \Omega \} \\
\tag 1
\end{alignat}
In the form $(1)$, $S$ is strongly reminiscent of the stabilizer of $\Omega$ under the action of $\operatorname{SL}_3^\pm(\mathbb{Z})$ by $M^T(\cdot) M$ on some set $X\ni\Omega$, which would automatically make of $S$ a group.
Let's define:
$$ X:=\{\Gamma \in M_3(\mathbb{Z})\mid \operatorname{det}(\Gamma)=-1\}\tag 2$$
(and thence, $\Omega\in X$.) Therefore, if $M\in \operatorname{SL}_3^\pm(\mathbb{Z})$ and $\Gamma \in X$, then $\operatorname{det}(M^T\Gamma M)=-1$, whence $M^T\Gamma M \in X$. So, we are left to check that the map $X\times\operatorname{SL}_3^\pm(\mathbb{Z})\to X$, defined by $(\Gamma,M) \mapsto M^T\Gamma M$, is indeed a (right) action:


*

*$I\in \operatorname{SL}_3^\pm(\mathbb{Z})$ and $\Gamma\cdot I=I^T\Gamma I=\Gamma, \forall \Gamma\in X$;

*$(\Gamma\cdot N)\cdot M=M^T(N^T\Gamma N)M=(NM)^T\Gamma(NM)=\Gamma\cdot(NM), \forall M,N\in \operatorname{SL}_3^\pm(\mathbb{Z}), \forall\Gamma\in X$
So, $S=\operatorname{Stab}(\Omega)$ with respect to this action, and thence $S \le \operatorname{SL}_3^\pm(\mathbb{Z})$. In particular, $S$ is a group.
