quadratic reduced forms I have a question about a proof of a theorem of reduced quadratic forms. Let me explain it!
I'm going to denote the form $f=aX^2+bXY+cY^2$ as $f=(a,b,c)$. The theorem I want to prove says:
Each form $f=(a,b,c)$ is equivalent to a form $f'=(a',b',c')$ such that $|b'|\leq |a'|\leq |c'|$. The form $f'$ is said reduced form of $f$.
I follow the proof till one critic moment. I'll show you the proof and I'll show you my problem.
Proof.
We take $a'\neq 0$ the smallest integer number represented by $f=(a,b,c)$, it means: $\forall m\in Z, m\neq 0$ such that $f\rightarrow m$, we have that $|m|\geq |a'|$.
We take $\alpha, \gamma\in Z$ such that $f(\alpha,\gamma)=a\alpha^2+b\alpha\gamma+c\gamma^2=a'$. We have that $a'$ is the smallest integer number represented by $f$, so $m.c.d(\alpha,\gamma)=1$. So that, we can choose $\beta,\delta\in Z$ such that $\alpha\delta-\beta\gamma=1$.
Let's $P=\begin{bmatrix}{\alpha}&{\beta}\\{\gamma}&{\delta}\end{bmatrix}$. We have that $P\in SL(2,Z)$; so the matrix $P^T \begin{bmatrix}{2a}&{b}\\{b}&{2c}\end{bmatrix} P$ is the matrix of a form $f''=a'X^2+b''XY+c''Y^2$ which is equivalent to $f$. 
Let's $Q=\begin{bmatrix}{1}&{-n}\\{0}&{1}\end{bmatrix}$. We transform $f''$ by the matrix $Q$ and we obtain $f'=a'X^2+(b''-2a'n)XY+(a'n-b''+c'')Y^2=a'X^"+b'XY+c'Y^2.$ (My problem and doubt come from here, although I'm goint to explain it after!!!).
We observe that $f'$ has $a'$ as first coefficient. We are going to show that we can choose $n\in Z$ such that 
$-|a'|\leq b''-2a'n\leq |a'|$, which is the same that $|b'|\leq |a'|$.
Doing the division, there exists uniques $q,r\in Z$ such that $b''=2|a'|q+r, 0\leq r\leq 2|a'|$.
We pick $sgn(a)=\frac{|a|}{a}=\left \{ \begin{matrix} 1 & \mbox{if } a>0
\\ -1 & \mbox{if }a<0\end{matrix}\right. $.
If $0\leq r\leq |a'|$ we define $n$ by the formula $|a'|q=a'n$, i.e., $n=\frac{|a'|}{a}q=sgn(a)q$. From here we obtain that $b'=r\leq |a'|$.
If $r>|a'|$ we define $b''=2|a'|(q+1)+(r-2|a'|)$. We have that, $-|a'|=|a'|-2|a'|<r-2|a'|<0<|a'|$, so we define $n$ by the formula $|a'|(q+1)=a'n$, i.e. $n=sgn(a')(q+1)$. So $r-2|a'|=b''-2a'n$, which implies that $|b'|\leq |a'|$.
Now, by the choice of $a'$ and because $f'\rightarrow c''$ we have that $|a'|\leq |c''|$. So that, $f\rightarrow c''$. Selected $n$, it's enough to coose $c'=c''$ (WHY? IT'S MY DOUBT!!!), $b'=b''-2a'n$.
I've been searching in diferent books but I've just found the proof for positive forms, not general as in here. Thank you for your help!
 A: Your error is in the transformation. Given form $\langle r,s,t \rangle$ which refers to $$ f(x,y) = r x^2 + s x y + t y^2,$$ the Hessian matrix is
$$ 
H = 
\left(
\begin{array}{cc}
2r & s \\
s & 2 t
\end{array}
\right)
$$
You may see the term Gram matrix, some authors call $H$ the Gram matrix, some call $H/2$ the Gram matrix. Gram was a real person. So was Hesse. The author who won the Nobel Prize in Literature was different.
Next you take
$$ 
Q = 
\left(
\begin{array}{cc}
1 & -n \\
0 & 1
\end{array}
\right)
$$
This should have given
$$ 
Q^T H Q = 
\left(
\begin{array}{cc}
2r & s - 2 r n \\
s - 2 r n & 2 r n^2 - 2 s n + 2 t
\end{array}
\right)
$$
which means the quadratic form with coefficient triple
$$\langle r, \; \; s - 2 r n, \; \; r n^2 - s n + t \rangle$$ 
You already chose $r$ so that $r$ has the smallest absolute value when $(x,y) \neq (0,0).$ 
Important Caution: if the discriminant $\Delta = s^2 - 4 r t = w^2$ for some integer $w,$ this becomes false, as there is then an integer representation of $0.$
When $\Delta$ is not a square, $|r| \neq 0$ is minimal. We may choose $n$ so that $|s - 2 n r| \leq |r|.$ Finally,
$$ f(-n,1) = r n^2 - s n + t $$ is a value of the form. We already chose $r$ as the minimum possible, so
 $$ |f(-n,1)| = |r n^2 - s n + t| \geq |r|. $$ 
Let's see: for indefinite forms, this is not what we call reduced. The traditional Gauss-Lagrange reduced forms $\langle a,b,c \rangle$ are those with $ac < 0$ and $b > |a+c|.$ Zagier introduced a variant in his 1981 book on Zeta functions, $\langle a,b,c \rangle$ is Zagier reduced if $a > 0,$ $c > 0,$ $b > a + c.$ In both cases, we are requiring $\Delta = b^2 - 4 a c > 0$ but not a square.
