Positive definite binary quadratic forms Please help me to solve this question or introduce references that help me: 
Let $f(x,y) = ax^2 + bxy + cy^2$ be a reduced positive definite form. 
Suppose that $\gcd(x, y) = 1$ and that $f(x, y) ≤ a + |b| + c$. Show 
that $f(x,y)$ must be one of the numbers $a, c, a - |b| + c$ or $a + 
|b| + c$.
 A: I suppose the main thing is that
$$ f \geq \left( \frac{4ac - b^2}{4c} \right) x^2, $$
$$ f \geq \left( \frac{4ac - b^2}{4a} \right) y^2. $$
These just come from completing the square. 
Reduced means $$ |b| \leq a \leq c  $$ with some additional conditions in case some things are equal. For example, $ac \geq b^2.$ Thus $4ac-b^2 \geq 3ac.$ Thus
$$ f \geq \left( \frac{3a}{4} \right) x^2, $$
$$ f \geq \left( \frac{3c }{4} \right) y^2. $$
Now, suppose $|y|\geq 2,$ so that $y^2 \geq 4$ and $f \geq 3c.$ With the condition $f \leq a + |b| + c$ and reduction this gives $y = \pm 2,$ $a = |b| = c$ and so $a=|b| = c = 1$ because $\gcd(a,b,c) = 1.$ Take $y=2$ and continue with $x^2 + 2x + 4.$
Next, if we do not have $a=|b| = c = 1,$ we have $y = \pm 1$ or $y=0.$ Continue with either $ax^2 + b x + c$ and $y=1$ or $a x^2$ with $y=0.$
A: This answer breaks the problem into three cases: $y = 0$, $y = \pm 1$, and $|y| \ge 2$.  Recall that because $f$ is reduced and positive definite, $|b| \le a \le c$.
Assume first that $y = 0$.  Then because $\gcd(x, y) = \gcd(x, 0) = 1$, $x$ must equal $\pm 1$.  In this case, $f(x, y) = f(\pm 1, 0) = a$, which satisfies the condition $f \le a + |b| + c$.
Next, assume that $y = \pm 1$.  If $x = 0$, then $f(0, \pm 1) = c$.  If $x = \pm 1$, then $f(\pm 1, \pm 1) = a \pm b + c = a \pm |b| + c$.  Each of those three representations meets the conditions $\gcd(x, y) = 1$ and $f \le a + |b| + c$.  If $|x| \ge 2$, we first derive the following inequality starting with the triangle inequality:
\begin{align}
|2ax + by| & \ge |2ax| - |by|\\
           & \ge 4a - b &&\text{because } |x| \ge 2 \text{ and } y = \pm 1\\
           & \ge 4a - a &&\text{because } |b| \le a\\
           & = 3a
\end{align}
We now find a lower bound for $f(x, y)$:
\begin{align}
f(x, y) & = ax^2 + bxy + cy^2\\
& = \frac{1}{4a} \left( 4a^2x^2 + 4abxy + 4acy^2 \right)\\
& = \frac{1}{4a} \left( 4a^2x^2 + 4abxy + b^2y^2 - b^2y^2 + 4acy^2 \right)\\
& = \frac{1}{4a} \left( (2ax + by)^2 + (4ac - b^2)y^2 \right)\\
& \ge \frac{1}{4a} \left( (3a)^2 + 4ac - b^2 \right) &&\text{because } |2ax + by| \ge 3a \text{ and } y = \pm 1\\
& \ge \frac{1}{4a} \left( 9a^2 + 4ac - a^2 \right) &&\text{because } |b| \le a\\
& = 2a + c\\
& = a + a + c\\
& \ge a + |b| + c &&\text{because } |b| \le a
\end{align}
Due to the upper-bound condition $f \le a + |b| + c$, the only possible representation here is $f(x, \pm 1) = a + |b| + c$.
Finally, assume that $|y| \ge 2$.
\begin{align}
f(x, y) & = \frac{1}{4a} \left( (2ax + by)^2 + (4ac - b^2)y^2 \right) &&\text{as above}\\
& \ge \frac{4ac - b^2}{4a}y^2\\
& \ge \frac{4ac - b^2}{4a} \cdot 4 &&\text{because } |y| \ge 2\\
& = \frac{4ac - b^2}{a}\\
& \ge \frac{4ac - ac}{a} &&\text{because } |b| \le a \le c\\
& = 3c
\end{align}
Then
\begin{align}
3c \le f(x, y) & \le a + |b| + c &&\text{as just shown and using a required condition}\\
& \le 3c &&\text{because } |b| \le a \le c
\end{align}
Because both $f(x, y)$ and $a + |b| + c$ are sandwiched between $3c$ and $3c$, the only possible representation here is $f(x, y) = a + |b| + c$.
Thus, for all pairs of integers $(x, y)$ such that $\gcd(x, y) = 1$ and $f(x, y) \le a + |b| + c$, $f(x, y)$ must be one of the numbers $a$, $c$, $a - |b| + c$, or $a + |b| + c$.
Remark: This is Problem 7 of Section 3.7 and is similar to Lemma 3.24 in I. Niven, H. S. Zuckerman, H. L. Montgomery, An Introduction to the Theory of Numbers, 5th ed., Wiley (New York), 1991.  That book is a helpful reference for questions like this.
