How to solve a bivariate quadratic (not necessarily Pell-type) equation?

Question

Simple Pell equations often have solutions that can be found with little work given certain conditions. These are of the form $x_{n}^{2} - A y_{n}^{2} = \pm 1$. There are harder equations that involve non-squared variable terms. In this view how is a solution to these equations found? As an example, what is the solution to the equation \begin{align} 4 x^{2} + 5 y^{2} + 20 x y - 24 x - 20 y + 8 = 0 ? \end{align}

Answer 1

Legendre established that the general bivariate quadratic

$$ax^2+bxy+cy^2+dx+ey+f=0\tag1$$

can be, in fact, transformed to the Pell-type equation,

$$p^2-Dq^2 = k\tag2$$

with discriminant $D=b^2-4ac$ and,

$$p = Dy-2ae+bd$$

$$q = 2ax+by+d$$

$$k = 4a(ae^2+cd^2-bde+Df)$$

Assuming for a given $D,k$ that $(2)$ has integer solutions $p,q$, one can then always recover $x,y$. See also this post.

Added: It does not seem to guarantee that if the transformed Pell-type equation $(2)$ has integer solutions, then the original $(1)$ has as well. For example, let's use a variation of your equation,

$$4x^2+20xy+5y^2-24x-20y\color{brown}{-73}=0$$

which can be transformed to,

$$p^2-320q^2 = -455680$$

By the Alpertron, this has small solution $p,q = 80,38$ which yields $x, y = \frac{37}{8}, \frac{5}{4}$.

However, another family starts with $p,q = 640,54$ and this does yield integer $x,y = 2,3$.

Answer 2

Since we have $$5(y-2)^2=4(12-5xy-(x-3)^2)$$ $y$ has to be even. Let $y=2m$ where $m\in\mathbb Z$. Then, we have $$5\cdot 4(m-1)^2=4(12-5xy-(x-3)^2)$$ $$\Rightarrow 5(m-1)^2=12-5xy-(x-3)^2$$ $$\Rightarrow (x-3)^2=5(-xy-(m-1)^2+2)+2$$ This implies that $(x-3)^2\equiv 2\pmod 5$. However, there is no such $x$ because $$a^2\equiv 0,1,4\pmod 5.$$