Vieta Jumping: Related to IMO problem 6, 1988: If $ab + 1$ divides $a^2 + b^2$ then $ab + 1$ cannot be a perfect square. The famous IMO problem 6 states that if $a,b$ are positive integers, such that $ab + 1$ divides $a^2 + b^2$, then $\frac{ a^2 + b^2}{ab + 1 }$ is a perfect square, namely, $gcd(a,b)^2$.  How about a modification of this problem: 


*

*If $a,b$ are (strictly) positive integers, such that $ab + 1$ divides $a^2 + b^2$, then $ab + 1$ cannot be a perfect square. 


I am looking for a proof of the claim above, or a counter-example.
One possible approach towards a proof is the following: 
Suppose there is such a pair $(a,b)$ as above, then by the famous IMO problem 6 from 1988, $\frac{ a^2 + b^2}{ab + 1 } = g^2$ where $g = gcd(a,b)$. Since $ab + 1$ is a perfect square, $a^2 + b^2 = c^2$ for some integer $c$. So that $(a,b,c)$ is a Pythagorean tripple, therefore there exists positive integers $n,m,l$ with $n$ coprime to $m$, such that $a = l(n^2 - m^2)$ and $b = 2lnm$ - by Euclids formula.
Then by plugging in $a$ and $b$ in terms of $n,m,l$ into the original equation, and solving for $l$, it is possible to obtain the following:
There exists positive coprime integers $n,m$ such that 
$\frac{(n^2 + m^2 + 1)(n^2 + m^2 - 1)}{2mn(n+m)(n-m)}$ is a perfect square.
If we put the quotient above into a program, as in this python code snipet:
N = 1000

for n in range(1,N):
    for m in range(n+1, N):
        A = (n*n + m*m + 1)*(n*n + m*m - 1)
        B = 2*m*n*(n+m)*(m-n)
        if A % B == 0:
            print(A/B)

The quotient always is 2, regardless of whether or not $n$ and $m$ are co prime! So if the very strong implication


*

*$2mn(n+m)(n-m) \mid (n^2 + m^2 + 1)(n^2 + m^2 - 1) \implies \frac{(n^2 + m^2 + 1)(n^2 + m^2 - 1)}{2mn(n+m)(n-m)} = 2$ 


holds, then the original problem will be solved. 
 A: Here is a bit of a hacky answer to your question in the affirmative
You observe that $k = \mathrm{gcd}(a,b)^2$.  After plodding through various resources, it is because one can Viete Jump:
$$ (a,b) \mapsto  \big(a_1, b_1\big) \mapsto \dots  \mapsto (k,0) $$
and the gcd is conserved. Once I agree with you, let $a = \text{gcd}(a,b) \, c$ and $b = \text{gcd}(a,b) \, d$ so that
\begin{eqnarray} ab+1 &=& \frac{a^2+b^2}{\mathrm{gcd}(a,b)^2}= c^2 + d^2  \\
&=& \,\text{gcd}(a,b)^2 \, cd + 1
\end{eqnarray}
In this way there are two conditions $c, d$ might solve (where the two $\square$ are different) and $\text{gcd}(c,d)=1$:
\begin{eqnarray*}
c^2 - \square \, cd + d^2 &=& 1 \\
c^2 + d^2 &=& \square
\end{eqnarray*}
Hopefully these two equations leads to a contradiction.

Added 11/15 The answer is definitely no.  Let $k = \mathrm{gcd}(a,b)$ We are trying to solve in integers:
\begin{eqnarray} c^2 - k\; cd + d^2 &=& 1  \\
c^2 + d^2 = \square
\end{eqnarray}
As I learned, the first can be solve with $(c,d) = (1,0)$ or $(k,1)$ and there are an infinite family of solutions using consecutive terms of a recursive sequence [2, 3]
$$ x_{n+1} = k \, x_n + x_{n-1} $$
There are sometimes ways to link Pythagorean triples to Pell equations [1] (Modular Tree of Pythagoras)
$$  x_{n+1}^2  + \frac{1}{2} x_n^2 < \sqrt{x_{n+1}^2 + x_n^2 } = x_{n+1}^2 \sqrt{1 + (x_n/x_{n+1})^2} < x_{n+1}^2  + \frac{1}{2} x_n^2 + 1$$
This cannot be an integer.  So any time we solve the Pell equation, we cannot also solve Pythagoras. $\quad\quad\square$

Old Answer
This is discussed on Wikipedia's article on Vieta Jumping:

Nobody of the six members of the Australian problem committee could solve it. Two of the members were George Szekeres and his wife, both famous problem solvers and problem creators. [...] The problem committee submitted it to the jury of the XXIX IMO marked with a double asterisk, which meant a superhard problem, possibly too hard to pose. After a long discussion, the jury finally had the courage to choose it as the last problem of the competition. Eleven students gave perfect solutions.

Among the eleven was Bau Chau Ngô (Fields Medal 2010).  His work on the Fundamental Lemma also has a jumpy flavor [1, 2, 3] but is quite advanced.
The discussion on YouTube is helpful as well.  These videos give a thorough discussion of different ways to solve


*

*The Legend of Question Six (Numberphile)

*The Return of the Legend of Question Six (Numberphile2)


These may not directly solve your problem but provide historical context and indicate possible strategies.

In the Wikipedia article, the example of Viete Jumping is IMO 1988/6 -- the same as asked in the question:

Let $a,b$ be positive integers such that $ab+1$ divides $a^2 + b^2$ show that $ \frac{a^2 + b^2}{ab+1}$ is  a perfect square.

and the solution goes in three steps 
#1 Let $a, b \geq 0$ be solutions to  $\frac{a^2 + b^2}{ab+1} = k$ such that $k$ is not a perfect square: $k \neq \square$
#2 Starting from $(a,b)$ we can try to generate another solution $(x,b)$ which solves the quadratic equation:
$$ x^2 - kb\, x +(b^2 - k) = 0 $$
The map $(a,b) \mapsto (a_1,b)$ is our Vieta jumping Since both $a, a_1$ are acceptable solutions we have:
$$ (x-a)(x-a_1) = x^2 - (a + a_1) x + aa_1 = 0$$
By the Viete equations (comparing the coefficients. We find out two things:


*

*$ a + a_1 = kb $ so that $a_1 = kb - a \in \mathbb{Z}$ (it is an integer)

*$ aa_1 = b^2 - k $ so that $a_1 = \frac{b^2 - k}{a} \neq 0$
#3 If $a \geq b$ we can deduce that $a_1 \geq 0$ (is positive) and additionally $  a > a_1 \ge 0 $


*

*From #2 $ a_1 = \frac{b^2 - k }{a} < \frac{b^2}{a} < \frac{a^2}{a}=a $

*$\frac{a_1^2 + b^2}{ab+1}= k > 0 $ implies that $ a_1b+1 > 0$ or $a_1 > - \frac{1}{b}$ but $a \in \mathbb{Z}$ so $a_1 \geq 0$.
Summary We've show that given two positive numbers $a,b$ solving $\frac{a^2+b^2}{ab+1}=k$ with $k \neq \square$ we can always find another solution $(a_1,b)$ solving the same equation with $a > a_1$.  
Then the Viete jump consists of a map:
$$ (a,b) \mapsto \left\{\begin{array}{rc}
(b,a) & \text{ if } a \leq b \\
(\frac{b^2-k}{a},b) & \text{ if } a \geq b
 \end{array}\right.$$

While this does not solve your problem -- to show that $ab+1 \neq \square$ -- it does indicate possibly where to start and some possible resources.
A quick use of Bezout formula shows that $ab+1$ should also divide 
$$ \big[(a^2 + b^2) + 2(ab+1)\big] + 
\big[(a^2 + b^2) - 2(ab+1)\big] = (a+b)^2 +(a-b)^2  $$
and this could lead to your contradiction. 
