What is the algebraic intuition behind Vieta jumping in IMO1988 Problem 6? Problem 6 of the 1988 International Mathematical Olympiad notoriously asked:

Let $a$ and $b$ be positive integers and $k=\frac{a^2+b^2}{1+ab}$. Show that if $k$ is an integer then $k$ is a perfect square.

The usual way to show this involves a technique called Vieta jumping. See Wikipedia or this MSE post.
I can follow the Vieta jumping proof, but it seems a bit strained to me. You play around with equations that magically work out at the end. I don't see how anyone could have come up with that problem using that proof.
Is there a natural or canonical way to see the answer to the problem, maybe using (abstract) algebra or more powerful tools? In addition, how can someone come up with a problem like this?
 A: I posted my answer in this separate subject (click here) i couldn't respond directly because my reputation was too low. After posting my answer my reputation was high enough to also place my reply here. So for my solution click on the link above.
A: At the heart of these so-called "Vieta-jumping" techniques are certain symmetries (reflections) on conics. These symmetries govern descent in the group of integer points of the conic. If you wish to develop a deeper understanding of these proofs then I highly recommend that you study them from this more general perspective, where you will find much beauty and unification. 
The group laws on conics can be viewed essentially as special cases of the group law on elliptic curves (e.g. see Franz Lemmermeyer's "poor man's" papers), which is a helpful perspective to know. See also Sam Northshield's expositions on  associativity of the secant method (both linked here).
If memory serves correct, many of these contest problems are closely associated with so-called Richaud-Degert quadratic irrationals, which have short continued fraction expansions (or, equivalently, small fundamental units).  Searching on "Richaud Degert" etc should locate pertinent literature (e.g. Lemmermeyer's Higher descent on Pell Conics 1).  Many of the classical results are couched in the language of Pell equations, but it is usually not difficult to translate the results into more geometric language.
So, in summary, your query about a "natural or canonical way to see the answer to the problem" is given a beautiful answer when you study the group laws of conics (and closely related results such as the theory of Pell equations). Studying these results will provide much motivation and intuition for generalizations such as group laws on elliptic curves.
See also Aubry's beautiful reflective generation of primitive Pythagorean triples, which is a special case of modern general results of Wall, Vinberg, Scharlau et al. on reflective lattices, i.e. arithmetic groups of isometries generated by reflections in hyperplanes.
A: There is a generalization of this problem too, that proposed in CRUX, Problem 1420, Shailesh Shirali. 
If $a, b, c$ are positive integers such that:
$$0 < a^2 + b^2 -abc \le c$$
then $a^2 + b^2 -abc$ is perfect square.
A: I don't have a strong math background, I think my solution is more suitable for the Math olympiad contest as a young student.
Problem Definition:
Let $a$ and $b$ be positive integers such that $ab+1$ divides $a^2+b^2$. Show that
 $\frac{a^2+b^2}{ab+1}$
is the square of an integer.
Proof:
Let $ t = \frac{a^2+b^2}{ab+1}$.
W.L.O.G we assume $1 \le a \le b$.
If $t=1$, we are done.
If $a=b$, we have  $2 b^2 = t (b^2+1) \ge 2 (b^2+1)$, contradiction.
If $a = 1$, we have $1+b^2 = t(b+1)$. 
As $t \equiv 1 (mod~b)$, we have  $t \ge 1+b$, contradiction.
Therefore, the major focus is on $t>1$ and $1 < a < b$.
Since $a^2 \equiv t ~(mod~b)$,  we define $k$ such that $a^2 = t + k b$, hence $k$ is an integer and $k < a$. 
Note that 
$ t = \frac{a^2+b^2}{ab+1} < \frac{a}{b} + \frac{b}{a}  \le 1 + b/2 \le b $.
If $k<0$, we have $a^2 = t+kb \le t - b \le 0$, contradiction.
If $k=0$, we have $t = a^2$, done.
Hence, the only remaining case is $0<k<a$.
In this case, we claim that 
$$ t = \frac{k^2+a^2}{k a+1} $$
Substitute $t$, we have  $ a^2+b^2 = (a^2 - k b)(ab+1)$, hence 
$$ b = (a^2 - k b)a - k = a t - k$$
As a result, $k+b = at$ and $kb = a^2-t$, that is, $k$ and $b$ are the solution of the equation
$$ x^2 - at x + (a^2-t) = 0 $$
One solution is $x = b$, the other solution is $x = k$, therefore the claim follows.
Consequently, whenever a feasible pair $(a,b)$ exists, we create another feasible pair $(k,a)$. 
By replacing $(a,b)$ by $(k,a)$ the pair become smaller as  $0<k<a<b$.
Because all the other case will result in $t$ is a square number, hence by repeating such replacement, we will finally get into the case where $t$ is a square number.
\qed
A: The idea has been known (at least) since Gauss' Disquisitiones Arithmeticae 200 years ago.  "Vieta jumping" is a name used only in competition manuals, the accepted term in mathematics is "reduction theory of quadratic forms". The reduction-theory solution is the canonical solution, and I do not know any solutions using other methods, but some presentations of the method can make it seem artificial.
The reason more competitors did not solve the problem is that in the quaint old days, high school students were not learning heavy machinery before going to the IMO. 
Even knowing the theory, it may not be easy to recognize within a few hours that the problem would fall to a straightforward application of the 'rotation' group of the integer points on the conic (showing that any positive integer solution can be moved to a smaller one with $ab=0$) by writing out the equation as $a^2 - kab + b^2=k$ and turning the handle. Stripped of computational details, that is what the solution does. 
It was also known since the 1800's that this type of quadratic form has special properties and is easier to analyze. 

For degree $(2,2,...,2)$ equations of higher total degree, such as Markov triples (solutions of $x^2+y^2+z^2=3xyz$), the "Vieta" transformation does not have a specific name, but has been the main tool for organizing the solutions since the first papers in the $19$th century where these equations were analyzed.
https://www.encyclopediaofmath.org/index.php/Hurwitz_equation
A: The same Wikipedia page quotes Arthur Engel about the history of the problem.

Nobody of the six members of the Australian problem committee could solve it … it was [then] sent to the four most renowned Australian number theorists. They were asked to work on it for six hours. None of them could solve it in this time … the jury finally had the courage to choose it as the last problem of the competition. Eleven students gave perfect solutions.

It is implied that the Australians had tried to solve the problem with more general tools and failed, and that the idea of Viète jumping was created specifically for this problem. 
Another reason why Viète jumping is the canonical solution to the problem is that the problems of the IMO usually have only a few solutions, with special prizes given out for particularly ingenious ones, such as to Boreico Iurie for IMO 2005 Q3:

Let $x,y,z\in\Bbb R^+$ such that $xyz\ge1$. Prove that
  $$\frac{x^5-x^2}{x^5+y^2+z^2}+\frac{y^5-y^2}{y^5+z^2+x^2}+\frac{z^5-z^2}{z^5+x^2+y^2}\ge0.$$

(I got this from a book about the mathematical olympiads in China from 2003 to 2006, which had the IMO problems for completeness.) In particular, the Viète jumping proof is the only one with mathematics simple enough for a competitor to understand.
If you insist on a more "natural" solution, a geometric interpretation of the technique is also on the Wikipedia page, involving lattice points on a hyperbola. Dubuque's answer has more on this, including the possible sources for the problem.
