Below are six methods - whose variety may prove somewhat instructive.
$(0)\ $ By the Parity Root Test, $\rm\: x^2-5\:x-1\:$ has no rational roots since it has odd leading coefficient, odd constant term and odd coefficient sum.
$(1)\ $ By the Rational Root Test, the only possible rational roots of $\rm\ x^2 -5\ x - 1\ $ are $\rm\ x = \pm 1\:.$
$(2)\ $ Complete your proof: show that $\rm\ (a,b) = 1\ \Rightarrow\ (ab,\:a^2-b^2) = 1\:.\:$ For example, if the prime $\rm\ p\ |\ a,\ a^2-b^2\ $ then $\rm\ p\ |\ b^2\ \Rightarrow\ p\ |\ b\:.\ $ Alternatively, since $\rm\ a,\:b\ $ are coprime to $\rm\ a-b,\:a+b\ $ then their products $\rm\ a\:b,\ a^2-b^2\:,\: $ are also coprime, by Euclid's Lemma.
$(3)\ $ Suppose it has a rational root $\rm\: R = A/B\:.\ $ Put it into lowest terms, so that $\rm\: B\:$ is minimal. $\rm\ R = 5 + 1/R\ \Rightarrow\ A/B = (5\:A+B)/A\:.\:$ Taking fractional parts yields $\rm\ b/B = a/A\ $ for $\rm 0\le b < B\:.\:$ But $\rm\ b\ne0\ \Rightarrow\ A/B = a/b\ $ contra minimality of $\rm\:B\:.\:$ So $\rm\:b = 0\:,\:$ i.e. $\rm\ A/B\ $ has fractional part $ = 0\:,\:$ so $\rm\ R = A/B\ $ is an integer. Then so too is $\rm\ 1/R = R-5\:.\:$ So $\rm\ R = \pm 1\:,\:$ contra $\rm\ R^2 - 1 = 5\:R\:.$
$(4)\ $ As in $\rm(3),\ \ R = A/B = C/A\:,\: $ with $\rm\:A/B\:$ in lowest terms, i.e. $\rm\:B =\: $ least denominator of $\rm\:R\:.\:$ By unique fractionization, the least denominator divides every denominator, therefore $\rm\:B\ |\ A\:,\:$ which concludes the proof as in $(3)$.
For more on the relationship between $(3)$ and $(4)$, follow the above link, where you'll find my analysis of analogous general square-root irrationality proofs, and links to an interesting discussion of such between John Conway and I.
$(5)\ $ As Euclid showed a very long time ago, the Euclidean gcd algorithm works also for rationals, so they too have gcds, and such gcds enjoy the same laws as for integers, e.g. the distributive law. Thus $\rm\ (r,1)^2 = (r^2,r,1) = (5r+1,r,1) = (r,1)\ $ so $\rm\ (r,1) = 1\:,\:$ so $\rm\ 1\ |\ r \ $ in $\rm\:\mathbb Z\:,\:$ i.e. $\rm\ r\in\mathbb Z\:,\:$ and the proof concludes as above. This is - at the heart - the same proof hinted by Aryabhata using non-termination of the continued-fraction algorithm (a variant of the Euclidean gcd algorithm). Alternatively, scaling the prior equations by $\rm\:b^2\:,\:$ where $\rm\ r = a/b\:,\:$ converts it to one using only integer gcd's, namely $\rm\ 1 = (a,b)^2 = (a^2,ab,b^2) = (5ab+b^2,ab,b^2) = b\:(a,b)\:$ so $\rm\ b\ |\ 1\:.\:$
These are essentially special cases of gcd/ideal-theoretic ways to prove that $\rm\: \mathbb Z\:$ is integrally-closed, i.e. it satisfies the monic Rational Root Test. Namely, a rational root of a monic polynomial $\rm\in\mathbb Z[x]\:$ must be integral. Perhaps the slickest way to prove such results is the elegant one-line proof using Dedekind's notion of conductor ideal. Follow that link (and its links) for much further discussion (both elementary and advanced).