This is a serious problem with the way Rudin complicates a simple problem. In order to prove that the set $A$ has no largest element it is not really necessary to find an explicit formula for a member $q\in A$ with $q > p$ where $p$ is a given member of $A$. We only need to prove that if $p \in A$ then there is a $q \in A$ with $q > p$ and for that that we don't need any mysterious / magical formula for $q$ in terms of $p$.
Hardy in his textbook A Course of Pure Mathematics does it so much better and also teaches the way things work in real-analysis. The crux of the argument by Hardy is that the set of positive rationals is partitioned into $A, B$ such that $A \cup B = \mathbb{Q}^{+}, A\cap B = \emptyset$ and further that we can find a member of $A$ and a member of $B$ which are as close to each other as we please.
Clearly $1 \in A, 2 \in B$ and given any positive integer $n$ we have the following chain of increasing rational numbers $$1, 1 + \frac{1}{n}, 1 + \frac{2}{n}, \ldots, 1 + \frac{n}{n} = 2$$ such that successive numbers in the above chain differ by $1/n$. Since the first member of the chain lies in $A$ and the last member of the chain lies in $B$ it follows that there is a last number in the chain which lies in $A$ and the next one belongs to $B$. Thus we have found two rationals $q, r$ with $q \in A, r \in B$ such that $r - q = 1/n$. Thus we can find a member of $A$ and a member of $B$ which are as close to each other as we please. Moreover we can choose $q, r$ both less than 2.
Now suppose we have a $p \in A$ then by definition of $A$ we have $p^{2} < 2$ and hence $\epsilon = 2 - p^{2}$ is a positive rational number. We can now find a positive integer $n$ such that $1/n < \epsilon/4$. Then by argument in previous paragraph we can find $q, r$ with $q \in A, r \in B$ with $r - q = 1/n < \epsilon/4$. Also both $q, r$ can be chosen to be less than $2$ so $r + q < 4$ and hence $r^{2} - q^{2} = (r - q)(r + q) < \epsilon$. And this means that $$(r^{2} - 2) + (2 - q^{2}) < \epsilon$$ Since $q\in A, r \in B$ each expression in parentheses in above equation is positive and their sum is less than $\epsilon$ so that each expression itself is less than $\epsilon$. Therefore we have $$r^{2} - 2 < \epsilon, 2 - q^{2} < \epsilon$$ By definition of $\epsilon$ it now follows that $$2 - q^{2} < 2 - p^{2}$$ or $q > p$ and thus we have found $q > p, q \in A$.
Note that the most of the deep significant theorems in real-analysis are existential in nature where it is of utmost importance to focus on the existence of a quantity with certain specific desired properties rather than explicitly finding such a quantity and the above proof is a typical example of proofs seen in real-analysis where it shows the existence of an element $q \in A, q > p$ given an element $p \in A$ without explicitly giving a formula for $q$ in terms of $p$.
Rudin on the other hand uses some sort of numerical technique to have explicit formula for $q$ in terms of $p$. Hardy also gives this approach by asking his readers to prove the following simple theorem:
If $x$ is a positive approximation to $\sqrt{2}$ then $(x + 2)/(x + 1)$ is a better approximation to $\sqrt{2}$ but in a different direction so that if $x < \sqrt{2}$ then $(x + 2)/(x + 1) > \sqrt{2}$.
Applying this rule twice we get $$x \to \frac{x + 2}{x + 1}\to\dfrac{\dfrac{x + 2}{x + 1} + 2}{\dfrac{x + 2}{x + 1} + 1} = \frac{3x + 4}{2x + 3}$$ so we can choose $q = (3p + 4)/(2p + 3)$ also.