I'm adding a new answer instead of updating or appending to my original one since this is quite different based on the new information provided. The answers by Ethan Bolker and Fabio Somenzi involve using
$$a + 2n(b - n) = k(b - 2n) \tag{1}$$
to create a quadratic equation in $n$, i.e.,
$$n^2 - (k + b)n + \frac{bk - a}{2} = 0 \tag{2}$$
which has a discriminant of
$$D = (k + b)^2 - 2(bk - a) = k^2 + b^2 + 2a \tag{3}\label{eq3}$$
The value of $n$ is
$$n = \frac{k + b \pm \sqrt{D}}{2} \tag{4}\label{eq4}$$
The smallest positive integer $n$ comes from subtracting $\sqrt{D}$ where $D$ is a perfect square of the same parity as $k + b$, and less than but as close as possible to $(k + b)^2$. Thus, from \eqref{eq3}, you want to start with the smallest $k$ such that $bk - a \gt 0$, say
$$k_0 = \left\lfloor \frac{a}{b} \right\rfloor + 1 \tag{5}\label{eq5}$$
This gives
$$b \ge bk_0 - a > 0 \; \Rightarrow \; -2b \le -2(bk_0 - a) \lt 0 \tag{6}\label{eq6}$$
Using this in \eqref{eq3},
$$(k_0 + b)^2 - 2(bk_0 - a) \ge k_0^2 + 2bk_0 + b^2 - 2b \tag{7}\label{eq7}$$
However, note that
$$(k_0 + (b - 1))^2 = k_0^2 + 2bk_0 + b^2 - 2b + (1 - 2k_0) \tag{8}\label{eq8}$$
Thus, the next smaller perfect square of $D$ is $(k_0 + (b - 1))^2$. Note going from $m^2$ to $(m + 1)^2$ involves adding $2m + 1$, then to $(m + 2)^2$ involves adding $2m + 3$, then add $2m + 5$ to get $(m + 3)^2$, etc. In other words, even for very large integers, you can quickly and easily go from a perfect square to the next perfect square by adding a value which you increment by $2$ each time. This is generally faster than incrementing a value and then squaring it, as https://agner.org/optimize/optimizing_cpp.pdf says at section 14.4 Integer multiplication:
Integer multiplication takes longer time than addition and subtraction (3 -10 clock cycles, depending on the processor).
For very large integers, using some integer handling package, multiplication can be quite a bit slower. In fact, a very fast method to handle extremely big calculations, in $O(n\log(n))$ for $n$ digits, was just announced recently, e.g., the April $4$, $2019$ article at Mathematicians Develop New Algorithm for Multiplying Large Numbers.
Starting at $k = k_0$ (from \eqref{eq5}) in \eqref{eq3}, and using \eqref{eq8}, you can quickly determine each next value for $k$ and compare it to the next possible perfect square, incrementing each value(s) as appropriate, until the values match, and the result gives an integer in \eqref{eq4}. This is relatively efficient, and is generally considerably faster than doing integer divisions (e.g., for factoring, such as in my original answer, or to check modulo values). For example, https://agner.org/optimize/optimizing_cpp.pdf says at section 14.5 Integer division:
Integer division takes much longer time than addition, subtraction and multiplication (27 - 80 clock cycles for 32-bit integers, depending on the processor).
This speed issue will vary with aspects like the size of the integers, the compiler being used, etc., but I believe in basically all cases integer division will be quite slow.