Extremal problem with positive integer numbers Let $a,b$ be two positive integer numbers such that $a\sqrt{3}-b\sqrt{7}>0$. Find the minimum value of
$$
S=(a\sqrt{3}-b\sqrt{7})(a+b).
$$
Attempt I have tried and guess that the minimum value of $S$ is $(55+36)(55\sqrt{3}-36\sqrt{7})$, where $(55,36)$ is the integer solution of the Pell equation $3a^2-7b^2=3$.
 A: Here is a sketch proof that the greatest lower bound for $S$ is $(3/2)(1/\sqrt3 + 1/\sqrt7) = 1.43297\ldots$. Call this $m$. Note that $(3 - \sqrt3m)/\sqrt3 = (\sqrt7m - 3)\sqrt7 = k$, say.
For integers $a,b$ such that $\sqrt3a - \sqrt7b > 0$ define
$$\delta(a,b) = (\sqrt3a - \sqrt7b)(a + b) - m.$$
As noted in the link, we can't have $3a^2 - 7b^2 = 1$ or $2$ (consider residues mod $3$ and $7$). But, as noted by Piyush Davyanakar, there are infinitely many $(a,b)$ such that $3a^2 - 7b^2 = 3$. We can get these more simply by $(a,b) = (1,0), (55, 36), (6049, 3960)$ and in general $a_{n+1} = 110a_n - a_{n-1}$, $b_{n+1} = 110b_n - b_{n-1}$.
This can be proved together with $3a_na_{n-1} - 7b_nb_{n-1} = 165$ by simultaneous induction.
Since $3a^2 - 7b^2 \ge 3$ we have
$$\delta(a,b)(\sqrt3a + \sqrt7b) \ge 3(a + b) - m(\sqrt3a + \sqrt7b) = (3 - \sqrt3m)a - (\sqrt7m - 3)b = k(\sqrt3a - \sqrt7b) > 0.$$
On the other hand,
$$\delta(a,b)(\sqrt3a + \sqrt7b)^2 = k(3a^2 - 7b^2)$$
and this equals $3k$ for infinitely many $(a,b)$. So $\delta(a,b)$ can be made as small as we like.
A: We need to minimise
$$S = \sqrt3 \left(1+\frac{a}b\right)\cdot \color{blue}{b^2\left(\frac{a}b-\sqrt{\frac73} \right)}$$
There are infinitely many convergents from the regular continued fraction expansion of $\alpha = \sqrt{\frac73}$ which keep the terms in blue bounded, i.e. satisfy $b^2\left(\frac{a}b-\alpha \right) < \frac12$, and only convergents satisfy this.  Further as we take higher convergents the terms not in blue gets lower.  Hence it is enough to seek the minimum (or greatest lower bound) from these convergents.
We have the bounds for convergents $a_n/b_n$
$$\frac1{2b_n^2\alpha-1}> \left|\frac{a_n}{b_n} - \alpha\right| > \frac1{2b_n^2\alpha+1} $$
$$\implies \frac{1}{2\alpha -1/b_n^2} > b_n^2\left|\frac{a_n}{b_n} - \alpha\right| > \frac1{2\alpha+1/b_n^2}$$
$$\implies \frac{\sqrt3 (1+a_n/b_n)}{2\alpha-1/b_n^2} > S_n > \frac{\sqrt3 (1+a_n/b_n)}{2\alpha+1/b_n^2}$$
As we take better convergents, we get $a_n/b_n \to \alpha$ and $b_n \to \infty$, so the greatest lower bound is 
$$S_n \to S_{glb} = \frac{\sqrt3(1+\alpha)}{2\alpha} = \frac{\sqrt3}2 + \frac3{2\sqrt7}\approx 1.432972113$$ 
A: You can always find solutions $(a,b)$ that give smaller values of $S$. 
Consider solutions to the equation $x^2-21y^2=1$, $(x_1,y_1)=(55,12),(x_2,y_2)=(6049,1320)$ solutions from $(x_3,y_3)$ onward can be generated using solutions to the recursive solution $$x_{k+1}=x_1x_k+21y_1y_k\\y_{k+1}=x_1y_k+y_1x_k$$
Solutions that satisfy $3a^2-7b^2=3$ are then given by $(a_i,b_i)=(x_i,3y_i)$
We can also write $(a,b)$ as $(\sec t, \sqrt\frac{3}
{7}\tan t$. Subtitute this in the expression for $S$.
$$S=(\sqrt{3}\sec t-\sqrt{7}\sqrt\frac{3}
{7}\tan t)(\sec t+\sqrt\frac{3}
{7}\tan t)\\ =\sqrt3(\sec t-\tan t)(\sec t+\sqrt\frac{3}
{7}\tan t)$$
Now at the global minimum we must have $dS/dt = 0$. We compute the derivative and we get $$\frac{dS}{dt}=\frac{\sqrt3}7(\sqrt{21}-1)\sec t(\sec t - \tan t)^2=0 \\ \implies \sec t - \tan t=0.$$
The solutions to this lie at $t=\frac{\pi}2+2n\pi$. At these points $(a,b)=(\infty, \infty)$, hence there is no upper bound on integer solutions. 
