How do i approach this inequality?

Question

Let $0≤a,b,c≤2$, and $a+b+c=3$. Determine the largest and smallest values of $$\sqrt{a(b+1)}+\sqrt{b(c+1)}+\sqrt{c(a+1)}.$$

Answer 1

$(a+b+c)((b+1)+(c+1)+(a+1))\geq(\sqrt{(a(b+1))}+\sqrt{(b(c+1))}+\sqrt{(c(a+1))})^2$ by Cauchy Inequality. We know the left side is $3\times6=18$, so maximum is $\sqrt{18}$.

For minimum, note that one of $a\geq b$, $b\geq c$ or $c\geq a$ must be true. Assume $a\geq b$ by symmetry. Then $\sqrt{(a(b+1))}+\sqrt{(b(c+1))}+\sqrt{(c(a+1))} \geq (\sqrt a + \sqrt c)\sqrt{(b+1)}+\sqrt{(b(c+1))} \geq \sqrt{((a+c)(b+1))}+\sqrt{(b)}$

The first inequality comes from $a+1 \geq b+1$.
The second inequality comes from $(\sqrt a + \sqrt c)^2 \geq (\sqrt{a+c})^2$

What this result means is that for any pair $a,b,c$ where $a \geq b$, if we let $c'=0$ and $a'=a+c$ the result will be smaller than the original because $\sqrt{((a+c)(b+1))}+\sqrt{(b)} = \sqrt{(a'(b+1))}+\sqrt{(b(c'+1))}+\sqrt{(c'(a'+1))}$
Hence we can set $c=0$ to minimize our result.

When $c=0$, $\sqrt{(a(b+1))}+\sqrt{(b(c+1))}+\sqrt{(c(a+1))}=\sqrt{(a(b+1))}+\sqrt{(b)}$ where $a+b=3$

Now $\sqrt{(a(b+1))}+\sqrt{(b)} = \sqrt{((3-b)(b+1))}+\sqrt{(b)}$. Note that since $a \leq 2$ and $b \leq a$ and $a+b=3$, $b$ can only take values between $[1,{3\over2}]$. This function will decrease as $b$ decrease in the range $[1,{3\over2}]$.

Proof: Let $b=x+1$, then $\sqrt{((3-b)(b+1))}+\sqrt{(b)}=\sqrt{(4-x^2)}+\sqrt{(x+1)}$ where $x \in [0,{1\over2}]$.

Now suppose $0\leq x_1<x_2 \leq{1\over2}$. Then we have
$(x_2+x_1)(\sqrt{x_2+1}+\sqrt{x_1+1}) \leq \sqrt6 < \sqrt15 \leq (\sqrt{4-{x_1}^2}+\sqrt{4-{x_2}^2})$

Hence ${{x_2}^2-{x_1}^2\over \sqrt{4-{x_1}^2}+\sqrt{4-{x_2}^2}} < {x_2-x_1\over \sqrt{x_2+1}+\sqrt{x_1+1}}$

Hence$ \sqrt{4-{x_1}^2}-\sqrt{4-{x_2}^2}<\sqrt{x_2+1}-\sqrt{x_1+1}$

Hence $\sqrt{4-{x_1}^2}+\sqrt{x_1+1}<\sqrt{x_2+1}+\sqrt{4-{x_2}^2}$

Hence $\sqrt{(4-x^2)}+\sqrt{(x+1)}$ decreases as $x$ decreases.

Hence $\sqrt{((3-b)(b+1))}+\sqrt{(b)}$ decreases as $b$ decreases.

So $b=1$ will minimize the result and the minimum is $3$ when $a=2, b=1, c=0$

Answer 2

By AM-GM, $2a+(b+1) \ge 2\sqrt{2a\cdot(b+1)}$.
Cyclically summing three such inequalities, we get $3\sqrt2 \ge \sum_{cyc} \sqrt{a(b+1)}$.
As $a=b=c=1$ gives equality, this is indeed the maximum.

For the minimum, we may note that the function is concave in each of its three variables, and hence will achieve minimum when at least two of its variables are on the boundaries of the interval. So WLOG we can consider the cases $a,b \in \{0, 2\}$, the feasible possibilities need $a \neq b$, to get the minimum of $3$, when $a=0, b=2, c=1$ or a cyclic permutation.