Inequality $a^2+b^2+c^2 \leq a^2b+b^2c+c^2a+1.$

Prove that:

$$a^2+b^2+c^2 \leq a^2b+b^2c+c^2a+1, (\forall) a,b,c \in [0,1].$$

I have no idea, I try $AM\geq GM$ but still nothing.

-
What's AM and GM? –  Patrick Li Jan 5 '13 at 0:24
@PatrickLi Arithmetic and Geometric mean‌​. –  Arthur Jan 5 '13 at 0:36

Notice that the equality case is whenever one variable is $1$ and the other two are $0$. So a "balanced" inequality like $AM \geq GM$ is unlikely to work, and we'd be better off trying a different approach. This inequality in particular breaks down to a Jensen's/smoothing/convexity-type approach.

First of all, rearrange the inequality to $a^2+b^2+c^2-a^2b-b^2c-c^2a \leq 1$.

Now, notice that the function $a^2+b^2+c^2-a^2b-b^2c-c^2a$ is convex when $a,b,c \in [0,1]$. You can prove this using easy algebra, or by noticing that the second derivative with respect to any of the three variables is nonnegative whenever $a,b,c \in [0,1]$.

Any convex function over some region attains its maximum on the boundary of that region. Therefore, it suffices to check the boundary of the domain of the inequality, $a,b,c \in [0,1]$. The boundary is wherever one of the variables is equal to $1$ or $0$.

In the first case, one of the variables, WLOG $a$, is $1$. Then the inequality is easy: it simplifies to $b^2-b-b^2c \leq 0$, which is true because $b^2 \leq b$.

In the second case, when one of the variables, WLOG $a$, is $0$, the inequality is also easy: it simplifies to $b^2+c^2-b^2c-1 \leq 0$, which is true because $b^2+c^2-b^2c-1 \leq b^2+c^2-b^2c^2-1 = -(1-b^2)(1-c^2) \leq 0$.

We have shown that the inequality holds on the boundary of the domain. Therefore, by the logic above, it holds everywhere, and we are done.

-