Find the maximum of $P$ Given $a, b, c \geq 0$ and $a + b + c = 6$. Find the maximum of:
$$P = (a-b)(b-c)(c-a)$$
My guess is the maximum of $P$ would be just $0$ but I don't know whether this guess is true
And if it's true, how do I prove it?
 A: Hint: Since the expression is cyclic, WLOG Either $ a \geq b \geq c $ or $ a \leq b \leq c$.
In the former, $P \leq 0$, so we may assume the latter case.
$$ P = |b-a| \times |c - b | \times | c - a |$$
Hint: Replacing $ (a, b, c) \rightarrow ( 0, b, c+a )$ increases the first and third terms while keeping the second term constant, hence increases $P$. So, we may assume $ a = 0, b+c = 6$.
Hence, we just need to maximize $ (6-c) ( 2c-6 ) c$ subject to $ 3 \leq c \leq 6$, which is a cubic.
It is left to the reader to show that the max occurs when $ c = 3 + \sqrt{3} $ using their favorite approach.
A: Suppose $c = \min\{a,b,c\},$ so
$$(a-c)^2 \leqslant a^2, \quad (b-c)^2 \leqslant b^2.$$
By the AM-GM inequality we have
$$\begin{aligned}P^2 = (a-b)^2(b-c)^2(c-a)^2 & \leqslant a^2b^2(a-b)^2 = \frac14 \cdot 2ab \cdot 2ab \cdot (a-b)^2 \\& \leqslant \frac14\left(\frac{2ab+2ab+(a-b)^2}{3}\right) ^3\\&=\frac{(a+b)^6}{108}\leqslant \frac{(a+b+c)^6}{108}\end{aligned}$$
Therefore
$$(a+b+c)^3 \geqslant 6 \sqrt 3 (a-b)(b-c)(c-a).$$
Equality occur when $a^2+b^2=4ab,\,c= 0$ and permutation.
Because $a+b+c=6,$ so
$$(a-b)(b-c)(c-a) \leqslant \frac{(a+b+c)^3}{6\sqrt 3} = 12\sqrt 3.$$
Equality occur when
$$a^2+b^2=4ab,\,c= 0,\,a+b+c=6. \quad (1)$$
Solve $(1)$ we get $a = 3+\sqrt3,\,b = 3-\sqrt3,\, c = 0$ or $a = 3-\sqrt3, \,b = 3+\sqrt3,\, c = 0.$
Thefore $P_{\max} =  12\sqrt 3.$
