How to prove that $(x^2+y)(y^2+z)(z^2+x)+2xyz \leqslant 10$, where $x,y,z \in \mathbb{R}^+$ and $x^2+y^2+z^2+xyz=4$ $x,y,z \in \mathbb{R}^+$ and $x^2+y^2+z^2+xyz=4$, prove
$$(x^2+y)(y^2+z)(z^2+x)+2xyz \leqslant 10$$
I try several trig substitutions but feel hopeless with the cyclic term here. The condition $x^2+y^2+z^2+xyz=4$ made it too difficult to homogenize the inequality. I don't even know how to do brutal force either.
 A: $$ x^2+y^2+z^2+xyz=4;\space\space x,y,z>0\qquad (1)\\ (x^2+y)(y^2+z)(z^2+x)+2xyz \le 10\qquad (2)$$
$(1)\Rightarrow 0<x,y,z\le 2 \text{ and}\space x=2\iff y=z=0$
If $x=2$ then $(2)$ is trivially verified.
Put $x=2-\epsilon$ where $0\le \epsilon\le 2$ so $(1)$ becomes
$$y^2+z^2+(2-\epsilon)yz=\epsilon(4-\epsilon)\qquad (1')$$
$(1’)$ is the equation of an ellipse whose axes are contained in the diagonals. In fact, changing coordinates by a rotation of $45^{\circ}$, straightforward calculation in $(1’)$ can transforms $(y,z)$ in $(y_1,z_1)$ giving $$\left(\frac{y_1}{\sqrt{2\epsilon}}\right)^2+\left(\frac{z_1}{2(4-\epsilon)}\right)^2=1$$ 
The concerned points $(y,z)$ are those of the red arc in the figure below corresponding to the value $\epsilon = 0.8$. 

From $(2)$ we get
$$F(\epsilon,y,z)=((2-\epsilon)^2+y)(y^2+z)(z^2+2-\epsilon)+(4-2\epsilon)yz\qquad (2’)$$ where $$\begin{cases}0\le \epsilon\le 2\\ 0<y,z\le \sqrt{\epsilon(4-\epsilon)}\end{cases}$$
For $\epsilon$ fixed, $(2’)$ is maximum when $y=z$ (@HN_NH exercise for) and this occurs when $y^2=\epsilon$ (easily get from $(1’)$ or from the drawn ellipse). It follows
$$F_1(\epsilon)=(2-\epsilon)^3(\epsilon+\sqrt{\epsilon})+(2-\epsilon)^2(\epsilon^2+\epsilon\sqrt{\epsilon})+(2-\epsilon)(3\epsilon+\epsilon\sqrt{\epsilon})+\epsilon^2\sqrt{\epsilon}+\epsilon^2$$
$$F_1(\epsilon)=(2\epsilon^2-6\epsilon+8)\sqrt{\epsilon}+2\epsilon^3-10\epsilon^2+14\epsilon$$
Now, $F_1(\epsilon)$ has in its domain a maximum at $\epsilon=1$ in whose case $F_1(1)=10$. This corresponds to $(x,y,z)=(1,1,1)$ in $(2)$; for the other allowed values the proposed inequality becomes $$(x^2+y)(y^2+z)(z^2+x)+2xyz \lt 10$$
A: Let $x=\frac{2a}{\sqrt{(a+b)(a+c)}}$ and $y=\frac{2b}{\sqrt{(a+b)(b+c)}},$ where $a$, $b$ and $c$ are positives.
Thus, the condition gives $z=\frac{2c}{\sqrt{(a+c)(b+c)}}$ and we need to prove that
$$x^2y^2z^2+3xyz+\sum_{cyc}(x^3y^2+x^3z)\leq10$$ or
$$\frac{32a^2b^2c^2}{\prod\limits_{cyc}(a+b)^2}+\frac{12abc}{\prod\limits_{cyc}(a+b)}+\sum_{cyc}\left(\tfrac{16a^3b^2}{\sqrt{(a+b)^5(a+c)^3(b+c)^2}}+\tfrac{8a^3c}{\sqrt{(a+b)^3(a+c)^4(b+c)}}\right)\leq5$$ or
$$\sum_{cyc}\left(16a^3b^2(b+c)\sqrt{\tfrac{a+c}{a+b}}+8a^3c(b+c)\sqrt{(b+c)(a+b)}\right)\leq$$
$$\leq5\prod_{cyc}(a+b)^2-32a^2b^2c^2-12abc\prod_{cyc}(a+b).$$
Now, by AM-GM $$\sqrt{(b+c)(a+b)}\leq\frac{1}{2}(2b+a+c)$$ and
$$ab\sqrt{\frac{a+c}{a+b}}=\frac{ab\sqrt{(a+c)(a+b)}}{a+b}\leq\frac{\left(\frac{a+b}{2}\right)^2\cdot\frac{1}{2}(2a+b+c)}{a+b}=\frac{1}{8}(a+b)(2a+b+c).$$
Thus, it's enough to prove that:
$$\sum_{cyc}\left(16a^3b^3\sqrt{\tfrac{a+c}{a+b}}+2a^2bc(a+b)(2a+b+c)+4a^3c(b+c)(2b+a+c)\right)\leq$$
$$\leq5\prod_{cyc}(a+b)^2-32a^2b^2c^2-12abc\prod_{cyc}(a+b)$$ or
$$16\sum_{cyc}a^3b^3\sqrt{\frac{a+c}{a+b}}\leq\sum_{cyc}(5a^4b^2+a^4c^2+6a^3b^3+2a^4bc+4a^3b^2c+2a^3c^2b-4a^2b^2c^2).$$
Now, by C-S $$\sum_{cyc}a^3b^3\sqrt{\frac{a+c}{a+b}}\leq\sqrt{\sum_{cyc}\frac{a^3b^3}{a+b}\sum_{cyc}a^3b^3(a+c)}.$$
Id est, it's enough to prove that:
$$256\sum_{cyc}\frac{a^3b^3}{a+b}\sum_{cyc}a^3b^3(a+c)\leq\left(\sum_{cyc}(5a^4b^2+a^4c^2+6a^3b^3+2a^4bc+4a^3b^2c+2a^3c^2b-4a^2b^2c^2)\right)^2,$$ 
which is obviously true after full expanding.
The last part for you. 
A: $\color{brown}{\textbf{Trigonometrical substitution.}}$
From the given conditions should
\begin{cases}
x,y,z \in [0,2]\\[4pt]
(2z+xy)^2 = (4-x^2)(4-y^2).\tag1
\end{cases}
Taking in account $(1),$ can be applied substitution
$$x=2\sin a,\quad y=2\sin b\quad \Rightarrow \quad z = 2\cos(a+b) = 2\sin c,\tag2$$
where
$$a\ge 0,\quad b\ge 0,\quad c\ge 0,\quad a+b+c=\dfrac\pi2.\tag3$$
Then the given inequality takes the form of
$$4(2\sin^2a+\sin b)(2\sin^2b+\sin c)(2\sin^2c+\sin a) + 8\sin a\sin b\sin c \le 5.\tag4$$
$\color{brown}{\textbf{The proof.}}$
Taking in account $(3),$ one can get
\begin{align}
&2\sin^2 a + \sin b = \sin^2 a + 1 - \cos^2 a + \cos (c+a)\\[4pt]
&= 1 + \cos a (\cos c - \cos a) + \sin a (\sin a -\sin c) 
= 1 - A \sin (a+\varphi),
\end{align}
where
\begin{align}
&A = \sqrt {(\cos c -\cos a)^2 + (\sin a - \sin c)^2} 
= \sqrt {2 - 2\cos (a-c)} = 2 \sin \dfrac{|a-c|}2,\\[4pt]
&\tan \varphi = \dfrac{\cos c -\cos a}{\sin a - \sin c}
= \dfrac{2\sin \frac{a-c}2 \sin \frac{a+c}2}{2\sin \frac{a-c}2 \cos \frac{a+c}2}
= \tan \frac{a+c}2.
\end{align}
Therefore,
$$2\sin^2 a + \sin b = 1 - 2 \sin \frac {|a-c|}2 \sin \frac{3a+c}2.$$
Using the symmetry of task by $a,b,c$, should
$$2\sin^2 a + \sin b \le 1,\quad 2\sin^2 b + \sin c \le 1,\quad
2\sin^2 c + \sin a \le 1.\tag5$$
On the other hand, is known the identity
\begin{align}
\sin(a+b+c) = \cos a \cos b \sin c + \cos a \sin b \cos c + \sin a \cos b \cos c - \sin a \sin b \sin c.
\end{align}
Taking in account $(3),$ one can get
\begin{align}
&\sin a \sin b \sin c = \cos a \cos b \sin c + \cos a \sin b \cos c 
+ \sin a \cos b \cos c - \sin(a+b+c)\\[4pt]
&= \dfrac12(\cos a \sin(b+c) + \cos c \sin(a+b) + \cos b \sin(c+a)) - 1\\[4pt]
&= \dfrac12(\cos^2 a + \cos^2 c + \cos^2 b) - 1
= \dfrac14(\cos 2a + \cos 2b + \cos 2c - 1)\\[4pt]
&= \dfrac18 (\cos(a+b)\cos(a-b) + \cos(b+c)\cos(b-c) + \cos(c+a)\cos(c-a) - 2)\\[4pt]
&= \dfrac18 (\sin c\cos(a-b) + \sin a\cos(b-c) + \sin b\cos(c-a) - 2),
\end{align}
$$\sin a \sin b \sin c \le \dfrac18.\tag6$$
Since from $(5),(6)$ should $(4),$ then the given inequality is proved.
