Nature of critical points of $f(x,y,z)$ How can I study the nature of the critical points of $f(x,y,z)=3xy^2+6y^2+x^2z+3z^2$?
Since
$$
\begin{cases}
f'_x=3y^2+2xz\\
f'_y=6xy+12y\\
f'_z=x^2+6z
\end{cases}
$$
the only critical point is $(0,0,0)$.
The Hessian matrix is
$$
H_f(0,0,0)=\left(
\begin{matrix}
0 & 0 & 0\\
0 & 12 & 0\\
0 & 0 & 6
\end{matrix}\right),
$$
whose determinant is zero. Now I'm stuck, because I've tried to study $f$ along many restriction, and every time I realized that $O$ is a minimum, but I am not able to prove (or disprove) that $O$ is indeed a minimum.
What can I do now?
 A: $3xy^2 +6y^2$ is certainly nonnegative in small enough balls around the origin, but the other part isn’t obviously always nonnegative, so let’s work there.  It would be convenient to deal with homogeneous polynomials, probably.  So try curves of the form $(t, 0, at^2)$ for various values of $a$.
A: Let $f(x,y,z)=3xy^2+6y^2+x^2z+3z^2$ then $f\in C^{2}({\bf R})$. For $(x,y,z)\in {\bf R}^{3}$ then $\nabla f(x,y,z)=(0,0,0)\iff \begin{cases}2xz+3y^{2}=0,\\6xy+12y=0,\\x^{2}+6z=0 \end{cases}$. Thus ${\bf 0}:=(0,0,0)$ is indeed the only critical point. The Hessian matrix $H_f$ at ${\bf 0}$ is given by $H_f({\bf 0})=\left.\begin{pmatrix} 2z&6y&2x\\6y&6x+12&0\\2x&0&6\end{pmatrix}\right|_{{\bf 0}}=\begin{pmatrix}0&0&0\\0&12&0\\0&0&6\end{pmatrix}$, then $\det(H_f({\bf 0}))=0$ and the Hessian-test is inconclusive.
Thus, we need to review in the definition directely, considering that $f({\bf 0})=0$.

*

*If $x=y=0$, then $3z^{2}=f(x,y,z)>f({\bf 0})=0$ if $z\not=0$, so $f({\bf 0})$ is not local maximum.

*If $z=-x^{3}$ and $y=0$, then $x^{5}(3x-1)=f(x,y,z)<f({\bf 0})=0$ if $0<x<\frac{1}{3}$, so $f({\bf 0})$ is not a local mimimum.

Recall $f$ has a local minimum at $p$ iff there exists $B_\delta(a)\cap {\rm dom}(f)$ such that $f(x)\geqslant f(p)$ over that ball. Therefore, $f({\bf 0})=0$ can not be a extreme value of $f$, then it is a saddle point.
