Nature of the critical point $(0,0)$ of the function $f(x,y)=x^6-2x^2y-x^4y+2y^2$ Consider the function $$f(x,y)=x^6-2x^2y-x^4y+2y^2.$$ The point $(0,0)$ is a critical point. Observe, 
\begin{align*}
f_x & = 6x^5-4xy-4x^3y, f_x(0,0)=0\\ 
f_y & = 2x^2-x^4+4y. f_y(0,0)=0\\
f_{xx} & = 30x^4-4y-12x^2y, f_{xx}(0,0)=0\\
f_{xy} & = 4x-4x^3, f_{xy}(0,0)=0\\
f_{yy} & = 4, f_{yy}=4
\end{align*}
So, in order to determine the nature of the above critical point, we need to check the Hessian at $(0,0)$ which is $0$ and hence the test is inconclusive. $$ H(x,y)= \det \begin{pmatrix} f_{xx} & f_{xy}\\ f_{yx} & f_{yy} \end{pmatrix}=\det \begin{pmatrix} 0 & 0 \\ 0 & 4 \end{pmatrix}=0$$So, I tried to see the function on slices like $y=0$ and $y=x$ but nothing worked. So please suggest me how do I find the nature of the critical point in this case?
 A: You have that
$$
g_a(x)=f(x,ax^2 ) = 2\left( {a^2  - a} \right)x^4  + \left( {1 - a} \right)x^6 
$$
With $0<a<1$ the function $g_a(x)$ has a local maximum. With $a>1$ the function has a local minimum. This means that $(0,0)$ is a saddle point.
A: You might note that your function factors as
$$(x^2-y)(x^4-2y).$$
So there are easy to find regions in the $xy$-plane where the function is positive an negative.  Close to the origin and between the curves $y=x^2$ and $y=x^4/2$, the function is negative.  This suggests trying the limit along the curve $y=x^3$.  It's not too horrible to analyse 
$$f(x,x^3) = 3x^6-2x^5 -x^7$$
around $x=0$ to see that it's negative there.  
Comparing with the curve given by $y=0$, we get a saddle point at $(0,0).$
A: With $g(x) = x^2$ we have
$$
f(g(x),y) = g(x)^3-2g(x)y-g(x)^2y +2y^2
$$
or
$$
f(g,y)=g^3-2g y-g^2y+2y^2
$$
and the hessian of $f(g,y)$ is
$$
H(g,y) = \left(
\begin{array}{cc}
 6 g-2 y & -2 g-2 \\
 -2 g-2 & 4 \\
\end{array}
\right)
$$
and also
$$
H(0,0) = \left(
\begin{array}{cc}
 0 & -2 \\
 -2 & 4 \\
\end{array}
\right)
$$
with eigenvalues 
$$
\left\{2 \left(1+\sqrt{2}\right),2 \left(1-\sqrt{2}\right)\right\}
$$
characterizing a saddle point.
A: You have $f(0,0)=0$. You can find both positive and negative values of $f(x,y)$ in any region around $(0,0)$, which means that you don't have a local extremum at this point:
$$f(x,x^3)= -x^5(1-x)(2-x)<0\quad\text{for}\,\, |x|<1, x\ne 0$$
and
$$f(x,0)=x^6>0\quad\text{for}\,\, x\ne 0 $$
