Second partial derivative test question I'm looking at the solution for finding maximum and minimum of 
$f(x,y) = xye^{-x² -y^2}$
To do so, it first finds the critical points using the second derivative, and these points are $(0,0),({\pm {1\over\sqrt[]{2}}},{1\over\sqrt[]{2}}),({\pm {1\over\sqrt[]{2}}},{-{1\over\sqrt[]{2}}}) $
By inspection it finds that
$
f(0,0)=0
$
$
f({\ {1\over\sqrt[]{2}}},{1\over\sqrt[]{2}})=({-{1\over\sqrt[]{2}}},-{1\over\sqrt[]{2}})={e^{-1}\over 2} > 0
$
$
f(-{{1\over\sqrt[]{2}}},{1\over\sqrt[]{2}})=({{1\over\sqrt[]{2}}},-{1\over\sqrt[]{2}})={-{e^{-1}\over 2}} < 0
$
And it reasons that ${e^{-1}\over 2}, -{e^{-1}\over 2}$ are maximum and minimum.
It also uses the Hessian matrix to find out about $(0,0)$.
The determinant of   $det(H(f)(0,0))=1$  but $f_{xx}=0$. 
It then claims that because because the matrix is undefined but it also claims that $(0,0)$ is a saddle point.
So I have two questions:
Is it correct that if  $det(H(f)(a,b)) > 0$ and $f_{xx}=0$ the second partial derivative test is inconclusive?
How can we claim that $(0,0)$ is a saddle point if the second derivative test shows that it is inconclusive? 
 A: 
Saddle point for sure. The Hessian of $f$ is:
$$
H_f = \begin{pmatrix}
2x(2x^2-3) y e^{-x^2-y^2} & (2x^2-1)(2y^2-1) e^{-x^2-y^2}
\\
(2x^2-1)(2y^2-1) e^{-x^2-y^2} & 2y(2y^2-3) x e^{-x^2-y^2} 
\end{pmatrix},
$$
thus
$$
H_f(0,0) = \begin{pmatrix}
0 & 1
\\
1 & 0
\end{pmatrix},
$$
and $\mathrm{det}(H_f(0,0)) = -1$ not $1$, which shows $(0,0)$ is a saddle point. 

Now to answer your first question. 
It is impossible for a smooth function that can have $\mathrm{det}(H_f(x_0,y_0)) > 0$ and $f_{xx}(x_0,y_0) = 0$ at $(x_0,y_0)$. 
Let's say if $f_{xx}(x_0,y_0) = 0$, then
$$
H_f(x_0,y_0) = \begin{pmatrix}
0 & f_{xy}(x_0,y_0)
\\
f_{yx}(x_0,y_0) & f_{yy}(x_0,y_0)
\end{pmatrix}.
$$
Unless you construct some special smooth functions that bear the property that $f_{xy} \neq f_{yx}$, the determinant $\mathrm{det}(H_f(x_0,y_0))  = - (f_{xy}(x_0,y_0))^2 \leq 0$. 

For the second question: when the test is inconclusive, 


*

*$f_{xx}$ and $f_{yy}$ have different signs, then we have a saddle point. You can look at the picture to see the geometric meaning of this (though in example they are zero): If our viewponint is somewhere on the $y$-axis and to observe the change in $x$, $f_{xx}>0$ means what we see is a concave up curve near the neighborhood of the point of interest $(x_0,y_0)$;  Moving our viewponint to somewhere on the $x$-axis and to observe the change in $y$, $f_{yy}<0$ means what we see is a concave down curve. Then, clearly at $(x_0,y_0)$ we have a saddle.

*$f_{xx}$ and $f_{yy}$ have the same signs, then we have a local maximum/minimum.
