2nd derivative test fail I trying to solve this problem in Advanced Calc by Buck, sec 3.6 problem 9: 
Let $f(x,y)=(y-x^{2})(y-2x^{2})$.
Show that the origin is a critical point for $f$ which is a saddle
point, although on any line through the origin, $f$ has a local minimum
at $(0,0)$.
Solution:
We have
\begin{align*}
f(x,y) & =(y-x^{2})(y-2x^{2})\\
 & =y^{2}-3x^{2}y+2x^{4}
\end{align*}
so that
\begin{align*}
f_{1}(x,y) & =-6xy+8x^{3} &  & (1)\\
f_{2}(x,y) & =2y-3x^{2} &  & (2)
\end{align*}
set (1) and (2) equal to zero and solve: 
from (2) 
$2y-3x^{2}=0\,\Longrightarrow y=\frac{3}{2}x^{2}$
in (1) 
$-6xy+8x^{3}=0\,\Longrightarrow-9x^{3}+8x^{3}=0\,\Longrightarrow x=0,\Longrightarrow y=0$
hence (0,0) is a critical point of $f$.
(0,0) is a saddle point:
By Theorem 19 page 157: If the determinant of the hessian (2nd order
partial derivatives matrix) is negative at $p_{0}$, then $p_{0}$is
a saddle point.
hence
$H=\left[\begin{array}{cc}
f_{11} & f_{12}\\
f_{21} & f_{22}
\end{array}\right]=\left[\begin{array}{cc}
-6y+24x^{2} & -6x\\
-6x & 2
\end{array}\right]$ 
$\Delta=-12y+48x^{2}-36x^{2}=12(x^{2}-y)$
at (0,0) $\Delta=0$, therefore the second derivative test is inconclusive.
However since $f(x,y)=(y-x^{2})(y-2x^{2})$, then if $x^{2}<y<2x^{2}$
we have $f(x,y)<0$, also if ($y<x^{2}$ or $y>2x^{2}$) we have $f(x,y)>0$,
and we have $f(x,y)=0$ if $x=y=0$. Therefore the critical point
(0,0) is a saddle point.
I don't feel confident about the proof of (0,0) is a saddle point. Also I am not sure how to do the last part (any line through the origin, $f$ has a local minimum
at $(0,0)$) any help appreciated.
 A: Yout proof that $(0,0)$ is a  saddle point is correct. Lines should be understood as straight lines, hence, if $y=cx$, where $c\neq0$, 
$$
f(x,cx)=(cx-x^2)(cx-2x^2)=x^2(c-x)(c-2x)>0 
$$
for $x\neq0$, but near 0. The cases $c=0$ and $x\equiv0$ are obvious.
A: This reply is intended to illustrate the situation presented by this function.  As you found the Hessian determinant function to be $ \ \mathbf{H_f} \ = \ 12·(x^2-y) \ , $ the value of the determinant is not only zero at the origin, but at all points on the parabola $ \ y \ = \ x^2 \ . $  Some authors refer to a critical point with zero determinant as "degenerate"; these seem often to occur where the critical point lies on a line or curve where $ \ \mathbf{H_f} \ = \ 0 \   $ everywhere.  So we must look at properties of the function (and sometimes the first partial derivatives as well) to understand better what is occurring at that critical point, which you did.
These graphs may help to give an idea of what the surface for $  \ f(x,y) \ = \ (y-x^{2})·(y-2x^{2})\ $ is like in the neighborhood of the origin.  You found the first partial derivatives to be zero for
$$ f_{x}(x,y) \  = \ 8x^3 - 6xy \ = \ 2x·(4x^2-3y) \ = \ 0 \ \ , $$
which gives the union of the parabola $ \ y \ = \ \frac43 x^2 \ $ and the $ \ y-$ axis [marked in blue in the first graph], and
$$ f_{y}(x,y) \  = \ 2y-3x^{2} \ = \ 0 \ \ , $$
yielding the parabola $ \ y \ = \ \frac32 x^2 \   $ [in green].  The intersection of all three curves is the single critical point $ \ ( 0 , 0 ) \ , $ as you found.
We find that the first partials have a complicated behavior in the vicinity of the point.  For $ \ x > 0 \ , $ both are positive "above" $ \ y \ = \ \frac32 x^2 \ , $  both negative "below" $ \ y \ = \ \frac43 x^2 \ , $ with the sign of $ \ f_y \ $ changing from positive to negative in the "sliver" between the two parabolas.  To the "left" of the $ \ y-$ axis [$ x < 0 $ ] , however, the sign of $ \ f_x \ $ is reversed from what it is in the "right" half-plane.  This is not at all what one expects near a local extremum.

This second graph of $  \ f(x,y) \ $ is pertinent to the proof you gave for the origin being a saddle point:  the function is positive in the region marked in green, which is everywhere except the parabolic "sliver" in red, where the function is negative.  A local extremum would appear surrounded by a region of only one color, so we see that the origin is a saddle point.
In regard to the demonstration provided by Przemysław Scherwentke, we observe that the function has been contrived so that very close to the origin, no straight line through the origin will be found in the region in red.  Along each line, the intersection of the function surface with a plane $ \ y = mx \ $ is a quartic curve $ \ z = 2x^4 - 3mx^3 + m^2x^2 \ $ , which has a local minimum at $ \ x = 0 \ $ for all values of $ \ m \ $ .

