Multivariable second derivative test: what if $f_{xx}(a, b) = 0$?

My textbook introduces the second derivative test for multivariable functions to determine local extrema. However, it does not discuss the case when $f_{xx} = 0$, so I'm at a loss as to what conclusion I can form. Here's the problem:

Determine the local extrema for $f(x, y) = x^3 - 3x + 3xy^2$

My solution:

$f_x = 3x^2 - 3 + 3y^2 = 0$

$f_y = 6xy = 0$ [this is true if $x = 0$ or $y = 0$]

Using the conclusion above:

$x = 0: 3y^2 - 3 = 0$, so $y = \pm1$

$y = 0: 3x^2 - 3 = 0$, so $x = \pm1$

Therefore, the critical points are $(0, -1)$, $(0, 1)$, $(-1, 0)$, and $(1, 0)$

Second partial derivatives:

$f_{xx} = 6x$

$f_{yy} = 6x$

$f_{xy} = 6y$

Therefore:

$D(x, y) = 36x^2 - 36y^2$

But at point (0, -1), I get that $D > 0$ and $f_{xx} = 0$. The textbook only specifies conclusions for $f_{xx} > 0$ or $f_{xx} < 0$.

Since $D=f_{xx}f_{yy}-f_{xy}^2$, if $f_{xx}=0$ then $D=-f_{xy}^2\leq 0$, so that either the critical point is a saddle point or the test fails. And indeed in your case $D=-36<0$ at $(0,-1)$.
• Oh, I see, I made a typo in my solution. I should've gotten that $D < 0$, not $D > 0$. Thank you for clarifying. I'll select your answer once the time limit elapses. – AleksandrH Feb 15 '17 at 17:05