Find the extrema of $f(x,y) = x^3\cdot y^3$ in $\mathbb{R}^2$ I am asked to find the extrema of
$$f(x,y) = x^3\cdot y^3$$
in $\mathbb{R}^2$
However, using the Hessian criteria, I get that the determinant of the Hessian matrix is zero for the two possible critical points: $(0,y)$ and $(x,0)$. Since it is zero, I can't know if it is a $\min$/$\max$ or a saddle point.
I don't know what method or what to do next. What method is there for when this happens? How can I determine wether they are $\min$ or $\max$?
Thanks!
 A: There are three techniques taught in Calculus for determining whether a critical point is a (local or global) extremum : the second derivative test, the first derivative test, and just evaluation.
Here, we find $\frac{\partial f}{\partial x} = 3x^2y^3$, so some critical points are $\{(0,y) \mid y \in \Bbb{R}\}$.  Also, $\frac{\partial f}{\partial y} = 3x^3y^2$, so more critical points are $\{(x,0) \mid x \in \Bbb{R}\}$.  Since both partials are defined everywhere, these are all the critical points.
If we try the equivalent of the second derivative test, we find that the Hessian is zero on both axes, which does not resolve the type of any of these critical points.
If we try the equivalent of the first derivative test, the critical points partition the plane into four quadrants.  So we look at the gradient
$$  \nabla f(x,y) = 3x^2y^2 (y,x)  $$
in each quadrant:

*

*QI, the gradient points toward the line $y = x$.  On that line, $f(x,x) = x^6$, which is unbounded above.

*QII, the gradient points toward the line $y = -x$.  On that line, $f(x,-x) = -x^6$, which is unbounded below.

*QIII has the same behaviour as QI.

*QIV has the same behaviour as QII.
So by what amounts to the first derivative test, we find that $f$ is unbounded above and below.

The third option in Calculus is to evaluate at the critical points.  Of course, $f = 0$ at all of these critical points.  To determine if any are local extrema, we need to look at small neighborhoods to see if the critical points give values of $f$ larger or smaller than so the neighbors.  Let $\varepsilon$ have small magnitude.  Then
$$  f(\varepsilon, y) = \varepsilon^3 y^3  $$
which can be both positive and negative by allowing the sign of $\varepsilon$ to vary, so $(0,y)$ critical points are not extrema.
$$  f(x,\varepsilon) = \varepsilon^3 x^3  $$
and a similar analysis shows the $(x,0)$ critical points are not extrema.  Therefore, none of the critical points are (even local) maxima or minima.
A way to go that is unrelated to Calculus of a single variable is to make a change of variables.  Let $u = xy$ and $v$ be some orthogonal coordinate.  Then $f(u,v) = u^3$ and single variable calculus will quickly show that this function increases without bound as $u \rightarrow \infty$ and decreases without bound as $u \rightarrow -\infty$.  So $f$ has no extrema.
