A basic doubt on local minima when first and second derivative test fails How to check whether the functions $f(x,y)=x^2y^2$ and $f(x,y)=x^2y^3$ has local minima at $(0,0)$. Actually, the problem is the first and second derivative both reaches to zero at $(0,0)$. So, Hessian is positive semi-definite. How do I check then? 
 A: Hints: $x^2y^2 \ge 0$ and $\operatorname{sign} y^2y^3 = \operatorname{sign} y$.
A: The problem with using the Hessian matrix is that it is only a higher-dimensional analogue of the "second-derivative test" in single-variable calculus.  So it is not going to be of much help in sorting out the nature of a critical point $ \ (a, \ b, \ c, \ \ldots) \ $ for a polynomial function $ \ C \ (x-a)^p \ (y-b)^q \ (z - c)^r \ \cdot \ \ldots \ $ , with $ \ p , \ q , \ r \ \ldots \ \ge \ 2 \ $ and $ \ C \ $ some non-zero real number.
As Christian Blatter remarks, one is left to investigate the behavior of the function/surface in the vicinity of the critical point by some other means.  In single-variable calculus, when confronted with a point for which $ \ f'(a) \ = \ 0 \ $ and $ \ f''(a) \ = \ 0 \ $ , we can whether the "concavity" of the function (sign of $ \ f'' (a) \ $ ) changes for values of  $ \ x \ $ at values slightly smaller or larger than $ \ a \ $ .  For functions of more than one variable, we can try taking "slices" or "cross-sections" of the function/surface which pass through the critical point to see what might be determined.
For your functions, "vertical planes" containing the coordinate axes ( $ \ x = 0  \ $ or $ \ y = 0 \ $ ) won't allow us to draw any conclusions, since their values are zero everywhere in those planes (which is what is causing the difficulty with the Hessian matrix).  However, other vertical planes $ \ y \ = \ kx \ $ , $ \ k \ \neq \ 0 \ $ will reveal "cross-sectional functions" $  \ z \ = \ x^2 \ ( kx )^2 \ = \ k^2 \ x^4  \ $ and $  \ z \ = \ x^2 \ ( kx )^3 \ = \ k^3 \ x^5  \ $ , which do tells us the the origin $ \ (0, \ 0) \ $ is a local (and, in fact, absolute) minimum for $  \ z \ = \ x^2 \ y^2 \ $ and a saddle point for $  \ z \ = \ x^2 \ y^3   \ $ .
We can extend this pretty directly to say that the critical point $ \ (a, \ b) \ $ is a local extremum for $ \ C \ (x-a)^p \ (y-b)^q \  $ , with integers $ \ p , \ q  \ \ge \ 2 \  \ , \ \ C \ \neq \ 0 \ $ when $ \ p \ $ and $ \ q \ $ have the same parity  ( $ \ p + q \ $ even) and is a saddle point when they have opposite parity ( $ \ p + q \ $ odd).  Analogous (though more complicated) statements can be made for polynomial functions of more variables.
