Maxima and minima of multivariable function $f(x,y)=6x^3y^2-x^4y^2-x^3y^3$ $$f(x,y)=6x^3y^2-x^4y^2-x^3y^3$$
$$\frac{\delta f}{\delta x}=18x^2y^2-4x^3y^2-3x^2y^3$$
$$\frac{\delta f}{\delta y}=12x^3y-2x^4y-3x^3y^2$$
Points, in which partial derivatives ar equal to 0 are: (3,2), (x,0), (0,y), x,y are any real numbers. Now I find second derivatives
$$\Delta_1=\frac{\delta f}{\delta x^2}=36xy^2-12x^2y^2-6xy^3$$
$$\frac{\delta f}{\delta y^2}=12x^3-2x^4-6x^3y$$
$$\frac{\delta f}{\delta x \delta y}=\frac{\delta f}{\delta y \delta x} = 36x^2y-8x^3y-9x^2y^2$$
$$\Delta_2=\begin{vmatrix}\frac{\partial^2 f}{\partial x^2}&\frac{\partial^2 f}{\partial y\partial x}\\\frac{\partial^2 f}{\partial x\partial y}& \frac{\partial^2 f}{\partial y^2} \end{vmatrix}$$
After plugging in the point (3,2) we get $\Delta_1<0$ and $\Delta_2>0$, so (3,2) is maxima. Now then I try to plug in (x,0) and (0,y) I obviously get $\Delta_1=0$ and $\Delta_2=0$ and I can't tell, using Sylvester's criterion, if those points are minima or maxima or neither. What should I do?
 A: You could try looking at further derivatives but generally in this case it's better to think of the function itself. Imagine you're at a point (0,y) for instance. How does f change when you move a little in the y-direction? How does f change when you move a little in the x-direction? 
A: If we go back to the function and its first and second derivatives, we see that they can be factored as
$$ f(x, \ y) \ = \ x^3 \ y^2 \ ( \ 6 \ - \ x \ - \ y \ ) \ \ , $$
$$ f_x \  = \ x^2 \ y^2 \ ( \ 18 \ - \ 4x \ - \ 3y \ ) \ \ , \ \ f_y \ = \ x^3 \ y \ ( \ 12 \ - \ 2x \ - \ 3y \ ) \ \ ,  $$
$$ f_{xx} \  = \ 6 \ x \ y^2 \ ( \ 6 \ - \ 2x \ - \ y \ ) \ \ , \ \ f_{yy} \ = \ 2 \ x^3  \ ( \ 6 \ - \ x \ - \ 3y \ ) \ \ , \ \ \text{and}  $$
$$ f_{xy} \  = \ x^2 \ y \ ( \ 36 \ - \ 8x \ - \ 9y \ ) \ \ .  $$
Because we are dealing with factors in the function which are power functions with exponents of three and higher, we can expect that "second derivative tests" may not be much help where the base of the power function becomes zero.  So we will need to look to the "behavior" of the function (and possibly its derivatives) in the vicinity, which is something of what john suggests.
If we look along the $ \ x-$ axis, that is, the set of points $ \ (c, \ 0) \ $ , we can investigate the behavior of the function  $ \ f(c, \ y) \ = \ c^3 \ y^2 \ ( \ 6 \ - \ c \ - \ y \ ) \ = \ (6 - c) \ c^3 \ y^2 \ - \ c^3 \ y^3 \ $ .  Close in to that axis, the quadratic term is dominant, so the function "behaves like" $ \ (6 - c) \ c^3 \ y^2 \ $ .  Hence, the surface described by $ \ f(x, \ y) \ $ has an "upward-opening" parabolic "cross-section" for $ \ 0 \ < \ x \ < \ 6 \ $ , and "downward-opening" for $ \ x \ > \ 6 \ $ and about the negative $ \ x-$ axis (the change at $ \ x \ = \ 6 \ $ comes of "crossing" the line $ \ x \ + \ y \ = \ 6 \ $ , on which the function is also zero) .  The graph below gives an idea of what this looks like.

views of the surface $ \ f(x, \ y) \ $ , "sighting" down the positive $ \ x-$ axis "short of" $ \ x \ = \ 6 \ $ , and a side-view of the negative $ \ x-$ axis portion
So the points on the $ \ x-$ axis other than $ \ (0, \ 0) \ $ and $ \ (6, \ 0) \ $ are either relative maxima or relative minima.
Performing a similar analysis near the $ \ y-$ axis, we have 
$$  f(x, \ c) \ = \ x^3 \ c^2 \ ( \ 6 \ - \ x \ - \ c \ ) \ = \ (6 - c) \ c^2 \ x^3 \ - \ c^2 \ y^4 \ \sim \ (6 - c) \ c^2 \ x^3 \ \ , $$
where the cubic term now dominates.  So the surface clearly has (close to) odd symmetry immediately about the $ \ y-$ axis everywhere except $ \ (0, \ 0) \ $ and $ \ (0, \ 6) \ $ , making all of those locations saddle points.

the view along the $ \ y-$ axis "short of" $ \ y \ = \ 6 \ $ , beyond which the direction of the "cubic cross-section" would be reversed
We have passed over the character of the origin and the points $ \ ( 0, \ 6) \ $ and $ \ (6, \ 0 ) \ $ , but we now have enough information to describe them as well.  For a surface representing a function of two variables, if the symmetry about a point is even and the "direction of concavity" is the same in both perpendicular directions, then the point will be a relative extremum; otherwise, it is a saddle point. (This can be extended to functions with more variables.)
We have found that the conditions for relative extrema are not met, so these three points are all saddle points.  A graph of the surface near the origin illustrates the situation:

We can make this a bit more evident by considering cross-sections through the surface along the "vertical" planes $ \ y \ = \ x \ $ and $ \ y \ = \ -x \ $ .  In these planes, we have the functions
$$ f(x, \ x) \ = \ 6 x^5 \ - \ 2  x^6 \ \ , \ \ f(x, \ -x) \ = \ 6 x^5 \ \ . $$
Both of these functions have odd (or nearly so) symmetry close to the origin, so the direction of concavity reverses on passing through the origin for both, producing a saddle point there.  With somewhat more work, we determine that there is similar behavior about $ \ ( 0, \ 6) \ $ and $ \ (6, \ 0 ) \ $ .
[Note that the function $ f(x, \ x) \ = \ 6 x^5 \ - \ 2  x^6 \ $ is zero at $ \ (3, \ 3) \ $ , which is a point on the "zero-line" $ \ x \ + \ y \ = \ 6 \ $ .  It has a maximum at $ \ (\frac{5}{2}, \ \frac{5}{2}) \ $ , along a "ridge" formed in the surface, which includes the relative maximum you found nearby at $ \ (3, \ 2) \ $ . ]
A: You just need to use the definition of maximum/minimum, together with the knowledge of the sign of $f$.
For instance, if you take a point of the form $(x,0), x<0$, you see that $f(x,0)=0$ and $f(x,y)<0$ for all neighbouring points. This means that points of the form $(x,0), x<0$ lead to local maxima.
As another example, you see that $f(0,0)=0$ and around that point $f$ always takes both positive and negative values. The origin is a saddle point.
Now you can use similar arguments for all other cases.

