Study extreme values of functions of several variables. Well, I have to solve the following problem:
$\textit{Study the extreme values of}$ $f(x,y)=x^2+y^2+\alpha x^3y^3$ $\textit{depending on}$ $\alpha$, $where $ $\alpha\in \mathbb{R}$.
It's easy to find that the extreme values of the function must verify that
$$\left\{ 
\begin{array}{ll}
2x+3\alpha x^2y^3 =0\\
2y+3\alpha x^3y^2=0
\end{array}
\right.$$
so this extreme values could be


*

*If $\alpha=0$, only $(0,0)$.

*If $\alpha>0$, $(0,0),(\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}})$.

*If $\alpha<0$, $(0,0),(\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}})$.


Now, we can prove that $(0,0)$ is a relative minimum. Applying the Sylvester theorem, I also proved that if $\alpha>0$, $(\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}})$ are saddle points. My problem is to determine what are $(\sqrt[4]{\frac{2}{3\alpha}},\sqrt[4]{\frac{2}{3\alpha}}),(-\sqrt[4]{\frac{2}{3\alpha}},-\sqrt[4]{\frac{2}{3\alpha}})$ for $\alpha<0$, whose hessian matrix is semidefinite positive. 
 A: For the case $\alpha<0$, let $t=\sqrt[4]\frac{-2}{3\alpha}$, so the critical points are given by $(0,0), (t,t), \text{ and } (-t,-t)$.
As remarked above, there is a relative minimum at $(0,0)$, 
and there are saddle points at $(t,t)$ and $(-t,-t)$ since
$\;\;\;f_{xx}=2+6\alpha xy^3=2+6\alpha\big(\frac{-2}{3\alpha}\big)=-2$,
$\;\;\;f_{yy}=2+6\alpha x^3y=2+6\alpha\big(\frac{-2}{3\alpha}\big)=-2$,
$\;\;\;f_{xy}=9\alpha x^2y^2=9\alpha\big(\frac{-2}{3\alpha}\big)=-6$, and
$\;\;\;D=f_{xx}f_{yy}-f_{xy}^2=4-36=-32<0$.
A: The only thing I can think of is restricting the function to lines through those points, computing the third derivatives of the restrictions and studying those. If they are nonzero along some direction, then it's neither a minimum nor a maximum, since it has an inflection point along that direction. If not, you move on to fourth derivatives. I am, at present, unable to find it, but I remember some page where the criterion for classifying zero-derivative points with higher derivatives was described in detail. IIRC, if the first nonzero derivative is of odd order, no max no min, if it is even-order, it's max or min, depending on the sign.
I agree that this is not particularly practical, seen as the third derivative is this mess, so hopefully someone else will come up with a better strategy in another answer. :)
