Second derivative test inconclusive I have the function $f(x,y) = x^2y(4-x-y)$ and $T = \{(x,y) | x\ge 0, y \ge 0, x+y \le 6\}$. With the second derivative test I have found that $(2, 1)$ is a relative maximum.
By calculating the first derivatives you will see that $x=0$ and $y \in [0, 6]$ are critical points (correct me if I am wrong). So by using those numbers, the second derivative test is inconclusive.
My questions are: What can I do if the second derivative test is inconclusive? Does this function have a relative minimum?
 A: The Weiertrass theorem states that, if $f:X\to \Bbb{R}$ is a continuous function and $X$ is a compact set, then there exists $x_1, x_2 \in X$ such that $$f(x_1)\leq f(x) \leq f(x_2) \; \forall x\in X.$$ We say that $x_1$ and $x_2$ are absolute minimum and maximum points. Note that every absolute minimum (maximum) point is a relative minimum (maximum) point. So, here, you have the existence of such point. To find it, you have to look at the bound of your $T$ (if there is no minimum points in $int(T)$, then they are in $\partial T$).
A: You may use the second derivative only on the interior of $T$. Since $T$ is compact $f$ must have global extreme values.
Calculate the extrema in the interior of $T$, that is $x,y>0$ and $x+y<6$, first.  You'll get only one candidate, namely $(2,1)$. The Hessian will reveal that this is a maximum with value $f(2,1)=4$.  
Now consider the boundary.  Since $T$ is compact we expect extrema on its boundary.  So if $xy=0$ we have $f(x,y)=0$. If $y=6-x$ we're looking for the extreme values of $x^2(6-x)(-2)$.  Candidates are $x=0$ (see above) and $x=4$.  Show that we have a minimum in $x=4$ (hence $y=2$) with value $-64$. 
So $f$ attaines its global maximum in $(2,1)$ and its global minimum in $(4,2)$.
