How to properly find a critical point for $xy^2$? Let $f(x,y) = xy^2$ and the domain $D = \lbrace (x,y)| x,y\geq0, x^2 + y^2 \leq 3 \rbrace$
$f_x(x,y) = y^2$ and $f_y(x,y) = 2xy$ 
Therefore, the critical points should be $\lbrace (x,y)| y = 0, \sqrt{3} \geq x \geq 0 \rbrace$. 
The determinant of the Hessian is 
$$\det(Hf(x,y))=
\begin{vmatrix}
0 & 2y \\ 
2y & 2x
\end{vmatrix} = 0-4y^2$$
But this doesn't make sense to me because, if $y = 0$, then $f(x,0) = 0$. But on this interval, suppose I chose $(x,y) = (1,\sqrt{2})$, this would be larger and would be the maximum value of the system.  
Why didn't finding the first partials and the Hessian allow me to compute the critical point? 
 A: Critical points are candidates for the max and min values. 

Boundary points are also candidates.

But for the specified domain $D$, the only possible critical points are the points where $y=0$, and those points are on the boundary, hence are not actually critical points.

It follows that the maximum and minimum values of $xy^2$ must be on the boundary.

For the minimum, it's obvious that the minimum value is $0$, which occurs at all points on the boundary where $x=0\;$or $y=0$.

For the maximum, it's obvious that the maximum is positive, hence the only candidates are boundary points where $x,y > 0.\;$It follows that the maximum occurs on the circular part of the boundary, excluding the two endpoints. 

Thus, letting $(x,y)=\bigl(\sqrt{3}\cos(t),\sqrt{3}\sin(t)\bigr)$, we want to maximize $\bigl(\sqrt{3}\cos(t)\bigr)\bigl(3\sin^2(t)\bigr)$, for $0 < t < {\large{\frac{\pi}{2}}}.\;$Then letting $g(t)=\bigl(\sqrt{3}\cos(t)\bigr)\bigl(3\sin^2(t)\bigr)$ and setting $g'(t)=0$, we get only one critical point for $0 < t < {\large{\frac{\pi}{2}}}$, namely $t=\cos^{-1}\left({\large{\frac{1}{\sqrt{3}}}}\right)$, which yields the maximum value for $f$ of $2$.

Alternatively, now that I see saulspatz's comment, since the circle has the equation $x^2+y^2=3$, maximizing $xy^2$ on the circular arc of the boundary, excluding the endpoints, is the same as maximizing the function $x(3-x^2)$ on the interval $0 < x < \sqrt{3}$. Taking the derivative and setting it to $0$, we get only one critical in the interval $0 < x < \sqrt{3}$, namely $x=1$, which yields the maximum value for $f$ of $2$.
A: On the circle of radius $r$, we have $x^2+y^2=r^2$. Therefore,
$$
xy^2=r^2x-x^3\tag1
$$
This implies that the interior critical points are at
$$
(x,y)=\frac r{\sqrt3}\left(\pm1,\pm\sqrt2\right)\tag2
$$
with the corresponding values of
$$
xy^2=\pm r^3\frac2{3\sqrt3}\tag3
$$
At the endponts of $x=\pm r$, we get the values of $xy^2=0$. Therefore,
$$
-r^3\frac2{3\sqrt3}\le xy^2\le r^3\frac2{3\sqrt3}\tag4
$$
Thus, the maxima and minima are on the bounding circle.
If $x^2+y^2=3$, then $r=\sqrt3$ and so $(4)$ says that
$$
-2\le xy^2\le2\tag5
$$
