Determine the absolute and relative extreme for the function $f(x,y)=xy(1-x^2-y^2)$ in the square $0\leq x\leq 1,\, 0\leq y\leq 1$ Determine the absolute and relative extreme for the function $f(x,y)=xy(1-x^2-y^2)$ in the square $0\leq x\leq 1,\, 0\leq y\leq 1$
 My work: 
Let $f:\mathbb{R}^2\rightarrow\mathbb{R}$ defined by $f(x,y)=xy(1-x^2-y^2)$ and $g:\mathbb{R}^2\rightarrow\mathbb{R}$ defined by
Let $L$ the lagrange function defined by
$L(x,y,\alpha)=f(x,y)+\alpha(g(x,y))$
Then we have the system:
$\begin{equation}
L_x=f'(x,y)+\alpha(g'(x,y))=0 \text{  (Derivative with respect x)}\\
L_x=f'(x,y)+\alpha(g'(x,y))=0 \text{  (Derivative with respect y)}\\
L_\alpha=g(x,y)=0
\end{equation}$ 
I have a problem when i go to define the $g$ function. Can someone explain me how define correctly this function?
 A: You cannot apply Lagrange multipliers here, because the method requires the functions to have continuous partial derivatives; and if you write the square as a curve, it will not be differentiable at the vertices. 
Lagrange multipliers works when you want to find extrema of a function on a differentiable curve. Even if the boundary curve were differentiable (which it isn't, in your case), you still need to address the extrema inside the square, and Lagrange Multipliers are useless there. 
For a case like yours, what one does is find the extrema in the interior (by just looking at the partial derivatives), and then look at the boundary. 
For the interior, you look at the partial derivatives and equate them to zero, 
$$
0=f_x=(xy-x^3y-xy^3)_x=y-3x^2y-y^3,
$$
$$
0=f_y=(xy-x^3y-xy^3)_y=x-x^3-3xy^2.
$$
Since we are looking at the interior of the square, $xy\ne0$. So the above equalities reduce to 
$$
1-3x^2-y^2=0,\  \ \ 1-x^2-3y^2=0.
$$
One quickly finds that $x^2=y^2=1/4$. As $x,y>0$, we get $x=y=1/2$. To analyze it, we need to look at the second derivatives:
$$
f_{xx}=-6xy,\ \ f_{yy}=-6xy,\ \ f_{xy}=1-3x^2-3y^2.
$$
Thus
$$
f_{xx}(1/2,1/2)f_{yy}(1/2,1/2)-f_{xy}(1/2,1/2)^2
=(-3/2)(-3/2)-(1-3/2)^2=9/4-1/4=2.
$$
As $f_{xx}(1/2,1/2)<0$, we conclude that $(1/2,1/2)$ is a local maximum. 
On the boundary, we need to go through each of the four sides of the square:


*

*$y=0$:  $f(x,0)=0$.

*$x=0$: $f(0,y)=0$.

*$y=1$: $f(x,1)=-x^3$, so the maximum occurs at $(0,1)$, the minimum at $(1,1)$. 

*$x=1$: $f(1,y)=-y^3$, so the maximum occurs at $(1,0)$, the minimum at $(1,1)$. 
In summary, the critical points are $(x,0)$, $(0,y)$, $(1,1)$, $(1/2, 1/2)$. 
Values:
$$
f(x,0)=0,\ \ f(0,y)=0, \ \ f(1,1)=-1,\ \ f(1/2,1/2)=1/8.
$$
So we have the absolute minimum of $-1$ at $(1,1)$, the absolute maximum of $1/8$ at $(1/2,1/2)$, and local mimima on the two edges $x=0$ and $y=0$. 
