How to apply Lagrange Multipliers with modulus ?? 
I had the question $$x^2+y^2+xy=1$$ and I had to find maximum value of $$\vert xy(x^2+y^2) \vert$$

I tried using Lagrange multipliers by forming
$x^3+3xy^2 = \lambda (2y+x)$
And $y^3+3yx^2 = \lambda (2x+y)$
Now I subtract it to get $(x-y)^3 = \lambda (y-x)$
And as $\lambda ≠ 0 , x=y.$ This gives$ x = {1 \over \sqrt{3}}$ and the expression gives max as $\frac{2}{9}$ but the answer is $2$ . What have I done wrong.
I know it's modulus , but with negative too, there will only be a change in the sign of $\lambda$ which will not affect any working.
Also Lagrange Multipliers is something I read as an extra piece of info online, so maybe I might be missing something trivial here.
Also is there any requirement on $\lambda$ to be positive ??
 A: Find the maximum $M$ and the minimum $m$ of $xy(x^2+y^2)$ on that region. Then:


*

*If $m\geqslant0$, the maximum of $\bigl\lvert xy(x^2+y^2)\bigr\rvert$ is $M$.

*If $M\leqslant0$, the maximum of $\bigl\lvert xy(x^2+y^2)\bigr\rvert$ is $-m$.

*Otherwise, the maximum of $\bigl\lvert xy(x^2+y^2)\bigr\rvert$ is $\max\bigl\{\lvert M\rvert,\lvert m\rvert\bigr\}$.


Concerning your approach, it seems that you tried to find the maximum of $\bigl\lvert xy(x^2+y^2)\bigr\rvert^2$, which is equal to $x^2y^2(x^2+y^2)^2$. It works, but then the system that you will have to solve will be$$\left\{\begin{array}{l}2xy^2\left(3x^4+4x^2y^2+y^4\right)=\lambda(2x+y)\\2xy^2\left(3x^4+4x^2y^2+y^4\right)=\lambda(x+2y)\\x^2+y^2+xy=1,\end{array}\right.$$instead of the one that you mentioned. It can be done, but it leads to tougher computations.
A: Hint.
Following the fact
$$
\max |f| = \max(-f,f) \ \ \text{and}\ \ \min |f| = \min(-f,f)
$$
With $f = x y(x^2+y^2)$ the lagrangian can be established as
$$
L(x,y,\lambda) = s f+\lambda(x^2+y^2+x y -1)
$$
where $s=\pm$
The stationary points are given by
$$
\nabla L = 0 = \left\{
\begin{array}{rcl}
 2 s y x^2+\lambda  (2 x+y)+s y \left(x^2+y^2\right)&=&0 \\
 2 s x y^2+\lambda  (x+2 y)+s x \left(x^2+y^2\right)&=&0 \\
 x^2+y x+y^2-1 &=&0\\
\end{array}
\right.
$$
and solving we have
$$
\left[
\begin{array}{ccc}
s f & x & y \\
 -2 s & -1 & 1 \\
 -2 s & 1 & -1 \\
 \frac{2 s}{9} & -\frac{1}{\sqrt{3}} & -\frac{1}{\sqrt{3}} \\
 \frac{2 s}{9} & \frac{1}{\sqrt{3}} & \frac{1}{\sqrt{3}} \\
\end{array}
\right]
$$
NOTE
Due to its homogeneity the problem, using $y = \mu x$, can be transformed into one without restrictions as
$$
f(\mu) = \frac{\mu(1+\mu^2)}{(1+\mu+\mu^2)^2}
$$
A: Here is an alternative and geometric way of examining the problem:
$$x^2+y^2+xy=1$$ 
Is the equation of an ellipse whose major axis is supported by the line $y=-x$.
Note that $xy(x^2+y^2)=(1-x^2-y^2)(x^2+y^2)=k$ solves to $$x^2+y^2=\dfrac{1\pm\sqrt{1-4k}}2=\phi_{\pm}(k)$$

This is a circle of radius $\phi_{\pm}(k)^{\frac 12}$.


*

*Purple circle $\phi_-(k)$ for $k\in[0,\frac 14]$
is not interesting because it gives a circle which is strictly inside the ellipsis (max radius $\frac 1{\sqrt{2}}$ is when $k=\frac 14$ which is smaller than minor axis of ellipse).


*

*Green circle on the other hand $\phi_+(k)$ intersects with the ellipse


and gives a growing circle when $k\to-\infty$.
Make $k$ vary in this simulation: https://www.desmos.com/calculator/41c7utwiy3
The maximum value of $|xy(x^2+y^2)|=-k$ is obtained for the circumcircle of the ellipse, whose radius is given by the major axis extension.
On the major axis $y=-x$ and it is easy to find that $x=\pm 1$ and $k=-2$.
A: You found by subtracting 
$$
(x-y)^3=λ(y-x).
$$
In the same way you can add the two equations to get
$$
(x+y)^3=3λ(x+y)
$$
As @saulspatz commented, you got the cases mixed up,  $λ\ne0$ does not force $x=y$. Correct is


*

*if $λ=0$ then $x-y=0=x+y$, which has only the trivial solution $x=y=0$ with objective value $0$.

*if $λ\ne0$ then $(x-y)^2=-λ$ or $x=y$ and also either $(x+y)^2=3λ$ or $x+y=0$. As a square of a real number is always positive, only the crossed variants give new extremal points.


*

*$x=y$: the constraint reduces to $3x^2=1$, which results in $x=y=\pm\frac1{\sqrt3}$, which you already found with objective value $|xy(x^2+y^2)|=\frac29$.

*$x=-y$: the constraint reduces to $x^2=1$, which results in $x=-y=\pm1$ with objective value $|xy(x^2+y^2)|=2$.


A: It is trivial to eliminate $\lambda$ from the vector equation
$$
(x^3+3xy^2,y^3+3yx^2)=\lambda(2x+y,x+2y)
$$
by calculating it as a ratio in the two obvious ways, and equating them. The result is the equation
$$
(x^3+2xy^2)(x+2y)=(y^3+3yx^2)(2x+y).
$$
As Cesareo pointed out, this is homogeneous, and solving it gives
$$
x=y,\quad x=-y,\quad y=(2-\sqrt3)x,\quad y=(2+\sqrt3)x
$$
as possibilities. In other words, four lines through the origin. I trust you to be able to find the intersection of these lines and the ellipse $x^2+xy+y^2=1$, list the points, and find the maximum.
