Largest circle in basin of attraction of the origin. We're given dynamical system:
$$
\dot x = -x + y + x (x^2 + y^2)\\
\dot y = -y -2x + y (x^2 + y^2)
$$
Question is what's the largest constant $r_0$ s.t. circle $x^2+y^2 < r_0^2$ lies in the origins basin of attraction.
So far with relatively easy algebra I've got:
$$ \dot r  = \frac{r}{2}(-2-\sin(2 \phi)+2r^2) \\
   \dot \phi = -(1+\cos^2(\phi))
$$
Which immediately shows $r_0 \geq \sqrt{1/2}$. How to show that there is no better bound?
 A: (This is only a "numerical" answer to the question)
A basin of attraction can be found numerically using reverse-time (or backward) integration. If we choose some initial point and integrate backwards for a sufficient time, we can obtain the set of the system states that it had before it reached the selected initial point. So, if we choose sufficiently many initial points that are contained in the basin of attraction (i.e. sufficiently close to the attracting steady state) and integrate backwards, we can get some idea of how the basin of attraction looks like.
The following picture demonstrates the aprroximation of the basin of attraction of the system:

The blue curves fill the basin of attraction. The red circle is the largest circle that fits in the basin of attraction.
I have choosen 36 initial points on the circle of radius 0.05. Here is the Matlab code:
axes 
hold on
r= 0.05; % radius of the circle of initial points
rpm= @(t,x)[-x(1)+x(2)+x(1)*(x(1)^2+x(2)^2);...
-x(2)-2*x(1)+x(2)*(x(1)^2+x(2)^2)]; % the right part of the system
for fi= 0:pi/36:2*pi
        % notice the backward in time direction of integration
    [t,z]= ode45(rpm,15:-0.1:0,r*[cos(fi) sin(fi)]);
    plot(z(:,1),z(:,2),'b');
end
grid on
h= ezplot('x^2+y^2=0.847^2',[-2 2],[-1.5 1.5]); % draw the circle
h.LineWidth= 1.7;
h.Color= 'red';
axis equal

A: After solving the system
$$ 
\dot r  = \frac{r}{2}(-2-\sin(2 \phi)+2r^2) \\
\dot \phi = -(1+\cos^2(\phi))
$$
we have
$$
\left\{
\begin{array}{rcl}
 r & = & \frac{\sqrt{3} \sqrt{3-\cos \left(2 \sqrt{2} \left(t-2 c_1\right)\right)}}{\sqrt{3 e^{2 t} c_2-\cos \left(2 \sqrt{2} \left(t-2
   c_1\right)\right)+\sqrt{2} \sin \left(2 \sqrt{2} \left(t-2 c_1\right)\right)+9}} \\
\phi & = & -\tan ^{-1}\left(\sqrt{2} \tan \left(\sqrt{2} t-2 \sqrt{2} c_1\right)\right) \\
\end{array}
\right.
$$
Regarding the  $r$ behavior the line of maxima after considering $c_1 = c_2 = 0$ is given by
$$
r_{max} = \frac{\sqrt{3} \sqrt{3-\cos \left(2 \sqrt{2} t\right)}}{\sqrt{\sqrt{2} \sin \left(2 \sqrt{2} t\right)-\cos \left(2 \sqrt{2} t\right)+9}}
$$
with $\min r_{max} = \sqrt{1-\frac{1}{\sqrt{13}}}\approx 0.850088$
A: Since the limit cycle looks like an ellipse centered at the origin, we'll try to find a Lyapunov function $V = a x^2 + b x y + c y^2$ s.t. $\dot V = (A x^2 + B x y + C y^2 + D) (V - V_0)$. This only requires equating the coefficients of two polynomials in $x$ and $y$, yielding, up to a constant factor,
$$V = 8 x^2 - 2 x y + 5 y^2, \\
\dot V = 2 (x^2 + y^2) (V - 6).$$
$V = 6$ is the equation of the limit cycle; the radius of the maximal circle is the minor semiaxis of the ellipse.
