Prove a dynamical system has at least one periodic orbit 
Question: Prove that the dynamical system 
  $$
\dot{x}=x(1-x^2-y^2)-y+\frac{1}{2}xy,
\\
\dot{y}=y(1-x^2-y^2)+x+y^2+2x^2,
$$
  where $(x,y) \in \mathbb{R}^2$, has at least one periodic orbit.

My Answer: I used $x=r\cos\theta$ and $y=r\sin\theta$ to write the system in polar coordinates. I got, 
$$
\dot{r}=r(1-r^2)+r^2\sin^3\theta+\frac{5}{2}r^2\cos^2\theta \sin\theta,
\\
\dot{\theta}=1+\frac{1}{2}r\cos\theta \sin^2\theta +2r\cos^3\theta.
$$
In previous examples we have looked at where $\dot{r}$ and $\dot{\theta}$ are greater than or less than $0$. But I am not sure how the $sine$ and $cosine$ components will affect $\dot{r}$ and $\dot{\theta}$. As $sine$ and $cosine$ oscillate do I only need to look at $\dot{r}=r(1-r^2)$, and $\dot{\theta}=1$ ?
 A: Using the Lyapunov function $V(x,y)=x^2+y^2$ one gets, similar to what you found,
$$
\dot V=\nabla Vf=2(x\dot x+y\dot y)=2V(1-V)+y(5x^2+2y^2)
$$
The cubic term is small against the quadratic term $2V=2(x^2+y^2)$ for $V\approx 0$ and it is small against the quartic term $-2V^2=-2(x^2+y^2)^2$ for large $V$. Thus the global behavior is defined by $\dot V=2V(1-V)$ which is repelling at the origin and attracting toward the origin for large $V$. By Poincaré-Bendixon a limit cycle exists if no other stationary points are present.
To quantify further this change with rising radius from repelling to attracting toward the origin, use that with careful application of geometric-quadratic mean inequalities resp. optimization of the term on the circles $x^2+y^2=V$ one finds
$$
|y(5x^2+2y^2)|\le 2V^{3/2}+3x^2|y| \le 2V^{3/2}+6(V/3)^{3/2}\le 4V^{3/2}
$$
so that
$$
2V(1-V-2\sqrt{V})\le\dot V\le 2V(1-V+2\sqrt{V})
\\\iff\\
2V(2-(1+\sqrt{V})^2)\le\dot V\le 2V(2-(\sqrt{V}-1)^2)
$$
The signs of the bounds then influence the sign of $\dot V$, using the upper bound $\dot V<0$ is guaranteed for $V> (1+\sqrt2)^2=3+2\sqrt3$, and using the lower bound $\dot V>0$ for $V\le(\sqrt2-1)^2=3-\sqrt8$. Thus there could be a limiting cycle in the (slightly larger) annulus $\frac13\le r\le 3$, as the vector field points inside the annulus on its boundary.
Let's see what numerical solutions look like in that annulus:

As can be seen, there are (inferred indirectly from the area that the depicted solutions avoid) a saddle point at $$x≈-0.57633933623069592370 ∧ y≈-0.2669188251569654917$$ and a sink at $$x≈-1.014456191487483836 ∧ y≈1.465883731297409543$$ inside that annulus (exact locations computed with WolframAlpha or the CAS of your choice). 
Which hints at the importance of the second condition for Poincaré-Bendixon, that there should not be any stationary points inside the annulus.
def f(u,t): x,y=u; a=1-x**2-y**2; return [a*x-y+0.5*x*y, a*y+x+y**2+2*x**2]
tspan = np.linspace(0,10,150);
units = [ np.array([ np.cos(u), np.sin(u) ]) for u in 2*np.pi/12*np.arange(0,12) ];
for k in range(1,10): plt.plot(k/3.0*np.cos(tspan),k/3.0*np.sin(tspan),lw=0.2)
for u0 in units: 
    sol = odeint(f, u0/3, tspan); x,y=sol.T; plt.plot(x,y,'b');
    sol = odeint(f, 3*u0, tspan); x,y=sol.T; plt.plot(x,y,'g');
plt.grid(); plt.show()

