Show that a system has a period solution by finding a trapping region (Poincaré-Bendixson Theorem) \begin{align*}
\dot{x}&=4x+2y-x(x^2+y^2)\\
\dot{y}&=-2x+y-y(x^2+y^2)
\end{align*}
I want to show that this system has at least one periodic solution by constructing a trapping region where the Poincaré-Bendixson theorem can be applied.
So far I've converted the system to polar coordinates and got:
\begin{align*}
\dot{r}&=-\frac{1}{2}r(-5+2r^2-3\cos(2\theta))\\
\dot{\theta}&=-r(2+3\cos(\theta)\sin(\theta)).
\end{align*}
Where I'm lost now is constructing the trapping region where $\dot{r}<0$ on the outside and $\dot{r}>0$ on the inside. Graphing this system using streamplot the region is visually clear, but I'm having trouble finding a closed form solution.

 A: There is a small sign error in the trigonometric term in your solution for $\dot{r}$. A complete solution follows the sake future readers.
Problem statement
Is there a periodic solution for the following dynamical system?
$$
%
\begin{align}
%
 \dot{x} &= 4 x+2 y - x\left(x^2+y^2\right)\\
%
 \dot{y} &= -2 x+y-y \left(x^2+y^2\right)
%
\tag{1}
\end{align}
%
$$
Solution method
Use the theorem of Poincare and Bendixson to identify a trapping region, here the gray annulus where the sign of the radial time derivative can change.

The invariant region must

*

*Be closed and bounded,

*Not contain any critical points.

Solution
Identify critical points
At what points
$
\left[
\begin{array}{c}
 x \\
 y \\
\end{array}
\right]
$ does
$
\left[
\begin{array}{c}
 \dot{x} \\
 \dot{y} \\
\end{array}
\right]
=
\left[
\begin{array}{c}
 0 \\
 0 \\
\end{array}
\right]
$?
The only critical point is the origin.
Switch to polar coordinates
The workhorse formula is
$$
%
\begin{align}
%
 x &= r \cos \theta, \\
%
 y &= r \sin \theta.
%
\end{align}
%
$$
With $r^{2} = x^{2} + y^{2}$, use implicit differentiation to find
$$
 r\dot{r} = x \dot{x} + y \dot{y}
\tag{2}
$$
Compute $\dot{r}$
Substituting into $(2)$ using $(1)$, and noting $\cos^{2} \theta = \frac{1}{2} \left( 1 + \cos 2\theta \right)$,
$$
%
\begin{align}
%
 r \dot{r} &= x \dot{x} + y \dot{y} \\
%
 &= r^{2} - r^{4} \color{blue}{+} 3x^{2} \\
%
 &= r^{2} + \frac{3}{2} r^{2} \left( 1 \color{blue}{+} \cos 2\theta \right) - r^{4} 
%
\end{align}
%
$$
Therefore
$$
  \dot{r} = -r^{3} + \frac{r}{2} \left( 5 \color{blue}{+} 3 \cos 2\theta \right)
\tag{3}
$$
Classify $\dot{r}$
Look for regions where the flow is outward $\dot{r}>0$, and regions where the flow is inward $\dot{r}<0$.
(Note the interesting comment by @Evgeny.)
Classify the problem by examining the limiting cases of  $\cos 2\theta$ at $\pm 1$
Outward flow: $\cos 2 \theta \ge -1$
$$
%
\begin{align}
%
 \dot{r}_{in} &= -r^{3} + \frac{r}{2} \left( 5 + 3 (-1) \right) \\
%
&= r(1-r^{2})
%
\end{align}
%
$$
When $r<1$, $\dot{r}>0$, and the flow is outward.
Inward flow: $\cos 2 \theta \le 1$
$$
%
\begin{align}
%
 \dot{r}_{in} &= -r^{3} + \frac{r}{2} \left( 5 + 3 (-1) \right) \\
%
&= r(4-r^{2})
%
\end{align}
%
$$
When $r>2$, $\dot{r}<0$, and the flow is inward.
Trapping region
The region between the two zones is the annulus centered at the origin with inner and outer radii
$$
%
\begin{align}
%
  r_{in} &= 1 \\
%
  r_{out} &= 2
%
\end{align}
%
\tag{4}
$$
There are no critical points. There region is closed and bounded. Therefore, a periodic solution exists.
Visualization
The vector field $\left[
\begin{array}{c}
 \dot{x} \\
 \dot{y} \\
\end{array}
\right]$ in $(1)$ is plotted against the gray trapping region in $(4)$. The red, dashed lines are nullclines which intercept at the critical point.

A: This is also the Limit Cycle problem in Control theory. It is numerically integrated and plotted.
Whether the initial point is outside or inside the limit cycles of this two-dimensional nonlinear dynamical systems is stable i.e., tend to fixed curve tangentially.
Limit Cycle for any boundary condition of the ode, the trajectory always falls into a (red) curved asymptote ( looks like an ellipse, but TBD?) into which all field lines join tangentially.
Still calculating boundary limit loop. Extreme radii do not look like $(r=1,r=2),$ but the field is same as the one given by OP.

