Determine the region of the phase plane in which all phase paths are periodic orbits

Question

Sketch the phase portrait of the dynamical system

\begin{align} \dot{x}&=y+2xy \\ \dot{y}&=-x+x^2-y^2\end{align}

and determine the region of the phase plane in which all phase paths are periodic orbits.

I have found that the equilibrium points $(0,0),(1,0), (-\frac{1}{2},\frac{\sqrt{3}}{2}),(-\frac{1}{2},-\frac{\sqrt{3}}{2})$ are respectively a centre and three saddle points,

and that the horizontal isocline is $(x-\frac{1}{2})^2-y^2=\frac{1}{4}$ and the vertical isoclines are $y=0$ and $x=-\frac{1}{2}$.

How to determine the region of the phase plane in which all phase paths are periodic orbits?

Answer 1

Note that the system is Hamiltonian since

$$\frac{\partial}{\partial x} (y+2xy) = 2y = - \frac{\partial}{\partial y} (-x+x^2 -y^2). $$

Indeed a Hamiltonian function is given by $H(x, y) = \frac{1}{2}x^2 -\frac{1}{3} x^3 + \frac{1}{2} y^2 + xy^2$. The flow lines are given by the level curve of $H$.

Now we consider the level set which contains the line $x = -1/2$ (From your analysis, we know that this line is part of the vertical nullclines, thus is part of a solution curve). Since $H(-1/2, 0) = 1/6$, after some calculations,

$$ \frac{1}{2}x^2 - \frac{1}{3} x^3 + \frac{1}{2} y^2 + xy^2= \frac 1{6}\Rightarrow \left( x+\frac 12\right)\left(y+ \frac{1}{\sqrt 3}(x-1)\right)\left(y - \frac{1}{\sqrt 3}(x-1)\right)=0.$$

So luckily this level curves is a union of three lines. The triangle bounded by these three lines are exactly where all the periodic orbit are located.