I am trying to assess the (Liapunov) stability of the equilibrium at $(0,0)$ of the system \begin{align*} x_1' &= -4x_2 + x_1^2 \\ x_2' &= 4x_1 + x_2^2. \end{align*} I plotted the phase portrait in Mathematica, and it looks like a stable (but not asymptotically stable) equilibrium, with orbits circling about the origin. However, my professor tells me the origin is actually unstable. I don't know how to prove or disprove this rigorously. The equilibrium is non-hyperbolic, and I haven't been able to find a Liapunov function for the system. Any guidance would be appreciated.
Edit: I tried polar coordinates: I get $$ r' = \frac1r(x_1x_1' + x_2x_2') = r^2(\cos^3 \theta + \sin^3 \theta)$$ and $$ \theta' = \frac{x_1x_2'-x_1'x_2}{r^2} = 4 + r\cos\theta\sin\theta(\sin\theta-\cos\theta).$$ So there's a half-plane where $r$ is increasing and a half-plane where it's decreasing. The angle $\theta$ is increasing in a neighborhood of the origin. Is it reasonable to say that since $$ 0 = \int_{\theta_0}^{\theta_0 + 2\pi} \cos^3 \theta + \sin^3 \theta d\theta = \int_{\theta_0}^{\theta_0 + 2\pi} \frac{r'}{r^2} d\theta = \int_{\theta_0}^{\theta_0 + 2\pi} \left(\frac{1}{r}\right)' d\theta, $$ that the orbits are in fact periodic?