Equilibrium points and homoclinic paths of dynamical system Exercise
Investigate the equilibrium points of
$\dot{x} = y \left( 16 \left(2x^2+2y^2-1 \right) -1 \right)$
$\dot{y} = x - \left(2x^2+2y^2-1 \right)\left(16x-4\right)$
and classify them according to their linear approximations.
Show that the homoclinic paths through $(0,0)$ are given by
$\left(x^2 + y^2 -\frac{1}{2}x \right)^2 - \frac{1}{16}\left(x^2+y^2\right)=0$
and that one homoclinic path lies within the order. Draw the phase portrait.
(Hint: Find the Hamiltonian function of the system)
Attempt
I have calculated the equilibrium points to be $(0,0), (\frac{3+\sqrt{3}}{8},0),(\frac{3-\sqrt{3}}{8},0)$
For the classifications of the equilibrium points I calculated the linearized system around each point and found that
The point $(\frac{3+\sqrt{3}}{8},0)$ is unstable saddle point for the almost linear system (original system)
The point $(\frac{3-\sqrt{3}}{8},0)$ is stable center point for the almost linear system.
The point $(0,0)$ is stable center point for the linearized system. This means that the Hartman-Grobman theorem for system linearization cannot be applied. Q1: What can I do to determine the classification of this point for the almost linear system?
For the homoclinic path I calculated the Hamiltonian function of the original system to be
$H(x,y) = 8x^4 + 8y^4 -8x^3 + \frac{3}{2}x^2 - \frac{1}{2}y^2 +16x^2y^2 -8xy^2$
If I set $H(x,y)=0$ then I get the equation that is supposed to represent the homoclinic paths but I do not understand why that is true. Q2: Can someone explain?
Q3: I also don't understand what the phrase "one homoclinic path lies within the order" means.
 A: Q1: For the Jacobian computed on $(0,0)$, it holds
$$\left| \begin{matrix}
0 & -1 \\ -3 & 0
\end{matrix} \right|
<-\infty$$
so $(0,0)$ is a saddle point of the linearized.
Q2: $H$ is continuous. Assume that $H(\phi(t,t_0,x^0)) = c$, for every $t\in\mathbb{R}$ ($H$ is a constant of motion for the Hamiltonian system), where $x^0$ lies on a homoclinic path of $(0,0)$. By definition of homoclinic path, $\lim_{t\rightarrow\pm\infty}\phi(t,t_0,x^0)=(0,0)$. Under the continuous function $H$, preimage of a closed set in $\mathbb{R}$ (e.g. $\{0\}$) is closed in $\mathbb{R}^2$ (e.g. $H^{-1}(\{0\})$). Let $t_k$ be a sequence such that $t_k\rightarrow\infty$. Then
$$
\begin{array}{rcl}
\phi(t,t_0,x^0) \in H^{-1}(\{c\}) & \Rightarrow & \lim\limits_{k\rightarrow\infty}\phi(t_k,t_0,x^0) \in H^{-1}(\{c\}) \\
& \Rightarrow & (0,0) \in H^{-1}(\{c\}) \\
& \Rightarrow & c=0
\end{array}
$$
because $H(0,0)=0$.
Also, $H^{-1}(\{0\})$ is an 1-dimensional regular submanifold of $\mathbb{R}^2$ (I don't know if this answers Q3).
