Existence and uniqueness of homoclinic orbit 
Consider the following 2D autonomous system of ODEs:
$$
\left\{
\begin{array}{ll}
\dot{x} = x^2 + 2y - x \\
\dot{y} = 3xy/2 - 3x^2 - y + 2x
\end{array}
\right.
$$

How can we prove the existence and uniqueness of a homoclinic orbit (i.e. a solution $X$ of the system
for which $\lim_{t \to \infty} X(t) = \lim_{t \to -\infty} X(t) = x_0,$ where $x_0$ is an equilibrium point of the system) for the system?
It is not hard to determine the equilibrium points: $(0,0), (-3, -6)$ and $(2/3, 1/9)$. The latter two are asymptotically stable equilibrium points, so there cannot be a homoclinic orbit for those points.
However, $(0,0)$ is a saddle equilibrium, so perhaps there is a homoclinic orbit here.
We can also see that the function $H(x,y) = - y^2 + x^2(1-x)$ is constant on the solutions of the system.
The set $H(x,y) = 0$ seems to be comprised of three solutions of the system, one of which is indeed a homoclinic orbit, so this proves existence.
However, how do we prove that this is unique? I don't really know how to approach this part.
 A: Let me explain the idea from my previous answer with few more additional details.
As it was mentioned in the question, the system has a smooth first integral $H(\mathbf{x})$ — a (locally non-constant) function that is constant along the system's trajectories. The key observation is that if such system has a trajectory $\gamma(t)$ homoclinic to a saddle $p$, then $H(p) \equiv H(\gamma(t))$. To prove that we can pick a sequence of moments $t_i \rightarrow +\infty$. By continuity, since $\gamma(t_i) \rightarrow p$ and $H(\mathbf{x})$ is smooth, then $H(\gamma(t)) \rightarrow H(p)$. But since $H(\mathbf{x})$ is constant at any point of $\gamma(t)$, we have that $H(\gamma(t)) \equiv H(p)$.
This gives us the following method to find all homoclinic trajectories for a 2D system of differential equations. First, find all saddle equilibria. Second, find the level sets of $H(\mathbf{x})$ that contain these saddles. What we have proven before means that a homoclinic trajectory must lie in the same level set as the saddle, to which it is homoclinic. Just take a look at level sets after that and you can find homoclinic (and even heteroclinic) trajectories this way.
A: Actually, proving uniqueness of the homoclinic orbit is much easier than expected, but it requires some heavy machinery.
The Hartman-Grobman theorem says that, near a hyperbolic equilibrium point, any system of ODEs is topologically conjugate to its linearization.
In the linear case, for a saddle equilibrium point, there are $2$ trajectories going out of the equilibrium and two trajectories coming in. So, this is the behaviour for a general system as well.
Using you function $H$, you already determined your trajectories, so the above discussion tells you that there is only one homoclinic orbit.
