Prove that an orbit is heteroclinic I was given the following ODE as part of an exercise:
$$\frac{dx_1}{dt} = x_2$$
$$\frac{dx_2}{dt} = -x_2 + \frac{x_1^2}{2} - \frac{1}{2}$$
The question is to show that a heteroclinic orbit exists for this system.
My attempt:
By considering the linearization around the fixed points (1,0) and (-1,0), I deduced that (1,0) is a saddle point and (-1,0) is a sink. Then I immediately concluded (by Poincare-Bendixson Theorem) that a heteroclinic orbit exists. My teacher told me that I needed to do something more before invoking the Poincare-Bendixson Theorem.
Any clues and ideas on what he meant? Or any other alternative ways to approach this problem?
 A: Okay, let's start from the form of your equations. When I see system $\dot{x} = y, \; \dot{y} = \dots$ the first thing which I think about is that maybe system is equivalent to $\ddot{x} = F(x)$. I read the second equation and I see that it's not the case: $\dot{x} = y, \; \dot{y} = -y + \frac{x^2-1}{2}$. But how exactly it's not the case? Well, there is only $-y$ term that spoils everything. If there is no such term it is conservative system, it has first integral and there is a lot can be said about dynamics. For example, if we get rid of $-y$ term, the system beecomes $\dot{x} = y, \; \dot{y} = \frac{x^2-1}{2}$ and $\Phi(x, y) = \frac{y^2}{2} - \frac{x^3}{6} + \frac{x}{2}$ is a first integral. Let's check how badly this $-y$ term spoils the picture. For this we have to calculate derivative with respect to your original system:
$$ \frac{d}{dt}(\Phi(x, y)) = \Phi_{x}(x, y) \cdot \dot{x} + \Phi_y(x, y) \cdot \dot{y} = - \frac{x^2-1}{2} \cdot y + y \cdot \left( -y + \frac{x^2-1}{2} \right ) = -y^2$$
It's a kind of Lyapunov function — you have to narrow the domain and pick some neighbourhood to apply Lyapunov theorem, but still it's useful here. For example, we get the following: when trajectory doesn't intersect $Ox$ axis the first integral decreases. From geometric point of view that means that along any level set vector field points in the same direction (or tangent). 
Referring to my earlier answer we can try proving the existence of heteroclinic trajectory using the case of trapping region. Pick a level set that contains saddle equilibrium. Vector field points inward along it except at two points (they are both on $Ox$ axis). The part of this level set is a boundary of our trapping region: vector field points inward, no trajectory can escape from this region. If you want to say that there is a problem with these two points, there is no problem really: one of them is a saddle of $C^1$-system and no trajectory can escape domain through it. Why system can't escape from another point on $Ox$ axis? Well, you can check $\ddot{x}$ for a trajectory that goes through this point and find does it really go in or out; I'll stay with Bony-Brezis theorem. Now we have key ingredient for applying Poincaré-Bendixson theorem — a forward invariant domain. But Poincaré-Bendixson theorem states that in compact domain of planar system only attractors are equilibria, limit cycles and heteroclinic contours or homoclinic loops. We know all equilibria for sure, but what do we know about limit cycles, heteroclinic contours or homoclinic loops? Here is the last ingredient: Bendixson-Dulac theorem. The divergence of this system is constant and equals $-1$ — that prevents existence of limit cycles, heteroclinic contours and homoclinic loops.
So, to sum up:


*

*We found forward-invariant compact region to which Poincaré-Bendixson theorem can be applied

*In this region we know all equilibria and their stability properties

*Also, we know that there is no limit cycles or homoclinic loops or heteroclinic contours due to Dulac-Bendixson

*If we pick unstable separatrix of saddle, it must stay in invariant region and it must go to attractor. The only attractor is a sink, so unstable separatrix goes to it. 


ADDED LATER
This illustration should clarify what trapping region I was using in my answer.

