LaSalle invariance, Lyapunov stability Trying to understand the LaSalle invariance principle.
Consider the system
$x' = y \\
y' = -y-6x-3x^2$
a) Using the test function $V(x,y) = 0.5y^2+3x^2+x^3$, show that the origin is asymptotically stable.
The principle roughly says that if $S$ is the union of all complete trajectories contained in $\{\textbf{x}: V'(\textbf{x})=0\}$, then any trajectory will end up in $S$.
In this case, some work gives us $V' = -y^2$ so $V'=0$ is true for the entire x-axis. Qualitatively, It seems clear that if $y=0$, then $x'=0$ but $y'$ will take us somewhere where $V'<0$. I.e. if we start at $y=0,x\neq0$, the resulting trajectory will not satisfy $V'=0$. The origin will be in $S$ however.
But isn't the sequence $\{(-2,0),(-2,0),\dots\}$ contained in $S$ as well? How can we, from the invariance principle, deduce that we will end up at the origin and not the other equilibrium?
b) Estimate the domain of attraction. (Adding this since it's relevant)
Here we use the level set of $V$ that passes through the other equilibrium point as an estimate, i.e. $0.5y^2+3x^2+x^3 = 4$. But isn't there a bunch of other points that are attracted to the origin? I'm struggling to understand how the level sets of $V$ interact with $V'$ and in general how $V$ and $V'$ interact with which equilibrium will be attracting.
 A: I think too that the lack of positivity for $V$ is an obstacle here. Since $\dot V=-|y|^2=0$ iff $y=0$, we set $y=0$ in the system and find out that the only trajectories that lie completely in $S$ are two equilibria $(0,0)$ and $(-2,0)$. Note that $V(-2,0)=4$. Therefore in the set $V(x,y)<4$ the system has no trajectories that lie completely in $S$ except for the origin, and it would be tempting to say that the origin is asymptotically stable by the LaSalle principle. However, the test function $V$ is not positive definite, and to apply the principle at least locally we need to find an invariant neighborhood D of the origin where $V(x,y)>0$ for all $(x,y)\in D\setminus \{0\}$, otherwise the condition $\dot V<0$ is useless, because it is possible that $V(x(t),y(t))\to-\infty$, and a trajectory can escape $(0,0)$ and go to $\infty$ instead. Finding such an invariant $D$ is not obvious and very often equivalent to finding another positive test function.
P.S. The original problem corresponds to the second order system
$$
x''+x'+\phi(x)=0
$$
where often the condition $x\phi(x)>0$ is assumed precisely to avoid this kind of trouble. See, for example, here (page 19 and 22-23).
P.P.S. The origin is indeed locally asymptotically stable (by linearization).
UPDATE:
It turns out that it is actually not hard to find the invariant $D$ here after one does some drawing. The problem was that the set $V(x,y)<4$ was unbounded and contained, in particular, points where $V$ was not positive definite, so that the application of the principle could not be justified. However, after a closer look at the set (see the figure below) one can notice that yes, it is unbounded, but it has two connected components, one in $x<-2$ (unbounded) and one in $x>-2$ (bounded). The latter can be chosen as $D$ since it is invariant and $V$ is positive definite there. Once a trajectory starts in there it cannot enter another component and escape. So 
$$
D=\{(x,y)\colon\ V(x,y)\le 4,\ x>-2\}.
$$
This is also the best region of attraction to origin one can ensure with this test function.

A: The Lasalle Invariance Principle only applies to test functions $V(x,y)$ that are positive definite. Yours is not. It may be possible to restrict the problem to the region where it is and apply the theorem there but I'm not completely sure about that, in fact given that $V(-2,0) > 0$ and that equilibrium point is the source of your problem it seems unlikely to me. 
