How to find lyapunov function for the system? I have to determine the stability of the system:
$$\begin{cases}x' = xy^4 - 2x^3 - y \\ y' = 2x + 2x^2y^3 -y^7\end{cases}$$
How to fetermina what is Lyapunov function $V$ for this system? 
I know that later I have to find:
$$\frac{dV}{dt} =\frac{\partial{V}}{\partial{x}}\frac{dx}{dt} + \frac{\partial{V}}{\partial{y}}\frac{dy}{dt}$$
but I can not realize how to find $V$ and where should I pay attention at.
 A: Assume that a Lyapunov function $V$ exists then
$$\dot{V} = \Big(x\dfrac{\partial V}{\partial x}-y^{3}\dfrac{\partial V}{\partial y}\Big)(y^4-2x^2)-y\dfrac{\partial V}{\partial x}+2x\dfrac{\partial V}{\partial y}$$
This suggests the function $V = x^2+\dfrac{y^2}{2}$ which is postive definite and radially  unbounded. Further, we have
$$\dot{V} = -(y^4-2x^2)^2$$
which is negative semi-definite. So we can conclude that the system is globally stable.
A: Using $x_1 = x$ and $x_2 = y$, you have the nonlinear planar system
$$
\begin{align}
\dot{x}_1 &= x_1 x_2^4 - 2 x_1^3 - x_2 \\
\dot{x}_2 &= 2 x_1 + 2 x_1^2 x_2^3 - x_2^7
\end{align} \tag{1}
$$
which has an equilibirum at $(x_1, x_2) = (0, 0)$. We can choose the Lyapunov function
$$
V(x_1, x_2) = x_1^2 + \frac{1}{2} x_2^2
$$
which is globally positive definite and radially unbounded. Its derivative is
$$
\dot{V}(x_1, x_2) = -(2 x_1^2 - x_2^4)^2
$$
so $\dot{V}(x_1, x_2) \leq 0$ and the system is Lyapunov stable. This was found with SOSTools, which can be applied here since the vector field is polynomial.
Edit: To account for comment by Hans Lundmark.
To show asymptotic stability with LaSalle, we need to show that no trajectory except the trivial solution can stay in the set $S = \{ (x_1, x_2) \in \mathbb{R}^2 : \dot{V}(x_1, x_2) = 0 \}$.
The condition $\dot{V}(x_1, x_2) = 0$ is true if $x_1 = \pm \frac{1}{\sqrt{2}}x_2^2$, so along the curve
$$
C(x_2) = \begin{bmatrix}
\pm \frac{1}{\sqrt{2}}x_2^2 \\
x_2
\end{bmatrix}
$$
The tangent vector to this curve is given as
$$
T(x_2) = \frac{d}{d x_2} C(x_2) =  \begin{bmatrix}
\pm \sqrt{2} x_2 \\
1
\end{bmatrix}
$$
Insert $x_1 = \pm \frac{1}{\sqrt{2}}x_2^2$ into the vector field in $(1)$ to get
$$
F(x_2) = \begin{bmatrix}
-x_2 \\
\pm \sqrt{2} x_2^2
\end{bmatrix}
$$
For $F$ and $T$ to be parallel, there must exist $k \in \mathbb{R}$ such that $T(x_2) = k F(x_2) \, \forall x_2 \in \mathbb{R}$. For the first row this is ensured by $k = \mp \sqrt{2}$. However, for the second row this choice of $k$ leads to $\mp 2 x_2^2 = 1$, which cannot be true for all $x_2 \in \mathbb{R}$.
So, the vector field is transversal to the curves $C$ and no solution (except the trivial) can stay in $S$ and so, the system is asymptotically stable.
