Use appropriate Liapunov function to determine the stability I want to use appropriate Liapunov function to determine the stability of the equilibrium point of this systems:
$x_1'=x_1-3x_2+x_1^3$
$x_2'=-x_1+x_2-x_2^2$
My solution:
$V(x)=ax_1^2+bx_2^2$
a and b are positive real number.
$gradV(x).f(x)=2ax_1^2-6ax_1x_2+2ax_1^4-2bx_1x_2+2bx_2^2-2bx_2^3$
and then for a=1 and b=-3 my solution failed because it is not guarantee that $x_2^3$ is positive.
Then I choose $V(x)=x_1^2+x_2^2$
$gradV(x).f(x)=2x_1^2-6x_1x_2+2x_1^4-2x_1x_2+2x_2^2-2x_2^3$
but I am stuck, What should I think for stability? Hint or answer, please. This question from Perko's book. (Chapter 2, Problem Set 9, Question 5-b) I would appreciate it if you help. Thank you!
 A: First, the eigenvalues of ${\partial f(0) \over \partial x}$ are $1\pm \sqrt{3}$, hence
since $1+\sqrt{3} >0$ we see that the origin is not a stable equilibrium (Hartman Grobman).
Second, the instability theorem in Perko that is based on Lyapunov type functions (Section 2.9, Theorem 3 (c), before the exercise in question) is not sufficiently refined to deal with Problem 5(b), as this
deals with conditions for a completely unstable equilibrium point. Since one of the
eigenvalues is negative, this does not apply here.
A more appropriate result is Chetaev’s instability theorem (see comments below).
Let $g(x) = (x_1^3,-x_2^2)^T$, note that there is some $k>0$ such that $\|g(x)\| \le k \|x\|^2$ for $x$
close to the origin.
Let $A=\begin{bmatrix} 1 & -3 \\ -1 & 1 \end{bmatrix}$, then the system is
$\dot{x} = Ax + g(x)$.
Let $B=\begin{bmatrix} 0 & -{1 \over 2} \\ -{1 \over 2} & -1 \end{bmatrix}$, and $V(x) = x^TBx = 2 x_2(x_2-x_1)$. It is not hard to see that for any $\epsilon>0$ there is some $x \in B(0,\epsilon)$ such that $V(x) >0$ (any small $x$ with $x_2>x_1$ will do).
Note that $BA+A^TB = I$ (in fact, this is the Lyapunov equation I solved to get $B$) and so
$\dot{V}(t) = \|x\|^2+g(x)^TBx+x^TBg(x)$. It is not hard to see that for $r>0$
sufficiently small that $\dot{V}(x) \ge { 1\over 2} \|x\|^2$. Hence Chetaev’s 
theorem allows us to conclude that the origin is an unstable equilibrium.
Note on Chetaev’s instability theorem:
I only have one reference to the above result, from Vidyasagar's "Nonlinear Systems Analysis". The result is Theorem 106 in Section 5.2.3 (Instability theorems). However, it is not referred to as Chetaev’s instability theorem.
There is another result in the same section Theorem 126 which is referred to as Cetaev's theorem (presumably lying in the null space of Russian transliteration)
which is more refined tool, but not needed here.
(Also, I should show that $V$ is a decrescent function, but that is straighforward
to show.)
