Show that the origin of the following system is stable (non-linear vs linear stability) $$\dot{x} = -2y -x^3 + x^2 y^2$$
$$\dot{y} = x - x^2 y$$
Jacobian evaluation at $(0, 0)$:
$$\begin{pmatrix}0 & -2\\ 1 & 0\end{pmatrix}$$
Clearly the determinant is greater than zero but the trace is zero. Hence the linearization gives the impression of a center, which is marginally stable.
How do I show the origin is stable?
Related question: If in the case that the linearization of a system at a point is not marginally stable, is the stability of the linear system, the same as that of the non-linear system at the point (stable vs unstable)? 
 A: Consider the Lyapunov function
$$
V(x,y)= \frac{x^2}2+y^2.
$$
Its derivative 
$$
\dot V= x\dot x+2y\dot y=-2xy-x^4+x^3y^2+2yx-2x^2y^2
=-x^4-x^2y^2(2-x)
$$
is nonpositive in $D=\{ (x,y)\in\mathbb R^2:\; x<2 \}$, thus, the origin is stable. Moreover, it is asymptotically stable, since the set $S=\{x\in D : \;\dot V(x)=0\}$ does not contain whole trajectories of the system except for the origin (see, for instance, H.Khalil, Nonlinear systems, Corollary 4.1).
The answer to the related question is no. The origin of the system
$$
\dot x=y,\quad \dot y=-x^3
$$
is stable (the Lyapunov function is $\frac{x^4}4+\frac{y^2}2$), but the linearized system
$$
\dot x=y,\quad \dot y=0
$$
is unstable.
A: Near the origin we have
$$
\dot{x} = -2y + O(r^3)\\
\dot{y} = x +O(r^3)
$$
here $r = \sqrt{x^2+y^2}$
The stream plot for the complete system

and for the approximate system

NOTE
Considering the approximated system we have
$$
x\dot x = -2x y\\
2y\dot y = 2x y
$$
adding
$$
\frac{d}{dt}(x^2+2y^2) = 0
$$
or
$$
x^2+2y^2 = C_0
$$
which means that the for the approximated system the origin is a center. 
[![enter image description here][3]][3]
