Determining stability of equilibrium point I want to determine the stability property of the equilibrium point (0,0) for the system $$x'=-xy^4-y\cos(x^2y) \\ y'=3x^5\cos(x^2y)-\sin(y) $$
I get the eigenvalues $-1$ and $0$ for the Jacobian evaluated at $(0,0)$. We have theorems regarding the case when either one eigenvalue has a real part, or when all the eigenvalues are negative. But since these eigenvalues ($0$ and $-1$) do not satisfy this, I feel lost. I know that one could show it by constructing 
a Lyapanov function, but I wasnt able to do so. Could anyone help me out? How do I determine the stability property of origin?
 A: The Lyapunov function is
$$
V(x,y)= \frac12 x^6+\frac12 y^2.
$$
Its derivative along the trajectories 
$$
\dot V= 3x^5\dot x+y\dot y=3x^5(-xy^4-y\cos x^2y)+y(3x^5\cos x^2y-\sin y)
=-3x^6y^4-y\sin y
$$
is non-positive in some neighborhood of the origin, thus, the origin is stable.
In order to prove that the origin is asymptotically stable, we should show that the set
$$
S=\{ (x,y): \dot V(x,y)=0 \}= \{ (x,y): y=0 \}
$$
does not contain whole trajectories of the system except for the origin. This follows from the fact that
$$
\dot y|_{(x,y)\in S}= 3x^5
$$
is nonzero for any $x\ne 0$.
A: Assuming that near $(0,0)$ the dynamical system behaves as
$$
\cases{
\dot x = -x y^4-y\\
\dot y = 3 x^5-y
}
$$
and assuming for the center manifold $h(x) = \sum_{k=0}^n a_k x^{k+1}$ we have
$$
\dot h(x) = h'(x)(-x h^4(x)-h(x)) = \dot y =3x^5-h(x)
$$
and equating coefficients we have for $n=4$
$$
\left\{
\begin{array}{rcl}
 \left(a_0-1\right) a_0&=&0 \\
 \left(1-3 a_0\right) a_1&=&0 \\
 \left(1-4 a_0\right) a_2-2 a_1^2&=&0 \\
 \left(1-5 a_0\right) a_3-5 a_1 a_2&=&0 \\
 a_4-a_0^5-6 a_4 a_0-3 \left(a_2^2+2 a_1 a_3+1\right)&=&0 \\\end{array}
\right.
$$
with solution $h(x)=3 x^5$ for $n=4$. For $n = 2,3, h(x)=0$ and for $n = 8$ we have $h(x) = 45 x^9+3 x^5$
The flow along the central manifold in both cases is stable. Regarding the case $n = 4$ we have a flow given by
$$\dot x = -3 x^5 - 81 x^{21}
$$
which is stable.
Attached a stream flow showing in red part of the center manifold.

NOTE
The $h(x)$ coefficients determination involves two solutions. The plot shown is for $n = 10$
