Stability when eigenvalues are zero : $x' = -x + y + x^2 + ax^3, \space y' = x - y + x^2 + bxy + y^3$ Exercise :

Determine the stability of $O(0,0)$ for the system of differential equations: 
  $$x' = -x + y + x^2 + ax^3$$
  $$y' = x - y + x^2 + bxy + y^3$$

Discussion :
Finding the matrix of the linearized system (the Jacobian), we get : 
$$J(x,y) = \begin{bmatrix} -1 + 2x + 3ax^2 &1 \\ 1 + 2x + by& -1 + bx +3y^2 \end{bmatrix}$$
and the matrix at $O(0,0)$ is : 
$$J(0,0) = \begin{bmatrix} -1 & 1 \\ 1 & -1\end{bmatrix}$$
Finding the eigenvalues of the matrix to proceed with determining its stability, we get : 
$$\det(J(0,0)-λI) = 0 \Rightarrow\dots \Rightarrow λ(λ+2) = 0 \Leftrightarrowλ=-2 \space \text{or} \space λ =0$$
My question is, how does one proceed in the case of a zero eigenvalue, which means that $λ_1 \cdot λ_2 = 0$ ? 
I know a way bypassing such issue is converting to polar coordinates, which can be done rather simply but it doesn't lead anywhere clear on this specific example.
The system also cannot be solved so it can't be worked around freely by this way (obviously). 
Finally, I've figured out 1-2 theorems working over the implicit function theorem to prove a strong case via it, but they include hypothesis and an assisting function that is a total mess.
I would really appreciate anyone's help on how to proceed on such problems where we are stuck with a zero eigenvalue.
 A: When there are some zero eigenvalues and no positive eigenvalues of the Jacobian, the stability is that of the flow along the center manifold. So what you to do (and this is the canon for this type of problems) is finding the equation on the center manifold (more precisely, on any center manifold in case it is not unique).
How do you do that? 
First pass to coordinates $$u=x+y,\quad v=x-y$$ (their axes are tangent to the stable and center spaces). After writing the equation in the form $$u'=P(u,v),\quad v'=Q(u,v),$$ the second equation along the center manifold will give the stability (which doesn't change with the change of variables). We start like this: write the graph of the center manifold as $$u=cv^2+dv^3+\cdots,$$ substitute in the first equation, deduce from the equation $$(cv^2+dv^3+\cdots)'=P(cv^2+dv^3+\cdots,v)$$ what are $c,d,\ldots$ (just equate equal powers) and replace that in the second equation $$v'=Q(cv^2+dv^3+\cdots,v)$$ now already the constants  $c,d,\ldots$ that you determined. The stability of the origin in this equation will be the stability of the origin in the original equation.
