Computing the first 2 terms of the Taylor series exp. for the center manifold & find the reduced equation on the center manifold. I am having difficulty solving the problem below. It is from Meiss Dynamics book. Can I please receive help solving the following system? Thank you
Consider the system $$x' = y$$ $$y'=-y+ax^2 + bxy.$$ Compute the first two terms of the Taylor series expansion for the center manifold and find the reduced equation on the center manifold. For what values of $a$ and $b$ is the origin stable? Unstable? Semi-stable? Note the linearization at the origin is not in Jordan Canonical Form.
 A: I am going to assume that
$a \ne 0, \tag 1$
for reasons which will become clear in what follows.  Under this assumption, the system
$\dot x = y, \tag 2$
$\dot y = -y + ax^2 + bxy ,\tag 3$
has a single equilibrium point at $(0, 0)$, for setting
$\dot x = \dot y = 0, \tag 4$
we see from (2) that
$y = 0, \tag 5$
and then from (3) that
$ax^2 = 0, \tag 6$
whence
$x = 0. \tag 7$
Here we have used assumption (1), for without it we cannot conclude that the critical set of the system (2)-(3) is a single point.  
The Jacobian matrix of this system at $(x, y)$ is given by
$J(x, y) = \begin{bmatrix} 0 & 1 \\ 2ax + by & bx - 1 \end{bmatrix}; \tag 8$
at $(0, 0)$ this becomes
$J(0, 0) = \begin{bmatrix} 0 & 1 \\ 0 &  -1 \end{bmatrix}; \tag 9$
this matrix has characteristic polynomial
$\det \left(  \begin{bmatrix} -\lambda & 1 \\ 0 &  -1 - \lambda \end{bmatrix}  \right) = \lambda(\lambda + 1); \tag{10}$
the roots of this polynomial are 
$\lambda = 0, -1, \tag{11}$
both of which are real.  Thus $(0, 0)$ is not a center.  
There has been some discussion in the comments suggesting that perhaps the system (2)-(3) should be replaced with
$\dot x = y, \tag{12}$
$\dot y = -x + ax^2 + bxy; \tag{13}$
this system has to zeroes; from (12) we have
$y = 0, \tag{12}$
and then (13) becomes
$0 = -x + ax^2 = x(ax - 1), \tag{13}$
whence
$x = 0, a^{-1}; \tag{14}$
therefore we need inspect the two zeroes $(0, 0)$ and $(a^{-1}, 0)$.
In this case the Jacobian matrix takes the form
$J(x, y) = \begin{bmatrix} 0 & 1 \\ -1 + 2ax + by & bx \end{bmatrix}, \tag{15}$
and we have
$J(0, 0) = \begin{bmatrix} 0 & 1 \\ -1  & 0 \end{bmatrix}; \tag{16}$
the characteristic polynomial is now
$\det \left(  \begin{bmatrix} -\lambda & 1 \\ -1 &  -\lambda \end{bmatrix}  \right) = \lambda^2 + 1, \tag{17}$
and the roots are
$\lambda = \pm i, \tag{18}$
so that $(0, 0)$ is in fact a center.
The Jacobian at $(a^{-1}, 0)$ is
$J(a^{-1}, 0) = \begin{bmatrix} 0 & 1 \\ 1  & ba^{-1} \end{bmatrix}, \tag{19}$
with characteristic polynomial
$\det \left (  \begin{bmatrix} -\lambda & 1 \\ 1  & ba^{-1} - \lambda \end{bmatrix} \right ) = \lambda^2 - a^{-1}b\lambda - 1, \tag{20}$
the roots of which are
$\lambda = \dfrac{a^{-1}b \pm \sqrt{b^2 a^{-2} + 4}}{2}. \tag{21}$
More to follow.  Stay tuned.
A: The equilibrium is attained at the solutions for
$$
\cases{
y=0\\
-y+a x^2+b x y = 0
}
$$
so $(0,0)$ is the equilibrium point. To qualify it we compute the jacobian at this point giving
$$
J = \left(
\begin{array}{cc}
 0 & 1 \\
 0 & -1 \\
\end{array}
\right)
$$
with eigenvalues $(1,\ 0)$ so the equilibrium manifold is one-dimensional. To find this manifold we proceed as follows.
For the dynamical system
$$
\cases{
\dot x=f(x,y)\\
\dot y=g(x,y)
}
$$
Proposing the solution
$$
y=h(x) = \sum_{k=1}^n a_k x^k
$$
we have
$$
\dot y=h_x(x)\dot x = h_x(x)f(x,h(x))=g(x,h(x))
$$
assuming $n=4$ equating the $x$ powers we arrive at
$$
\left\{
\begin{array}{rcl}
 a_1&=&0 \\
a_2 &=& a \\
a_3 &=& a b-2 a^2\\
\end{array}
\right.
$$
and solving we have
$$
h(x) = a x^2+a(b-2a) x^3+ O(x^4)
$$
as a near origin approximation.
Follows a plot showing the stream plot for $a = -\frac 12, b = 1$ showing in thick blue a near origin center manifold segment and in red dashed, a path beginning at $(0.5,0.5)$
NOTE
The central manifold approximate flow for $n=4$ is given by
$$
\dot x = h(x) = a x^2+a (b-2a) x^3+ O(x^4)
$$

