Qualitative behavior of critical point at the origin Determine the qualitative behavior of the critical point at the origin for the following system for all possible values of $a$:
$\dot{x} = -y + ax(x^2+y^2)$
$\dot{y} = x + ay(x^2+y^2)$
My question: I attempted to use the Local Center Manifold theorem to show that the center manifold: $x = h(y) = a_0 + a_1y+a_2y^2 +...$ for $a_0, a_1, ...$ are parameters to be determined and $h(0) = h'(0) = 0$, must be $0$. To do this, assume $h(y)\neq 0$ for $y\neq 0$. Now, we replace $x$ by $h(y)$ from the sysem above, and from the identity: $\dot{x} = \dot{y}\  h'(y)$, we get the following equation for all values of $a$:
$-y + a(a_0 + a_1y+ a_2y^2 + ...) (a_0^2+a_1^2y^2 + 2a_0a_1y+2a_0a_2y^2 +...+y^2) = (a_1 + 2a_2y + 3a_3y^2 + ...)[a_0 + a_1y+ a_2y^2 + ... + ay(a_0^2 + a_1^2y^2 + 2a_0a_1y + 2a_0a_2y^2 + ... +y^2)]$
Since $h(0) = h'(0) = 0$, we instantly get $a_0 = a_1 = 0$. But then the $-y$ term on the LHS of the equation above is never cancelled with anything, so the equation, after matching terms by terms, cannot be true for every $y\neq 0$. Thus $h(y)$ does not exist in this case.
Therefore, $h(y) = 0$ is the only choice, which implies $x = 0$. But if this is the case, then $0 = -y$, so $y = 0$ as well. Thus the critical point is a saddle point? Is this a correct conclusion?
 A: I know this is not the method you were going for, but I could not resist offering another possible solution.
According to the Wikipedia article on Lyapunov Stability:

Lyapunov, in his original 1892 work, proposed two methods for
  demonstrating stability. The first method developed the solution in
  a series which was then proved convergent within limits. The second
  method, which is almost universally used nowadays, makes use of a
  Lyapunov function $V(x)$ which has an analogy to the potential
  function of classical dynamics. It is introduced as follows for a
  system  $\dot{x} = f(x)$ having a point of equilibrium at x=0.
  Consider a function $V(x)$ : $\mathbb{R}^n \rightarrow \mathbb{R}$ 
  such that:
  
  
*
  
*$V(x)=0$ if and only if $x=0$
  
*$V(x)>0$ if and only if $x \ne 0$ 
  
*$\dot{V}(x) = \frac{d}{dt}V(x) = \sum_{i=1}^n\frac{\partial       V}{\partial x_i}f_i(x) \le 0$ for all values of $x$ (negative
  semidefinite). Note: for asymptotic stability,  $\dot{V}(x)<0$  for $x
> \ne 0$ is required (negative definite).
  

In your example, a candidate for a Lyapunov function is $V(x,y) = x^2 +y^2$.
Note: the method in the article, as it stands, will give you the behavior for $a=0$ and $a<0$. Here is the additional case:

Suppose $X$ is a $C^{1}$ vector field on an open set $\Omega \subset
> \mathbb {R}^n$, $0 \in \Omega$ is a critical point of $X$, and $V :
> \Omega \rightarrow \mathbb {R}$ is a continuous function such that
  
  
*
  
*$V (0) = 0$
  
*there exists $\Omega_{-} \subset \Omega$ such that $\Omega_{-} \cap B_{\delta}(0)= \emptyset$ for any $\delta >0$, $V (x) < 0 \ \forall
> x\in \Omega_{-}$, $V (x) = 0 \ \forall x\in \partial \Omega_{-} \cap
> B_{\epsilon} (0)$ for some $\epsilon >0$;
  
*$V$ is strictly decreasing on the part of orbits that stay in $\Omega$.
  
  
  Then $0$ is unstable.

In the case of $a>0$, let $V(x,y) = -x^2 -y^2$ and use the theorem above.
