Stability of a fixed point in a nonlinear system with no linear part The following nonlinear system has a fixed point in the origin. I want to know if this fixed point is stable:
\begin{align*}
&\dot{\alpha}=\alpha^2-2\beta(\alpha+\beta)\\
&\dot{\beta}=\beta^2-2\alpha(\alpha+\beta)
\end{align*}
Note that the linearization of the system leads to $\dot{\alpha}=\dot{\beta}=0$ to first order, so this method is useless. To find out the stability, I wrote this equations as:
\begin{align*}
\dot{\alpha}=\left(\alpha,\beta\right)
\cdot A\cdot
\left(\begin{array}{c}
\alpha\\
\beta
\end{array}\right),\quad
A=\left(\begin{array}{cc}
1 & -1\\
-1 & -2
\end{array}\right)
\end{align*}
\begin{align*}
\dot{\beta}=\left(\alpha,\beta\right)
\cdot B\cdot
\left(\begin{array}{c}
\alpha\\
\beta
\end{array}\right),\quad
B=\left(\begin{array}{cc}
-2 & -1\\
-1 & 1
\end{array}\right)
\end{align*}
Since $A$ and $B$ are real and symmetric, they are diagonalizable, with real eigenvalues. It turns out that $A$ and $B$ have one eigenvalue positive and the other negative (in fact, $A$ and $B$ have the same eigenvalues), explicitly:
\begin{align*}
&\text{Eigenvalues of $A$:}\quad \frac{1}{2} \left(-1-\sqrt{13}\right),\frac{1}{2} \left(\sqrt{13}-1\right)\\
&\text{Eigenvalues of $B$:}\quad \text{same of $A$}
\end{align*}
If, for example, the eigenvalues of $A$ were both positive, then $\dot{\alpha}$ would be positive for all points except origin, and then the origin would be unstable (because trajectories starting out of the origin would increase their $\alpha$'s to infinity). But when both $\dot{\alpha}$ and $\dot{\beta}$ are positive or negative depending on the point $(\alpha,\beta)$, as in this case, how can you determine the stability of the fixed point?
EDIT: The phase portrait is:

 A: This is mostly a recap of the observations made in the comments, plus some more analysis, because I think it's a nice problem to analyse.
First, both the functional form of the system and the reflection symmetry (see also the phase plane) suggest it's a good idea to introduce $x = \alpha+\beta$, $y = \alpha-\beta$, to obtain
\begin{align}
 \dot{x} &= \frac{1}{2}(y^2 - 3 x^2), \tag{1a}\\
 \dot{y} &= 3 x y.\tag{1b}
\end{align}
We can simplify system (1) somewhat by rescaling $y \to \sqrt{3} y$ and $t \to \frac{1}{3} t$, yielding
\begin{align}
 \dot{x} &= \frac{1}{2}(y^2 - x^2), \tag{2a}\\
 \dot{y} &= x y.\tag{2b}
\end{align}
The phase plane of system (2) looks like this:

What's immediately obvious, is that this phase plane is highly symmetric. It seems to be invariant under rotation over an angle of $\frac{2 \pi}{3}$ (= 120 deg), and rotation over an angle of $\frac{2 \pi}{6}$ seems to keep the shape of the orbits invariant, but changes their flow direction. You can check that both of these observations are indeed correct by considering
\begin{equation}
 \begin{pmatrix} \xi_1 \\ \eta_1 \end{pmatrix} := R(\frac{2\pi}{3}) \begin{pmatrix} x \\ y \end{pmatrix}\quad \text{and} \quad  \begin{pmatrix} \xi_2 \\ \eta_2 \end{pmatrix} := R(\frac{2\pi}{6}) \begin{pmatrix} x \\ y \end{pmatrix},
\end{equation}
with
\begin{equation}
 R(\theta) = \begin{pmatrix} \cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix}
\end{equation}
is the matrix that rotates a vector about the origin over angle $\theta$. Substituting $\xi{1,2}$ and $\eta_{1,2}$ in system (2), you obtain
\begin{align}
 \dot{\xi_1} &= \frac{1}{2}(\eta_1^2 - \xi_1^2), \\
 \dot{\eta_1} &= \xi_1 \eta_1,
\end{align}
and
\begin{align}
 \dot{\xi_2} &= -\frac{1}{2}(\eta_2^2 - \xi_2^2), \\
 \dot{\eta_2} &= -\xi_2 \eta_2,
\end{align}
which implies the above observations.
It seems a good idea to 'factor out' this rotational symmetry present in system (2). Introducing polar coordinates $x = r \cos \theta$, $y = r \sin \theta$, we obtain the dynamical system
\begin{align}
 \dot{r} &= -\frac{1}{2} r^2 \cos(3\theta),\tag{3a}\\
 \dot{\theta} &= \frac{1}{2} r \sin(3\theta),\tag{3b}
\end{align}
which is indeed invariant under $\theta \to \theta + \frac{2 \pi}{3}$. Now, we rewrite the above in terms of the new angle $\phi := 3 \theta$ to obtain
\begin{align}
 \dot{r} &= -\frac{1}{2} r^2 \cos(\phi), \tag{4a}\\
 \dot{\phi} &= \frac{3}{2} r \sin(\phi). \tag{4b}
\end{align}
Now, the nice thing is that we can reinterpret system (4) as being the `polar representation' of some Cartesian system. That is, if we introduce $X$ and $Y$ by $X := r \cos \phi$ and $Y := r \sin \phi$, we can rewrite system (4) in terms of $X$ and $Y$ to obtain
\begin{align}
\dot{X} &= -\frac{1}{2}(X^2 + 3 Y^2), \tag{5a}\\
\dot{Y} &= X Y. \tag{5b}
\end{align}
The phase plane of system (5) looks as we would have expected:

Moreover, system (5) turns out to be Hamiltonian, i.e. of the form
\begin{align}
 \dot{X} &= \frac{\partial H}{\partial Y},\\
 \dot{Y} &= -\frac{\partial H}{\partial X},
\end{align}
with
\begin{equation}
 H(X,Y) = -\frac{1}{2} Y(X^2 + Y^2).
\end{equation}
Therefore, orbits lie on level sets of $H$, i.e. those curves where $H = E$ (= constant), which can be used to express $X$ in terms of $Y$ as
\begin{equation}
 X = \pm \sqrt{-\frac{2 E + Y^3}{Y}}.
\end{equation}
As a final remark, the instability of the origin can be derived from system (5) as follows. Consider the horizontal line $\ell = \left\{ (X,Y) \vert Y=0 \right\}$, i.e. the $X$-axis. Substituting $Y=0$ in system (5), we obtain $\dot{Y} = 0$; therefore, the line $\ell$ is invariant under the flow. In other words, every point on $\ell$ stays on $\ell$ for all time. 
It so happens that the origin $(0,0)$ lies on the line $\ell$. Taking a point $(-\epsilon,0) \in \ell$ arbitrarily close to (and directly to the left of) the origin, we see that this point will flow away from the origin, because the flow on $\ell$ is given by $\dot{X} = -\frac{1}{2} X^2$. This holds for all $\epsilon > 0$; hence, we have found an unstable flow direction at the origin, spanned by $(-1,0)$. Therefore, the origin is a (nonlinearly) unstable fixed point of system (5), and in extension of system (1).
