How to prove a fixed point is stable? \begin{align*}
\dot{x} &= 2 x - \frac{8}{5} x^2 - xy\\
\dot{y} &= \frac{5}{2} y - y^2 - 2 xy
\end{align*}
So I have this dynamical system. I linearized it and found that the fixed point (at $x = 1.25$, $y = 0$) is the boundary case (https://en.wikipedia.org/wiki/Linear_dynamical_system#/media/File:LinDynSysTraceDet.jpg). $\tau$ = -2 and $\bigtriangleup$ = 0, which means the linearization says that it is a line of fixed point, which doesn't make sense since there is only 1 fixed point. My professor says that for non-linear system there is disturbance for the boundary case, so 3 possible outcomes are possible. It can be either saddle node, line of fixed points, or stable point. I tried pplane software (you can download it online for free too), and when I zoomed it for x = [1.2, 1.3], y = [-.05, .05]. It does look like there is a line of fixed point slightly above the fixed point (1.25, 0). I'm kind of stuck here. How do I prove the stability of this fixed point now? I also found the Lyapunov function for the system: 
$$V = \ln(x) - \frac{4}{5}\ln(y).$$
The two eigenvalues are -2 & 0 with eigenvectors (1,0) & (5, -8) respectively for fixed point (1.25, 0). This problem is just very weird. I have no idea what eigenvalue of 0 means. I also graphed out all the eigenvectors of the other fixed points too. Basically, I can't tell if the fixed point (1.25, 0) is stable or not. Please help!!
SOME IDEAS: Let's say I have a region D bounded by four points: (1.24, .01), (1.26, .01), (1.24, 0), (1.26, 0). It's easy to see that the Lyapunov inside this region is always positive, and $\dot{V}$ is always negative when x, y > 0. This proves that the fixed point (1.25, 0) is stable at least around this neighborhood. And that means it has to be stable! Does this approach mathematically sound? Thank you! 
 A: The system $$\begin{align*}
\dot{x} &= 2 x - \frac{8}{5} x^2 - xy\\
\dot{y} &= \frac{5}{2} y - y^2 - 2 xy
\end{align*}$$
has other equilibrium points as well. For example $(0,0)$ is an equilibrium point. The linearized system does not tell you the whole story unless you have a structurally stable equilibrium point such as a saddle point. 
A: Let $(x, y) = (-5 \xi/8 + \zeta + 5/4, \xi)$. The system in the new coordinates is
$$\dot \xi = \frac 1 4 \xi^2 - 2 \xi \zeta, \\
\dot \zeta = -2 \zeta + \frac 5 {32} \xi^2 - \frac 1 4 \xi \zeta - \frac 8 5 \zeta^2.$$
The linearized $(\xi, \zeta)$ system has one zero and one negative eigenvalue. Then the center manifold of the form $\zeta = h(\xi)$ exists (an approximation to it can be found by constructing a power series solution to $\dot \zeta = h'(\xi) \dot \xi$).
Since $h(0) = h'(0) = 0$, we necessarily have $h(\xi) = O(\xi^2)$. Therefore $\dot {\xi_c} = \xi_c^2/4 + O(\xi_c^3)$ for the vector field restricted to the center manifold. The fact that $\xi_c = 0$ is unstable implies that $(\xi, \zeta) = (0, 0)$ is also unstable. In the original coordinates, the trajectories starting at points close to $(5/4, 0)$ converge to $(5/4, 0)$ if $y < 0$ and to $(0, 5/2)$ if $y > 0$.
