Unstable fixed point Consider the system 
$\dot{x} = x(1-4x^2-y^2)-\frac{1}{2}y(1+x) $
$\dot{y} = y(1-4x^2-y^2)-2x(1+x) $
Show that origin is an unstable fixed point
I made $\dot{x} = 0$ and $\dot{y}=0$ and $\dot{x} = \dot{y}$ . Then found an equation,$x=0,y=\frac {\sqrt3}{2}$ but when I did y=0, i found imaginary x. Then I did $\frac{\dot{y}}{\dot{x}}$. I cant continue now on.
I am editing, actually adding one more question.I confused because of that
Here is it,
By considering, $\dot{V}$ , wherer $V=(1-4x^2-y^2)^2$, show all trajectories approach the ellipse $4x^2+y^2=1$ as t goes to infinity
 A: Hints:


*

*Find the Jacobian, $J(x,y)$

*Evaluate the eigenvalues of the Jacobian at the point $(x,y) = (0,0)$.


Note: It looks like you were trying to find all of the fixed points. I am not sure if you were supposed to find all of the fixed (critical) points, but there are four of them, which includes the point $(x,y) = (0,0)$.
The fixed points are:


*

*$x = -\dfrac{1}{8},   y = -\dfrac{1}{4}$

*$x = 0,   y = 0$

*$x = \dfrac{1}{16} (1-\sqrt{65}),   y = \dfrac{1}{8} (\sqrt{65}-1)$

*$x = \dfrac{1}{16} (1+\sqrt{65}),   y = \dfrac{1}{8} (-1-\sqrt{65})$


Here is a phase portrait showing these.

Updates to Question


*

*You are given $V=(1-4x^2-y^2)^2$.

*Evaluate $V' = 2(1-4x^2-y^2)(-8x x' - 2yy')$

*Substitute in $x'$ and $y'$ and simplify.

*Do you notice anything that looks like the ellipse $4x^2+y^2=1$ in the phase portrait?

*This is what this part of the exercise is after (notice how all of the trajectories approach this ellipse).

A: $\newcommand{\+}{^{\dagger}}%
 \newcommand{\angles}[1]{\left\langle #1 \right\rangle}%
 \newcommand{\braces}[1]{\left\lbrace #1 \right\rbrace}%
 \newcommand{\bracks}[1]{\left\lbrack #1 \right\rbrack}%
 \newcommand{\ceil}[1]{\,\left\lceil #1 \right\rceil\,}%
 \newcommand{\dd}{{\rm d}}%
 \newcommand{\ds}[1]{\displaystyle{#1}}%
 \newcommand{\equalby}[1]{{#1 \atop {= \atop \vphantom{\huge A}}}}%
 \newcommand{\expo}[1]{\,{\rm e}^{#1}\,}%
 \newcommand{\fermi}{\,{\rm f}}%
 \newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,}%
 \newcommand{\half}{{1 \over 2}}%
 \newcommand{\ic}{{\rm i}}%
 \newcommand{\iff}{\Longleftrightarrow}
 \newcommand{\imp}{\Longrightarrow}%
 \newcommand{\isdiv}{\,\left.\right\vert\,}%
 \newcommand{\ket}[1]{\left\vert #1\right\rangle}%
 \newcommand{\ol}[1]{\overline{#1}}%
 \newcommand{\pars}[1]{\left( #1 \right)}%
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\pp}{{\cal P}}%
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,#2\,}\,}%
 \newcommand{\sech}{\,{\rm sech}}%
 \newcommand{\sgn}{\,{\rm sgn}}%
 \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
 \newcommand{\ul}[1]{\underline{#1}}%
 \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}$
Linearize the equations:
$$
\dot{x} \sim \half\pars{x - y}\,,\quad\dot{y} \sim y - 2x\qquad\imp\qquad
{\dot{x} \choose \dot{y}}
\sim
\pars{%
\begin{array}{rr}
\half & -\,\half
\\[2mm]
-2 & 1
\end{array}}{x \choose y}
$$
The matrix eigenvalues $\braces{\lambda}$ are given by: $\ds{\pars{\half - \lambda}\pars{1 - \lambda} - 1 = 0\quad\imp\quad\lambda^{2} - {3 \over 2}\,\lambda + \half = 0}$:
$$
\lambda_{\pm} = {3/2 \pm \root{\vphantom{\large A}9/4 - 2} \over 2} = {3/2 \pm 1/2 \over 2} = 
\left\lbrace%
\begin{array}{rcl}
\lambda_{+} & = & 1
\\[1mm]
\lambda_{-} & = & \half
\end{array}\right.
$$
You'll have some behaviors $\propto \expo{t}$ and $\propto \expo{t/2}$ which 'blows up' $x$ and $y$ when $x, y \sim 0$. 
