Understanding for a system why not asymptotically stable when $\epsilon = 0$ at the origin Consider the system $$x' = (\epsilon x+2y)(z+1)$$ $$y' = (-x+\epsilon y)(z+1)$$
 $$ z' = -z^3$$
Show that the origin is not asymptotically stable when $\epsilon =
    0$.
I am told if we start with z = 0, z remains at 0 and the system reduces to $$x' = 2y$$
$$y' = −x$$ 
(1). My question comes how do we know we need to set z = 0? 
This vector field points normal to  $\begin{bmatrix} x \\ 2y \end{bmatrix}$ and hence tangent to any ellipse $x^2 + 2y^2 = r^2$ . 
(2). Another question is how we derive the ellipse equation here?
Thus given any neighborhood of (0, 0, 0), for r sufficiently small, the solution $$x(t) = \sqrt2 cos t$$ $$y(t) = sin t$$ $$z(t) = 0$$ starts in the given neighborhood and does not approach the origin.
(3). How do we derive the solution to x,y,and z here and how do we see the solution does not approach the origin?
 A: I am not sure about everything you wrote as it seems some information is missing, so you still need to work through some things.
We are given the system:
$$\tag 1 x' = (\epsilon x+2y)(z+1) \\ y' = (-x+\epsilon y)(z+1) \\ z' = -z^3$$
For $\epsilon = 0$, $(1)$ reduces to:
$$\tag 2 x' = 2y(z+1) \\ y' = -x(z+1) \\ z' = -z^3$$
As an aside, note that $z$ is decoupled from the other equations and you can solve for it explicitly and then substitute back into the original system and reduce it from a $3x3$ to a $2x2$.
To have something to compare against, lets numerically do a sample of this system with the IC: $x(0) = 1, y(0) = 2, z(0) = 10$ (a larger $z$ was chosen to see how it affects the solution). Here is a solution plot, notice how fast the curve spirals down and that it appears to settle into an elliptical pattern with $z$ going to zero.

Now, if we wanted to analyze this, one approach would be to find the critical points and depending on those, evaluate the Jacobian at each of those critical points. However, it is clear that the only critical point for $(2)$ is $(x, y, z) = (0, 0, 0)$.
Because of this, we can just drop the nonlinear terms in $(2)$ and write the linearized system as:
$$\tag 3 x' = 2y \\ y' = -x $$
If we numerically solve this we should be expecting an ellipse. Here is plot of the numerical solution of this system:

Lets also do a phase portrait for the system and we expect to see ellipses for varying initial conditions.

Now, we can write $(3)$ in matrix form, find the eigenvalues and the eigenvectors and write the solution for this system. The eigenvalues are purely complex, $\lambda_{1,2} = \pm~ \sqrt{2} ~ i$, which means we have ellipses as solutions. These are given by:
$$x(t) = c_1~ \cos \sqrt{2} t + \sqrt{2}~ c_2~ \sin \sqrt{2} t \\ y(t) = - \dfrac{c_1~ \sin \sqrt{2} t}{\sqrt{2}} + \sqrt{2}~ c_2~ \cos \sqrt{2} t $$
You have to figure out how they came up with their ellipse argument from the items you are discussing in class.
Note, they use the ellipse $x^2 + 2 y^2 = r^2$. If you substitute their solution for $x(t)$ and $y(t)$, what do you notice? $(\sqrt{2} \cos t)^2 + 2 (\sin t)^2 = r^2 \implies 2(\sin^2 t + \cos^2 t) = r^2 \implies r = \sqrt{2}$. Somehow, they are arguing that the vector field points normal to the solution you show and hence tangent to any ellipse (which is what the derived solution above is). 
