Differential equation, Stability , Lyapunov function Given a system of differential equations 
\begin{eqnarray}
x'&=&2y(z-1)\\
y'&=&-x(z-1) \quad (1)\\
z'&=&xy
\end{eqnarray}
Note that $u_0$=(0,0,0) is an equilibrium point of the system. Let $V(x,y,z)=x^2+2y^2$ is a Lyapunov function for (0,0,0). Since $V(u_0)=0$, $V(u)> 0$ for $u\neq u_0$ and $\frac{d}{dt} V(x,y,z)=0$ for $u\neq u_0$ then  $u_0$=(0,0,0) is stable by Lyapunov theorem.
But I don't understand that $u_0$=(0,0,0) is stable, infact it might be unstable. 
Here is my argument:
Let $r(t)=(x(t),y(t),z(t))$ be the flow  of (1). Since $\frac{d}{dt} V(x,y,z)=0$ then $r'(t)\cdot \nabla V=0$. Hence, $r(t)$ always be in a surface $V(x,y,z)=x^2+2y^2=C$ (A cylinder). If we make the radius of cylinder smaller then $x$ and $y$ component of $r(t)$ tend to 0 but $z$ component of $r(t)$ can be infinite since the height of cylinder is infinite. So, $r(t)$ can be far away from (0,0,0) and $u_0$=(0,0,0) is unstable.
Where is the error of my argument? Can we guarantee that $z$ component of $r(t)$ can't go to infinity? Can we esplain that $u_0$=(0,0,0) is stable without Lyapunov?
Thanks a lot.
 A: The solution stays bounded, and $z$ undergoes periodic motion. The reason for this is symmetry: If you replace $x$ by $-x$ and $t$ by $-t$, the system is unchanged. Therefore, while $(x,y)$ travels once around its ellipse $x^2+2y^2=\text{constant}$ from $x=0$, $y>0$ back to the same point, $z$ must have returned to its original value.
I know that $x$, $y$ traverses the entire ellipse because the first two equations are just a variation of $x'=2y$, $y'=-x$, which has that property. More precisely, introduce some new time variable $\tau$. Write $x_\tau$ etc for the derivative wrt $\tau$ and $x_t$ etc for the derivative wrt $t$, then $x_\tau=x_t\tau_t$, so if $\tau$ is defined by $\tau_t=z-1$, then the first two equations become $x_\tau=2y$, $y_\tau=-x$. This works fine so long as $z<1$. But we are interested in stability at the origin, and if we start close to the origin then $z$ has no chance to grow as big as $1$ in the time it takes to traverse the ellipse.
A: It is true that $(0,0,0)$ is Lyapunov stable.
The general idea has been suggested in other solutions, here are the truly gory details.
Let $p(t) = (x(t), y(t), z(t))$.
We need to establish that a unique solution exists for all time. Let $r>0$ and let $K_r = \overline{B}(0,r)$. Since the system is smooth, it is Lipschitz on any bounded set. 
Let $p(t_0) \in  B(0,\frac{1}{2}r)$ be an initial condition at time $t_0$. By the existence and uniqueness theorem, a solution exists in $K_r$ for some small time interval $[t_0,t_1]$. As above, we notice that $x(t)^2+2 y(t)^2=x_0^2+ y_0^2$, and hence $(x(t),y(t))$ remains bounded. Since $\dot{z} = x y$, we have $|z(t)-z(t_0)| \leq \frac{1}{2}(x_0^2+ y_0^2)(t-t_0)$. Consequently, we see that for any given $t_1$ we can choose $r$ large enough so that $p(t) \in K_r$ for all $t\in [t_0,t_1]$. It follows that the solution exists and is unique for all $t \geq 0$.
Now we will establish that $t \mapsto p(t)$ is periodic. We know that $(x(t),y(t))$ lies on the ellipse $x^2+2y^2 = V_0$, where $V_0 = x_0^2+2 y_0^2$. First we will show that $(x(t),y(t))$ traverses the entire ellipse without $z(t)$ growing 'too much'. Then we will show that $z(t)$ is periodic as well.
We know that $|z(t)| \leq  |z(t_0)|+\frac{1}{2}V_0 (t-t_0)$. Hence if $|z(t_0)| < \frac{1}{4}$ and $(t-t_0) < \frac{1}{2 V_0}$, then $|z(t)| < \frac{1}{2}$. In particular, for any $T>0$, if $V_0$ and $|z(t_0)|$ are 'small enough', then $|z(t)| < \frac{1}{2}$ for any $t \in [t_0,t_0+T]$.
Since $(x(t),y(t))$ lie on the ellipse, we can write $x(t) = \sqrt{V_0} \cos \theta(t)$, $y(t) = \sqrt{\frac{v_0}{2}} \sin \theta(t)$ for some $C^1$ function $\theta$. Differentiating gives
\begin{eqnarray}
\dot{x}(t) &=& -\sqrt{V_0}\sin \theta(t) \dot{\theta}(t) &=& -2 \sqrt{\frac{v_0}{2}} \sin \theta(t) (1-z) \\
\dot{y}(t) &=& \sqrt{\frac{v_0}{2}} \cos \theta(t) \dot{\theta}(t)  &=& \sqrt{V_0} \cos \theta(t) (1-z)
\end{eqnarray}
from which it follows that $\dot{\theta}(t) = \sqrt{2} (1-z)$. Now choose $T = 4 \sqrt{2} \pi$, which gives a neighborhood $U$ of $(0,0,0)$ such that if $p(t_0) \in U$, then $|z(t)| < \frac{1}{2}$ for any $t \in [t_0, t_0+T]$. Then we have $\dot{\theta}(t) \geq \frac{1}{\sqrt{2}}$, and hence $\theta(t_0+T)-\theta(t_0) \geq \frac{1}{\sqrt{2}} T = 4\pi$  (I choose $4 \pi$ so I can find a point where the positive $x$-axis is crossed and still have $2 \pi$ 'left to go'). Hence $(x(t),y(t))$ traverses the entire ellipse in some $[t_0,t_0+\delta] \subset [t_0, t_0+T]$. (It does not yet follow that $(x(t),y(t))$ is periodic.)
Let $\tau_0 \geq t_0$ be the first time at which $(x(t),y(t))$ crosses the positive $x$-axis, let $\tau_1 > \tau_0$ be the first time at which $(x(t),y(t))$ crosses the positive $y$-axis, and let $\Delta = \tau_1 -\tau_0$. For $t \in [\tau_0+\Delta, \tau_0+2\Delta]$, define $\tilde{x}(t) = -x(2\tau_0+2 \Delta-t)$, $\tilde{y}(t) = y(2\tau_0+2 \Delta-t)$ and $\tilde{z}(t) = z(2\tau_0+2 \Delta-t)$. It is straightforward to verify that $\tilde{p} = (\tilde{x},\tilde{y},\tilde{z})$ satisfies the differential equation with initial condition $\tilde{p}(\tau_0+\Delta)=p(\tau_0+\Delta)$.
By uniqueness, we have $p(t) = \tilde{p}(t) $ for $t \in [\tau_0+\Delta, \tau_0+2\Delta]$, and hence $(x(\tau_0+2\Delta), y(\tau_0+2\Delta),z(\tau_0+2\Delta))=(-x(\tau_0), y(\tau_0),z(\tau_0))$ (note that $y(\tau_0) = 0$ by choice of $\tau_0$).
Now we repeat the process for the remaining semi-ellipse. For $t \in [\tau_0+2\Delta, \tau_0+4\Delta]$, define $\tilde{x}(t) = x(2\tau_0+4 \Delta-t)$, $\tilde{y}(t) = -y(2\tau_0+4 \Delta-t)$ and $\tilde{z}(t) = z(2\tau_0+4 \Delta-t)$. Repeating the above procedure, we obtain $p(t) = \tilde{p}(t) $ for $t \in [\tau_0+2\Delta, \tau_0+4\Delta]$, and hence $(x(\tau_0+4\Delta), y(\tau_0+4\Delta),z(\tau_0+4\Delta))=(x(\tau_0), -y(\tau_0),z(\tau_0)) = p(t_0)$ (note that $y(\tau_0) = 0$ by choice of $\tau_0$).
It follows by uniqueness that $p(t+4 \Delta) = p(t)$ for all $t\geq t_0$. Furthermore, $|z(t)| \leq |z(t_0)| +\frac{1}{2}V_0 4 \Delta$ for all $t\geq t_0$.
Hence for all $\epsilon > 0$, there is a neighborhood $U$ of $(0,0,0)$ such that if $p(t_0) \in U$, then $\|p(t)\| <\epsilon$ for all $t \geq t_0$.
A: At a specific height $z$ the equations $x'=2y(z-1)$ and $y'=-x(z-1)$ (looking at only the $x,y$ coordinates) make the trajectory go around an ellipse $x^2+2y^2=C$. (By this I mean ignoring where the $z$ coordinates are, i.e. looking at the projection of the trajectory onto the $xy$ plane.) 
As the curve goes around the ellipse, the signs on $x,y$ go through the usual things in the four quadrants, so that $xy$ is first positive, then $0$, then negative, then $0$ and so on. So maybe this accounts for $z$ not going "off to infinity" by means of $z'=xy$, since because of the alternating signs on $xy$ the $z$ coordinate will oscillate rather than escape to infinity. Of course a better check would be to actually find the trajectories in closed form, but I don't see that easily. Consider this answer as only a possible explanation of why the trajectory might not escape to infinity in the $z$ direction.
