Stability of zero of a system of equations Given a system
$$
\begin{cases}
\frac{dx}{dt}=ax-y-x^5\\
\frac{dy}{dt}=ay-z-y^5\\
\frac{dz}{dt}=az-x-z^5\\
\end{cases}
$$
We want to study the stability of the zero solution.
We can linearize the system and get a matrix
$$
\begin{pmatrix}
a & -1 & 0\\
0 & a & -1\\
-1 & 0 & a\\
\end{pmatrix}
$$
The eigenvalues are $a-1$, $a+\frac12+\frac{\sqrt{3}}{2}i$, $a+\frac12-\frac{\sqrt{3}}{2}i$.
If $a>-\frac12$, there is an eigenvalue with positive real part, meaning instability.
If $a<-\frac12$, all eigenvalues have negative real part, meaning asymptotic stability.
If $a=-\frac12$, linearization no longer works. I try to find a Lyapunov function. I tested $V=x^2+y^2+z^2$, but this fails to work.

Question: How to study the stability of zero when $a=-\frac12$?

 A: I remember that when eigenvalues with real part are equal to $0$, the system is still Lyapunov stable (there are several types of stability). But I took this course longtime ago and almost forget all. This solution should be easier than the one I propose below.
Now we find the closed-form solution of the linearized system for the case $a =-\frac{1}{2}$.
$$
\begin{cases}
\frac{dx}{dt}=-\frac{1}{2}x-y\\
\frac{dy}{dt}=-\frac{1}{2}y-z\\
\frac{dz}{dt}=-\frac{1}{2}z-x\\
\end{cases}
\Longleftrightarrow  
\begin{cases}
\left(\frac{dx}{dt}+\frac{1}{2}x\right)e^{\frac{t}{2}}=-e^{\frac{t}{2}}y\\
\left(\frac{dy}{dt}+\frac{1}{2}y\right)e^{\frac{t}{2}}=-e^{\frac{t}{2}}z\\
\left(\frac{dz}{dt}+\frac{1}{2}z\right)e^{\frac{t}{2}}=-e^{\frac{t}{2}}x\\
\end{cases}
\Longleftrightarrow  
\begin{cases}
\frac{d}{dt}\left(e^{\frac{t}{2}}x  \right)=-e^{\frac{t}{2}}y\\
\frac{d}{dt}\left(e^{\frac{t}{2}}y  \right)=-e^{\frac{t}{2}}z\\
\frac{d}{dt}\left(e^{\frac{t}{2}}z  \right)=-e^{\frac{t}{2}}x\\
\end{cases}\\ \tag{1}
$$
For the sake of simplicity, let's denote
$$(X(t),Y(t),Z(t)) = \left(e^{\frac{t}{2}}x,  e^{\frac{t}{2}}y,e^{\frac{t}{2}}z\right) $$
Then
$$(1)\Longleftrightarrow  
\begin{cases}
\frac{dX}{dt}=-Y\\
\frac{dY}{dt}=-Z\\
\frac{dZ}{dt}=-X
\end{cases} \tag{2}$$
We have
$$(2) \Longrightarrow  X'''(t) =-X(t) \Longleftrightarrow  X(t)=c_1e^{-t}+c_2e^{\frac{t}{2}}\sin\left(\frac{\sqrt{3}}{2}t\right)+c_3e^{\frac{t}{2}}\cos\left(\frac{\sqrt{3}}{2}t\right)$$
We deduce easily $Y(t)$,$Z(t)$  and also the solution $(x(t),y(t),z(t))$
$$
\begin{cases}
x(t)=c_1e^{-\frac{t}{2}}+c_2\sin\left(\frac{\sqrt{3}}{2}t\right)+c_3\cos\left(\frac{\sqrt{3}}{2}t\right)\\
y(t)=c_4e^{-\frac{t}{2}}+c_5\sin\left(\frac{\sqrt{3}}{2}t\right)+c_6\cos\left(\frac{\sqrt{3}}{2}t\right)\\
z(t)=c_7e^{-\frac{t}{2}}+c_8\sin\left(\frac{\sqrt{3}}{2}t\right)+c_9\cos\left(\frac{\sqrt{3}}{2}t\right)\\
\end{cases} \tag{3}$$
With $(c_i)_{i=1,...,9}$ can be determined from initial condition.
It's my memory is still good, $(3)$ is stable as the solution converges to a stable solution $(x^{*}(t),y^{*}(t),z^{*}(t))$ for $t \to +\infty$
$$\begin{cases}
x^{*}(t)=c_2\sin\left(\frac{\sqrt{3}}{2}t\right)+c_3\cos\left(\frac{\sqrt{3}}{2}t\right)\\
y^{*}(t)=c_5\sin\left(\frac{\sqrt{3}}{2}t\right)+c_6\cos\left(\frac{\sqrt{3}}{2}t\right)\\
z^{*}(t)=c_8\sin\left(\frac{\sqrt{3}}{2}t\right)+c_9\cos\left(\frac{\sqrt{3}}{2}t\right)\\
\end{cases} $$
Then the initial system of equations is Lyapunov stable for $a = -\frac{1}{2}$.
