Linearization of $C^r$ ODE system Consider the following ODE system
\begin{align*}
\dot{x} &= x^5 + y^3 = f(x\,,y) \\
\dot{y} &= x^3 - y^5 = g(x\,,y)
\end{align*}
Clearly, $(x\,,y) = (0\,,0)$ is the only fixed point. Linearizing
$$
J = 
\begin{bmatrix}
5x^4 & 3y^2 \\
3x^2 & -5y^4
\end{bmatrix}
$$
evaluating at the fixed point will just give a zero matrix. Thus, need to use some other approach to analyze the stability, e.g., Lyapunov function.
However, I can't find any Lyapunov function related to this problem. Since this ODE system is at least $C^3$ in both $x$ and $y$. So I am wondering, for the linearization, is it possible to go all the way to 3rd-order derivative? Maybe something like
$$
\begin{bmatrix}
\frac{\partial^3f}{\partial x^3} & \frac{\partial^3f}{\partial y^3} \\
\frac{\partial^3g}{\partial x^3} & \frac{\partial^3g}{\partial y^3}
\end{bmatrix}
$$
This way the matrix will not be a zero matrix and can find corresponding eigenvalues and eigenvectors.
 A: From
\begin{align*}
\dot{x} &= x^5 + y^3 \\
\dot{y} &= x^3 - y^5
\end{align*}
we have
\begin{align*}
x^3\dot{x} &= x^8 + x^3 y^3 \\
y^3\dot{y} &= x^3y^3 - y^8
\end{align*}
and then
$$
\frac 14\frac{d}{dt}(x^4-y^4) = x^8+y^8
$$
analyzing the behavior along a line $y(t) = \alpha x(t)$ we conclude
$$
\frac 14(1-\alpha^4)\frac{d}{dt}x^4 = (1+\alpha^8)x^8
$$
so depending on $\alpha$ the dynamics along $y(t) = \alpha x(t)$ changes concluding that the origin is unstable.
A: Looking for the central manifold, considering the system
$$
\cases{
\dot x = f(x,y) = x^5+y^3\\
\dot y = g(x,y) = x^3-y^5
}
$$
and considering $y = h(x) = \sum_{k=0}^n a_kx^{k+1}$ we have
$$
h'(x)\dot x = \dot y\Rightarrow h'(x)f(x,h(h))=g(x,h(x))
$$
for $n = 6$ we get two solutions:
$$
h_1(x) = -\frac{2 x^7}{27}-\frac{x^5}{6}-\frac{x^3}{3}-x + O(x^9)\\
h_2(x) = -\frac{2 x^7}{27}+\frac{x^5}{6}-\frac{x^3}{3}+x + O(x^9)
$$
Attached a stream flow plot showing the phase plane in light blue and $h_1(x), h_2(x)$ in red. As can easily be verified, $y=h_1(x)$ has stable flow and $h_2(x)$ has unstable flow.

