ODE on Sphere. Determining its angular change Consider the ODE system
$$
x_1'=x_2,~~~~~x_2'=-ax_2-bx_1+x_3,~~~~~x_3'=0.
$$
I would like to know its behaviour on the unit sphere $S^2$, i.e. know the change of radius and angle.
So I can write it in polar coordinates and just ignore $x_3$?

 A: $\newcommand{\Neg}{\phantom{-}}\newcommand{\Vec}[1]{\mathbf{#1}}$Edit (based on clarifications in the comments): In matrix form, your system of ODEs is
$$
\left[\begin{array}{@{}c@{}}
x_{1}' \\
x_{2}' \\
x_{3}' \\
\end{array}\right]
= \left[\begin{array}{@{}c@{}}
\Neg 0 & \Neg 1 & 0 \\
-b & -a & 1 \\
\Neg 0 & \Neg 0 & 0 \\
\end{array}\right]
\left[\begin{array}{@{}c@{}}
x_{1} \\
x_{2} \\
x_{3} \\
\end{array}\right]
$$
The standard analytic approach is to diagonalize the coefficient matrix $A$ (or put it into Jordan canonical form if it isn't diagonalizable). The solutions are
$$
\Vec{x}(t) = \exp(tA)\Vec{x}_{0}.
$$
Qualitatively, the eigenspaces of $A$ are fixed directions, i.e., fixed points of the linear action on the projective plane; generalized eigenspaces are invariant sets of the action; and the relative sizes of eigenvalues tell whether a given fixed point or invariant set is attracting or repelling.

Edit II (based on further comments): The eigenvalues of $A$ are $0$ and $\frac{1}{2}(-a \pm \sqrt{a^{2} - 4b})$. If these are all real, the corresponding eigenvectors are $(1, 0, b)$ and $(1, \lambda, 0)$.
For instance, if $b < 0 < a$, then
$$
\frac{-a - \sqrt{a^{2} - 4b}}{2} < 0 < \frac{-a + \sqrt{a^{2} - 4b}}{2}.
$$
The first eigenspace is repelling, the second is a saddle, the third is attracting.
The new plot indicates the flow of your system when $a = 2$ and $b = -4$. The curve you call $C$ is in red. By happy coincidence, the curves align nicely with the curves in your diagram.
In an eigenbasis, $\exp(tA)$ is the diagonal matrix with diagonal entries
$$
e^{\lambda_{1}t} < e^{\lambda_{2}t} < e^{\lambda_{3}t}.
$$
As $t \to \infty$, every point of the sphere (or projective plane) approaches the $\lambda_{3}$ eigenspace. It should be clear geometrically why asking about the change of radius and angle isn't really adequate language to capture the qualitative behavior of your ODE. :)


Not an answer, but too long for a comment.
The plot below shows the vector field (scaled overall by $\tfrac{1}{4}$ to make the arrows easier to see) corresponding to the system
$$
x_{1}' = x_{2},\quad
x_{2}' = -ax_{2} - bx_{1} + x_{3},\quad
x_{3}' = 0
$$
for $a = 0.4$ and $b = 0.2$. The field itself is blue, the tangential component to the sphere is green, and the normal component is red.

