$ab$-plane phase portraits I have the following system of equations:
$x'_1=ax_1+bx_3$
$x'_2=bx_2$
$x'_3=-bx_1+ax_3$
I found the solution:
$x_1=c_1e^{at} cos(bt)+c_3 e^{at}sin(bt)$
$x_2=c_2e^{bt}$
$x_3=-c_1e^{at} sin(bt)+c_3 e^{at}cos(bt)$
Now I'm trying to answer this:
Sketch the region in the $ab$–plane where this system has different
types of phase portraits.
I don't even understand what the question is asking. Am I supposed to play around plugging in $0$ for $a$ and $b$?
 A: The system is given as
$$\begin{align}x'_1&=ax_1+bx_3 \\ x'_2&=bx_2 \\ x'_3&=-bx_1+ax_3\end{align}$$
The eigenvalues are given by
$$(b-\lambda ) \left(a^2-2 a \lambda +b^2+\lambda ^2\right) = 0 \implies \lambda_{1, 2, 3} = a-i~ b, a+i~ b, b$$
The solution is given by
$$\begin{align}x_1(t)&= c_1 e^{a t} \cos (b t)+c_3 e^{a t} \sin (b t)\\y_1(t) &= c_2 e^{b t}\\z_1(t) &=  -c_1 e^{a t} \sin (b t) + c_3 e^{a t} \cos (b t)\end{align}$$
Lets investigate system stability using eigenvalues in the different quadrants of the $ab-$plane

*

*Case 1: $a = b = 0$, solution is a constant

*Case 2: $b = 0, a > 0$, solution lies on a line and is unstable

*Case 3: $b = 0, a < 0$, solution lies on a line and is stable

*Case 4: $a <0, b \ne 0$, system is stable

*Case 5: $a > 0, b \ne 0$, system is unstable

Try other cases to see if anything interesting shakes out to complete the analysis, like $a = \pm b$.
For case $1$, we have the solution as
$$X(t) = \begin{pmatrix} x(t) \\ y(t) \\ z(t) \end{pmatrix} = c_1 \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} + c_2 \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} + c_3 \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} $$
For case $2$, we have

For case $3$, we have

For case $4$, we have

For case $5$, we have

Note: Because this system is decoupled, you could have also just stated what $y(t)$ does and then analyzed $x(t), z(t)$ using a $2D$, $ab-$plane, similar to these notes, which is preferred. These notes are also very helpful for bifurcations in general.
