$$y'_{1}(t)=ay_{1}(t)+by_{2}(t)$$ $$y'_{2}(t)=-by_{1}(t)+ay_{2}(t)$$
I know my determinant is $a^2+b^2$ and my trace is $2a$. To find my eigenvalues I think I look at $a\pm bi$.
In the $ab$ plane identify all the regions where this system possesses a saddle, a sink, a spiral sink.
How do I go about this?
Consider that $$ (y_1+iy_2)'=(a-ib)(y_1+iy_2) $$ and $$ (y_1-iy_2)'=(a+ib)(y_1-iy_2). $$
You might want to check your calculations, since the equation for eigenvalues, for a 2 by 2 matrix, in terms of its determinant and trace are given in this question as
$$ \lambda = \frac12 \left(\text{tr} \pm \sqrt{\text{tr}^2-4\det}\right). $$
In general it might be easier to solve it with
$$ \det\left(A - \lambda I\right)=0, $$
where $A$ is a matrix, such that $\dot{\vec{y}}=A\vec{y}$, $I$ the identity matrix of the same size as $A$ and $\lambda$ the eigenvalues, which have to be solved for such that the equation is true.
Here is another way of solving this problem, along the lines of my answer here: Trace Determinant Plane Differential Eqns. Let's assume that we are allowed use the trace-determinant plane and the different behavioral regions of it.
Let us first put $A_{a,b}=\begin{pmatrix} a& b\\ -b & a\end{pmatrix}$. Then like the OP states $\operatorname{tr}(A_{a,b})=2a$ and $\det{A_{a,b}}=a^2+b^2$. After the answer I referred to above, let us consider the coordinate change from the $ab$-plane to the $td$-plane
$$\Phi: \mathbb{R}^2\to \mathbb{R}^2,\,\, \begin{pmatrix} a\\b \end{pmatrix}\mapsto \begin{pmatrix} 2a\\a^2+b^2 \end{pmatrix}.$$
As in my previous answer, this coordinate change is natural as $\Phi(a,b)=(\operatorname{tr}(A_{a,b}),\det(A_{a,b}))$.
Next we need to optimize $\Phi$ a bit. First note that $\Phi\begin{pmatrix}a\\-b\end{pmatrix}=\begin{pmatrix}a\\b\end{pmatrix}$, so that $\Phi$ is two-to-one except when $b=0$, in which case it is one-to-one. Next note that $\dfrac{(2a)^2}{4}=a^2\leq a^2+b^2$, so that the image of $\Phi$ is precisely
$$R=\left\{(t,d)\in\mathbb{R}^2\,\left|\, \dfrac{t^2}{4}\leq d\right.\right\}.$$
Thus we have two inverses for $\Phi$, one with the upper half $ab$-plane as the image and another with the lower half $ab$-plane as the image:
\begin{align*} \Phi^{-1}_+:&R \to \mathbb{R}\times \mathbb{R}_{\geq0},\, \begin{pmatrix}t\\d\end{pmatrix}\mapsto \begin{pmatrix}\frac{t}{2}\\\sqrt{d-\frac{t^2}{4}}\end{pmatrix}\\ \Phi^{-1}_-:&R \to \mathbb{R}\times \mathbb{R}_{\leq0},\, \begin{pmatrix}t\\d\end{pmatrix}\mapsto \begin{pmatrix}\frac{t}{2}\\-\sqrt{d-\frac{t^2}{4}}\end{pmatrix} \end{align*}
Here is a humble caricature summarizing the situation (the trace-determinant plane is from Hirsch, Smale and Devaney's Differential Equations, Dynamical Systems, and an Introduction to Chaos (3e, p.64)):
This exactly matches with the OP's conclusion in a comment.
Finally note that in the critical case $\dfrac{t^2}{4}=d$, we have $b=0$, so that $A_{a,b}=A_{\frac{t}{2},0}=\begin{pmatrix}\frac{t}{2}&0\\0&\frac{t}{2}\end{pmatrix}$ is a diagonal matrix; hence accordingly we have either a star sink (if $t<0$), a star source (if $t>0$), or all equilibrium solutions (if $t=0$).
