I'm given the following system of linear equations which is 3x3 $$ \left[\begin{array}{rrr|r} 1 & a+b & a-b & ab \\ 0 & a^2-9 & 0 & a-3 \\ 0 & 0 & b+2 & b-2 \end{array}\right] $$
I need to determine all values for $a$ and $b$ so that the system is
(a) inconsistent;
(b) consistent with exactly one solution;
(c) consistent with infinitely many solutions.
For (a) i got $a = -3; b = -2$
But for(b) and (c) i tried different methods but still do not get how to get the final values for $a$ and $b$.
If the matrix is invertible, then the system is consistent with exactly one solution. (b) consists of any values $a$ and $b$ that make the matrix invertible. While those that don't are in (a) and (c).
I know that determinants are generally frowned on these days, but the one for this matrix is trivial, and makes case (b) obvious. Which just leaves dividing the two cases that make the determinant $0$ between (a) and (c).
The question as posed is only to determine whether the system
$$\left[\begin{array}{rrr|r} 1 & a+b & a-b & ab \\ 0 & a^2-9 & 0 & a-3 \\ 0 & 0 & b+2 & b-2 \end{array}\right]$$
can have values $a,b$ such that it is $(i)$ inconsistent, $(ii)$ consistent with a unique solution, or $(iii)$ consistent with an infinite number of solutions. This is the trichotomy of matrices, and these are the only possible options.
$(i)$ is easily resolved (as you have done): have either $a=-3$ or $b=-2$ or both.
$(iii)$ is also easily resolved: simply find the set of values $a,b$ such that one of the rows in our matrix is all-zero. There is only one such value, namely $a=3$ (with $b\neq -2$ for consistency).
Because we are covering all the possible options of the solution space of the matrix, we have $(ii)$ taking over the remaining space. So this space is for $a\neq \pm 3,\, b\neq -2$.
