Finding values for matrix elements that produce one, infinetly or no solutions System of three equations with three unknowns. Augmented:
$\begin{pmatrix} a & (a-1) & (a-2) & b\\ 0 & (a-3) & (a-4) & 1\\ 0 & 0 & (a-5) & 0 \end{pmatrix}$
Problem: For what values of a and b does the augmented matrix have (i) one solution, (ii) infinitely many solutions, or (iii) no solution.
I've been strungling with this and similar problems for a while now. Is there a particular algoritim for solving these kinds of problem?
I know that each equation describes a plane, and I know what it means to have one, infinitely or no solutions. I just find it difficult to find the correct a and b values for solving the problem. Especially since it tends to get messy when row reducing.
(I'm not looking for the answer, just ideas for procedures and possibly ways to think about the problem.)
 A: HINT: (for a strictly computational procedure) For an rref augmented matrix to have no solution, it has a nonzero number in the last column, and zeroes in all other columns. For an augmented matrix to have infinitely many solutions, it must be consistent, and the number of columns of variables is greater than the number of nonzero rows. What can you say about matrices with unique solutions? It must be consistent, but the number of variables is equal to the number of nonzero rows. What does this tell you about $a,b$?
A: Your matrix is in augmented form, i.e. $[A|b]$. The matrix $A$ is in upper triangle form and so the determinant is given by the product of its diagonal elements
$\det(A) = a(a-3)(a-5)$
The system has exactly one solution if $\det(A) \ne 0$. 
To check, when it has no or infinitely many solution, look back at the augmented matrix.
Looking at the last row:
$(a-5)x_3 = 0$. 
If we want to be able to choose $x_3$ freely (to get more than one unique solution) we need $a=5$.
Finally assume that $a \ne5$ and therefore $x_3 = 0$ and look at the second row. With the assumptions taken, this tells us:
$(a-3)x_2 = 1$.
This only has a solution if $a \ne 3$. Therefore $a=3$ gives the case with no solutions. By using the same argument for the first line you also find that $a=0$ yields no solution.
A: There are solutions if and only $\;\DeclareMathOperator\rk{rank}\rk A=\rk [A\mid b]$,. When that is the case, the solutions are an affine subspace of dimension $3-\rk A$.
Now $\det A=a(a-3)(a-5)$, hence


*

*If $a\neq 0,3,5$, $A$ has maximal rank $3$, so $\rk [A\mid b]=3$, and there is exactly $1$ solution.

*If $a=0$, the augmented matrix is
$$\begin{bmatrix}
0&-1&-2&b\\0&-3&-4&1\\0&0&-5&0
\end{bmatrix}\rightsquigarrow\begin{bmatrix}
0&0&0&-b+\frac13\\0&-3&-4&1\\0&0&-5&0
\end{bmatrix} $$
This row reduction shows $\rk A=2$, hence the system has no solution if $a=0, b\ne \dfrac13$, has a subspace of solutions of dimension $1$ if $a=0,b=\dfrac13$.

*If $a=3$, the augmented matrix is
$$\begin{bmatrix}
3&2&1&b\\0&0&-1&1\\0&0&-2&0
\end{bmatrix}\rightsquigarrow\begin{bmatrix}
3&2&1&b\\0&0&-1&1\\0&0&0&-2
\end{bmatrix} $$
Thus, $\rk A=2,\enspace\rk [A\mid b]=3$: the system has no solution.

*If $a=5$, the augmented matrix is
$$\begin{bmatrix}
5&4&3&b\\0&2&1&1\\0&0&0&0
\end{bmatrix} $$
Again the space of solutions has dimension $1$.

