How to Solve a AU=B System when Determinant of A=0 Let
$$A=\begin{pmatrix} 1 & 5 & -1\\ -2 & -10 & 5\\ -2 & -10 & -1\end{pmatrix}$$
and 
$$B=\begin{pmatrix} 0 \\ 15 \\ -15\end{pmatrix}.$$
To find a vector $U$ such that $A$ maps $U$ to $B$ then you must solve the system $AU = B$.
Isolating for U yields $U = A^{-1}B$
However, when attempting to solve the inverse of $A$ produces all $0$'s in the bottom row during matrix inversion algorithm, otherwise known as a determinant of $0$.
 A: After Gaussian elimination we get a system in echelon form,
$$\begin{pmatrix}1&5&-1\\0&0&3\\0&0&0\end{pmatrix}\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}0\\15\\0\end{pmatrix}.$$
So by the second equation, $z=5$ and by the first $x+5y=5$. This is all you can say.
A: Yes, it is true that $\det(A)=0$. So $A^{-1}$ is meaningless. Nevertheless we can solve $Ax=b$ with $b=(0,15,-15)^T$ by, say, $x=(1,4/5,5)$. For this, we write $x=(x_1,x_2,x_3)$ and obtain three linear equations, namely
$$
x_1+5x_2-x_3= - 2x_1 - 10x_2 + 5x_3 - 15= - 2x_1 - 10x_2 - x_3 + 15=0.
$$
If we substitute $x_3$ from the last equation, we obtain $- x_1 - 5x_2 + 5=0$. With $x_1=5-5x_2$ all equations are satisfied.
A: The most general solution of a square linear system $AX=B$ can be written
$$X=X_0+X_1$$
where :


*

*1) $X_0$ is a particular solution of system $AX=B$.

*2) $X_1$ is the general solution of the associated homogeneous system $AX=0$ (in more abstract terms : the generic element of the kernel of $A$).
In the case at hand, 
1) has solution (see explanations below):
$$X_0=\begin{pmatrix}5\\0\\5\end{pmatrix}.$$
2) has solution
$$X_1=k\begin{pmatrix}-5\\1\\0\end{pmatrix}  \ \text{for any real} \ k.$$
Thus the general solution is 
$$X_0+X_1=\begin{pmatrix}5-5k\\k\\5\end{pmatrix}  \ \text{for any real} \ k.$$
Remark : this formulation "particular solution of... + general solution of ..." is the same for linear differential equations.

Explanation for (1) : There are in fact many solutions ; here is a way to obtain one :
In system 
$$\begin{cases}x&+&5y&-&z&=&0& \ \ (a)\\-2x&-&10y&+&5z&=&15& \ \ (b)\\-2x&-&10y&-&z&=&-15& \ \ (c)\end{cases}$$
Substracting (c) from (b), one gets $z=5$. Plugging this value in (a),(b) or (c) gives the same equation $x+5y=5$. It suffices then to take $x=5$ and $y=0$.
Explanation for (2) : 
In homogeneous system 
$$\begin{cases}x&+&5y&-&z&=&0& \ \ (a)\\-2x&-&10y&+&5z&=&0& \ \ (b)\\-2x&-&10y&-&z&=&0& \ \ (c)\end{cases}$$ 
Substracting (c) from (b), one gets this time $z=0$. Plugging this value in (a),(b) or (c) gives the same equation $x+5y=0$. It suffices then to take $x=-5y$.
Thus the general solution of the homogeneous equation is :
$$\begin{pmatrix}x\\y\\z\end{pmatrix}=\begin{pmatrix}-5y\\y\\0\end{pmatrix}=y\begin{pmatrix}-5\\1\\0\end{pmatrix}.$$
Please note that, in fact, arbitrary constant $k$ is ordinate $y$.
