Here's the entire question: Let $A$ be an 8 $\times$ 5 matrix of rank 3, and let $b$ be a nonzero vector in $N(A^T)$.
a) Show that the system $Ax = b$ must be inconsistent. Gonna take a wild stab at this one... If the rank is 3, that means the dimension of the column space is 3. But $A$ has 5 columns, so they are not all linearly independent and therefore $Ax = b$ is inconsistent.
b) How many least squares solutions will the system $Ax = b$ have? Explain.
On previous problems, I found the best least squares linear fit, where the approximation of $x$ was a vector that contained sometimes regular numbers, and sometimes variables. Does this mean that there must be either 1 linear solution or infinite (because you can always find an approximation)? In the example that apparently had an infinite number of least squares solutions, it appeared that one row of $A^TA$ was a constant multiple of another row, leading to a row of zeros in reduced row echelon form. From this problem I know that $A^TA$ is a 5x5 matrix, but I don't think I can prove that any rows are a scalar multiple of other rows, so I'm guessing I have to use some other means of figuring this out.
Sorry if I sound like I have no idea what I'm talking about. Just wanted to try out the problem to my best ability before asking about it.