# Error in linear regression

In linear regression we have $$Ax=b$$. Since the equality is an approximate equality, an error vector is used, that is, $$Ax+e=b$$. We know that using the least square method (to minimize the squared sum of the elements of $$e$$) the best $$x$$ is given by: $$x=A^+b$$ where the plus sign represents the pseudoinverse: $$A^+=(A^TA)^{-1}A^T$$. Depending on $$A$$ and $$b$$, there must be some error which is often nonzero as in linear regression we are doing a non-perfect curve estimation. However, $$e=b-Ax$$ which is $$e=b-AA^+b$$ and since $$AA^+=I$$ always holds, the error is always zero, that is, $$e=b(I-AA^+)=b(I-I)=zero$$. Why is that? I think the error vector should not be zero regardless of $$A$$ and $$b$$. Can one explain this to me. Thank you!

## 1 Answer

No.

In linear regression the matrix $$A$$ has more rows than columns, and hence (according to the "Definition" section of the wikipedia article) $$AA^+$$ is not the identity matrix, but rather the projection matrix onto the column space of $$A$$.

The normal equations folded into the formula $$x=A^+b$$ forces the fitted error vector $$e=b-AA^+b$$ to perpendicular to the column space of $$A$$, so the analysis of variance (or Pythagorean theorem) $$\|b\|^2=\|Ax\|^2+\|e\|^2$$ holds

• Assume the system has more columns than the rows, then would the $AA^+$ be the identity matrix? – Mahdi Rouholamini Nov 23 at 3:00