# Least squares and pseudo-inverse

Let $b\in \mathbb{R}^m$,$A\in M_{m\times n}(\mathbb{R})$ with $m>n$ and $rank(A)=n$, and the element $x^*\in \mathbb{R}^m$ solution of least squares of $Ax=b$.

i) Show that $r^*=b-Ax^*\in N(A^T)$ where $N(A)$ denotes the null space.

ii) Find the pseudo-inverse of $A$ and warrants your answer

What I did

Let $z^*=Ax^*\in R(A)$, so by orthogonal decomposition theorem $b=z^*+r^*$ where $z^*\in R(A)$ and $r^*\in N(A^T)$ then $r^*=b-z^*=b-Ax^*$.

But in the part ii) I don't know how to do.

• Again $A^{-1}$ doesn't make any sense for non-square matrices! – Ehsan M. Kermani May 23 '15 at 2:12
• do you mean that $x^* = A^{\dagger}b$, where $A^{\dagger}$ is the pseudoinverse of $A$? – etothepitimesi May 23 '15 at 2:37
• @el.Salvador I made a mistake, $x^*$ it's just the solution of least squares – Roland May 23 '15 at 2:42
• @askazy, the solution of $Ax=b$ in the least squares sense is exactly $x^* = A^{\dagger}b$. – etothepitimesi May 23 '15 at 2:45
• Please stop using the tag "self-learning" in all your questions. That tag is for questions that is about how to self-study. Just because you came across a problem while reading something of your own does not mean that the problem is about self-learning. – Henning Makholm May 23 '15 at 14:12

## 2 Answers

you can show that $rank(A^\top A) = rank(A) = n.$ define $x^* = (A^\top A)^{-1}A^\top b$ that is $A^\top A x^* = A^\top b.$ the pseudo inverse of $A$ is $(A^\top A)^{-1}A^\top.$

1 Given the general least squares problem, $$\lVert \mathbf{A} x - b \rVert_{2}^{2} > 0$$ and the constraint $$\lVert \mathbf{A} x - b \rVert_{2}^{2} < \lVert b \rVert_{2}^{2}$$ ($b$ is not in the null space), we can decompose the problem as $$\color{blue}{\mathbf{A}x} = \color{blue}{{b}_{\mathcal{R}\left( \mathbf{A} \right)}} + \color{red} {{b}_{\mathcal{N}\left( \mathbf{A}^{*} \right)}}.$$ This resolves the data vector into its $\color{blue}{range}$ space and $\color{red}{nullspace}$ components. The derivation is shown in How does the SVD solve the least squares problem?

The residual error vector is the component of the data vector which resides in the null space: $$\color{red}{r\left(x_{LS}\right)} = \color{blue}{\mathbf{A}x} - ( \color{blue}{{b}_{\mathcal{R}\left( \mathbf{A} \right)}} - \color{red} {{b}_{\mathcal{N}\left( \mathbf{A}^{*} \right)}}) = \color{red} {{b}_{\mathcal{N}\left( \mathbf{A}^{*} \right)}}$$ 2 The aforementioned post may be your answer.