I'm trying to understand a conclusion in [1], [2].

There they state a definition and a theorem:

Definition(5.2 in [1], 2.2 in [2]): For a general linear System $Ax=y$, we say that $\hat{x}$ is a least-squares solution, if $||A\hat{x}-y|| = \min_x ||Ax-y||$. We say that $\hat{x}$ is a minimum least-squares solution, if $\hat{x}$ has the smallest norm among all least-square solutions.

Theorem(5.1 in [1], 2.1 in [2]): Let there exist a matrix $G$ such that $Gy$ is a minimum least-squares solution of a linear system $Ax=y$. Then it is necessarcy and sufficent that $G = A^+$, the Moore-Penrose generalized inverse of matrix A.

Now, both papers want to find a minimum least-squares solution of the linear system $H\beta = T$, that is $$||H\hat{\beta} - T|| = \min_\beta ||H\beta - T||, \qquad (1)$$ where $H \in \mathbb{R}^{N \times \tilde{N}}$, $\beta \in \mathbb{R}^{\tilde{N} \times m}$ and $T \in \mathbb{R}^{N \times m}$.

They argue as follows: "According to the theorem, the minimum least-squares solution of above linear system is $\hat{\beta} = H^+T$, where $H^+$ is the Moore-Penrose generalized inverse of matrix $H$.".

Let be $\hat{\beta}$ the solution found in $(1)$. To apply the theorem, there must exist a matrix $G$ so that $\hat{\beta} = GT$. But can we make sure there always exist such a matrix $G$? From my point of view, I can't see that this always holds, because of the dimensions of $\hat{\beta}$ and $T$.


When does the least squares solution exist?

$$\mathbf{A}x = b$$

$$\mathbf{A} \in \mathbb{C}^{m\times n}_{\rho}, \qquad b \in \mathbb{C}^{m}, \qquad x \in \mathbb{C}^{n}$$

In short, when the data vector $b$ is not in the null space: $$ b\notin\color{red}{\mathcal{N}\left( \mathbf{A}^{*} \right)} $$ The least squares solution is the projection of the data vector $b$ onto the range space $\color{blue}{\mathcal{R}\left( \mathbf{A}\right)}$. A projection will exist provided that some component of the data vector is in the range space.


This post works through a derivation of the least squares solution based upon the singular value decomposition of a nonzero matrix. How does the SVD solve the least squares problem?

Another approach is to start with the guaranteed existence of the singular value decomposition $$ \mathbf{A} = \mathbf{U} \, \Sigma \, \mathbf{V}^{*}, $$ and build the Moore-Penrose pseudoinverse $$\mathbf{A}^{+} = \mathbf{V} \Sigma^{+} \mathbf{U}^{*}.$$ The pseudoinverse is a matrix projection operator onto the range space $\color{blue}{\mathcal{R}\left( \mathbf{A}\right)}$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.