Consider a linear system of equations $Ax = b$.

  • If the system is overdetermined, the least squares (approximate) solution minimizes $||b - Ax||^2$. Some source sources also mention $||b - Ax||$.

  • If the system is underdetermined one can calculate the minimum norm solution. But it does also minimize $||b - Ax||$, or am I wrong?

But if least squares is also a minimum norm, what is the difference, or the rationale of the different naming?

  • 4
    $\begingroup$ When $Ax=b$ is undetermined (so there are infinitely many solutions), the "least norm" solution is the solution $x$ such that $\|x\|$ is as small as possible. $\endgroup$ – littleO Apr 26 '17 at 17:44

First, it's important to understand that there are different norms. Most likely you're interested in the euclidean norm:

$\| x \|_{2} =\sqrt{\sum_{i=1}^{n}x_{i}^{2}}$

Next, note that minimizing $\| b-Ax \|_{2}^{2}$ is equivalent to minimizing $\| b-Ax \|_{2}$, because squaring the norm is a monotone transform. It really doesn't matter which one you minimize.

If $A$ has full column rank, then there is a unique least squares solution.

However, if $A$ doesn't have full column rank, there may be infinitely many least squares solutions. In this case, we're often interested in the minimum norm least squares solution. That is, among the infinitely many least squares solutions, pick out the least squares solution with the smallest $\| x \|_{2}$. The minimum norm least squares solution is always unique. It can be found using the singular value decomposition and/or the Moore-Penrose pseudoinverse.

  • $\begingroup$ To determine the minimum norm solution, $A$ must be full row rank, right? $\endgroup$ – plasmacel Apr 26 '17 at 17:47
  • 1
    $\begingroup$ No- you can use the Moore-Penrose pseudoinverse to find a minimum norm least squares solution no matter what rank $A$ has. The formulas for the pseudoinverse based on the inverse of $AA^{T}$ or $A^{T}A$ only work if $A$ has full row or column rank respectively. $\endgroup$ – Brian Borchers Apr 26 '17 at 17:48
  • 1
    $\begingroup$ Yeah, I just wanted to note that if $A$ has full row rank, then there are infinitely many (non least squares approximation) solutions, so the minimum norm solution is not an approximation. Thanks by the way, you cleared up some things for me. $\endgroup$ – plasmacel Apr 26 '17 at 17:52

Linear system $$ \mathbf{A} x = b $$ where $\mathbf{A}\in\mathbb{C}^{m\times n}_{\rho}$, and the data vector $b\in\mathbf{C}^{n}$.

Least squares problem

Provided that $b\notin\color{red}{\mathcal{N}\left( \mathbf{A}^{*}\right)}$, a least squares solution exists and is defined by $$ x_{LS} = \left\{ x\in\mathbb{C}^{n} \colon \lVert \mathbf{A} x - b \rVert_{2}^{2} \text{ is minimized} \right\} $$

Least squares solution

The minimizers are the affine set computed by $$ x_{LS} = \color{blue}{\mathbf{A}^{+} b} + \color{red}{ \left( \mathbf{I}_{n} - \mathbf{A}^{+} \mathbf{A} \right) y}, \quad y \in \mathbb{C}^{n} \tag{1} $$ where vectors are colored according to whether they reside in a $\color{blue}{range}$ space or $\color{red}{null}$ space.

The red dashed line is the set of the least squares minimizers.


Least squares solution of minimum norm

To find the minimizers of the minimum norm, the shortest solution vector, compute the length of the solution vectors.

$$ % \lVert x_{LS} \rVert_{2}^{2} = % \Big\lVert \color{blue}{\mathbf{A}^{+} b} + \color{red}{ \left( \mathbf{I}_{n} - \mathbf{A}^{+} \mathbf{A} \right) y} \Big\rVert_{2}^{2} % = % \Big\lVert \color{blue}{\mathbf{A}^{+} b} \Big\rVert_{2}^{2} + \Big\lVert \color{red}{ \left( \mathbf{I}_{n} - \mathbf{A}^{+} \mathbf{A} \right) y} \Big\rVert_{2}^{2} % $$ The $\color{blue}{range}$ space component is fixed, but we can control the $\color{red}{null}$ space vector. In fact, chose the vector $y$ which forces this term to $0$.

Therefore, the least squares solution of minimum norm is $$ \color{blue}{x_{LS}} = \color{blue}{\mathbf{A}^{+} b}. $$ This is the point where the red dashed line punctures the blue plane. The least squares solution of minimum length is the point in $\color{blue}{\mathcal{R}\left( \mathbf{A}^{*}\right)}$.

Full column rank

You ask about the case of full column rank where $n=\rho$. In this case, $$ \color{red}{\mathcal{N}\left( \mathbf{A} \right)} = \left\{ \mathbf{0} \right\}, $$ the null space is trivial. There is no null space component, and the least squares solution is a point.

In other words, $$ \color{blue}{x_{LS}} = \color{blue}{\mathbf{A}^{+} b} $$ is always the least squares solution of minimum norm. When the matrix has full column rank, there is no other component to the solution. When the matrix is column rank deficient, the least squares solution is a line.

  • $\begingroup$ Great answer! What would be your go to reference textbook on this topic? I $\endgroup$ – AimForClarity Jul 16 '17 at 16:38
  • $\begingroup$ This post shows a bit more about equation (1) from Laub's delightful book. $\endgroup$ – dantopa Jul 16 '17 at 18:41
  • 2
    $\begingroup$ Wonderful, thank you. $\endgroup$ – AimForClarity Jul 16 '17 at 23:12
  • $\begingroup$ In "Provided that $b\notin\color{red}{\mathcal{N}\left( \mathbf{A}^{*}\right)}$, a least squares solution exists and is defined by" why are you specifying "Provided that $b\notin\color{red}{\mathcal{N}\left( \mathbf{A}^{*}\right)}$"? Seems to be unnecesary. $\endgroup$ – user598716 Apr 11 '20 at 11:57
  • $\begingroup$ @user598716 You touch on a delicate point. If the data vector is in the null space then we have the trivial solution. You have a strong argument. Perhaps the qualification could be ignored. Perhaps it could be clarified as leading to the trivial solution. Your insight furthers the discussion. $\endgroup$ – dantopa Apr 12 '20 at 18:58

If a system is overdetermined, there is no solution and thus we may want to find $x$ such that $||Ax-b||$ is as small as it can be (as there is no way to make $||Ax-b||=0$).

On the other hand, if the system is underdetermined, there are infinitely many solutions and thus one can find a solution of minimal norm and this is called the minimum norm solution.

  • 2
    $\begingroup$ Keep in mind that an underdetermined system can also be inconsistent. In that case there aren't any exact solutions to $Ax=b$, but there are still solutions that minimize $\| Ax -b \|_{2}$ and among those there is a unique minimum norm least squares solution. $\endgroup$ – Brian Borchers Feb 20 '19 at 5:14
  • 1
    $\begingroup$ @BrianBorchers you are absolutely correct! Thanks for pointing this out! $\endgroup$ – Zuriel Feb 20 '19 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.