# geometrical representation of minimum norm least square solution

Given three equations in two variables is like giving a triangle on the plane. Now I want to describe the minimum norm least square solution $x=A^+b$ in terms of the geometry of the triangle. Here $A+$ denotes Moore-Penrose pseudoinverse.

I consider that in the equations the sum squares of the coefficients are 1. So, for the system of equations \begin{cases}a_{11}x+a_{12}y=b_1\\a_{21}x+a_{22}=b_2 \\a_{31}x+a_{32}y = b_3 \end{cases}, how can I describe the point $A^+b$ geometrically?

I would appreciate any help.

## 1 Answer

Fundamental Theorem of Linear Algebra

A matrix $\mathbf{A}\in\mathbb{C}^{3\times 2}_{\rho}$, induces a row space and a column space with the orthogonal decomposition: \begin{align} % \mathbf{C}^{2} = \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \oplus \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \mathbf{C}^{3} = \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \oplus \color{red} {\mathcal{N} \left( \mathbf{A}^{*} \right)} % \end{align}

Method of least squares

Given an appropriate data vector $b\in\mathbb{C}^{3}$, the linear system $$\mathbf{A} x = b$$ has a least squares solution defined by $$x_{LS} = \left\{ x\in\mathbb{C}^{2} \colon \big\lVert \mathbf{A} x - b \big\rVert_{2}^{2} \text{ is minimized} \right\}$$ The least squares solution is computed using $$x_{LS} = \color{blue}{\mathbf{A}^{+} b} + \color{red}{ \left( \mathbf{I}_{n} - \mathbf{A}^{+} \mathbf{A} \right) y}, \quad y \in \mathbb{C}^{n} \tag{1}$$

Decomposing the data vector

The least squares solution resolves the data vector into $\color{blue}{range}$ and $\color{red}{spaces}$ as $$b = \color{blue}{b_{\mathcal{R}}} + \color{red}{b_{\mathcal{N}}}$$ The portion of the data vector in the range space is $$\color{blue}{x_{LS}} = \color{blue}{\mathbf{A}^{+}b} = \color{blue}{b_{\mathcal{R}}}$$

To draw specific diagrams, pick the case where the rank of the matrix is $\rho=1$.

Column space

The minimization occurs in the column space. Row space

The set of least squares minimizers in (1) is an affine space, represented by the dashed red line in the figure below.

The operator $\color{red}{ \left( \mathbf{I}_{n} - \mathbf{A}^{+} \mathbf{A} \right) y}$ projects the vector $y$ onto the null space $\color{red}{\mathcal{N} \left( \mathbf{A} \right)}$.

The blue point $\color{blue}{x_{LS}}$ is the pseudoinverse solution $$\color{blue}{x_{LS}} = \color{blue}{\mathbf{A}^{+}b}$$ and is the solution of the minimum norm. This is where the affine space of the least squares minimizers $x_{LS}$ intersects the range space, $\color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)}$. 