7
$\begingroup$

The pseudoinverse $A^+$ of A is the matrix for which $x = A^+Ax$ for all x in the row space of A. The nullspace of $A^+$ is the nullspace of $A^T$.

I don't understand this cause the above seems to imply that $A^+=A^T$ which doesn't make sense as $x = A^+Ax$ while $A^TA$ gives a matrix which is not an identity matrix.

Here is the source-Page 2 on "Finding the pseudo Inverse"

$\endgroup$
  • 1
    $\begingroup$ Use the SVD of $A$ to express the pseudoinverse of $A$. The relationship should then become clear. $\endgroup$ – K. Miller Nov 24 '16 at 13:27
  • $\begingroup$ I will get $A^+=VΣ^+U^T$ where matrix $Σ^+$ whose first r rows have $\frac{1}{σ_1}, \frac{1}{σ_2}, ..., \frac{1}{σ_r}$ on the diagonal. But I feel to see why or the intuition behind how that will equate to the transpose. $\endgroup$ – newbie125 Nov 24 '16 at 14:04
  • $\begingroup$ Well $A^T = V\Sigma^TU^T$. They both have the same nullspace, namely, $U_{r+1:m}$, where $A$ is an $m\times n$ matrix. $\endgroup$ – K. Miller Nov 24 '16 at 16:37
  • $\begingroup$ So the Σ doesn't matter but it is because both consists of the matrix multiplication between V and $U^T$ right? The Σ is more like the scalar product between the two right? $\endgroup$ – newbie125 Nov 24 '16 at 17:47
6
$\begingroup$

Define matrix

Start with a matrix $$ \mathbf{A} \in\mathbb{C}^{m\times n}_{\rho} $$

Fundamental Theorem of Linear Algebra

The Fundamental Theorem of Linear Algebra can be expressed as $$ \begin{align} % \mathbf{C}^{n} = \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \oplus \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \mathbf{C}^{m} = \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \oplus \color{red} {\mathcal{N} \left( \mathbf{A}^{*} \right)} % \end{align} $$

Singular value decomposition

The singular value decomposition of the matrix is $$ \begin{align} \mathbf{A} &= \mathbf{U} \, \Sigma \, \mathbf{V}^{*} \\ % &= % U \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}}} & \color{red}{\mathbf{U}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cccc|cc} \sigma_{1} & 0 & \dots & & & \dots & 0 \\ 0 & \sigma_{2} \\ \vdots && \ddots \\ & & & \sigma_{\rho} \\\hline & & & & 0 & \\ \vdots &&&&&\ddots \\ 0 & & & & & & 0 \\ \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{V}_{\mathcal{N}}}^{*} \end{array} \right] \\ % & = % U \left[ \begin{array}{cccccccc} \color{blue}{u_{1}} & \dots & \color{blue}{u_{\rho}} & \color{red}{u_{\rho+1}} & \dots & \color{red}{u_{n}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S}_{\rho\times \rho} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{v_{1}^{*}} \\ \vdots \\ \color{blue}{v_{\rho}^{*}} \\ \color{red}{v_{\rho+1}^{*}} \\ \vdots \\ \color{red}{v_{n}^{*}} \end{array} \right] % \end{align} $$ The connection to the Fundamental Theorem is intimate: $$ \begin{array}{ll} % column \ vectors & span \\\hline % \color{blue}{u_{1}} \dots \color{blue}{u_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \\ % \color{blue}{v_{1}} \dots \color{blue}{v_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{u_{\rho+1}} \dots \color{red}{u_{m}} & \color{red}{\mathcal{N} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{v_{\rho+1}} \dots \color{red}{v_{n}} & \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \end{array} $$

Pseudoinverse matrix

The least squares solution with the SVD produces the pseudoinverse:

$$ \begin{align} \mathbf{A}^{+} &= \mathbf{V} \, \Sigma^{+} \mathbf{U}^{*} \\ % &= % V \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S}^{-1} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % % U \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{U}_{\mathcal{N}}}^{*} \end{array} \right] \\ % \end{align} % $$ $$ \mathbf{A}^{+} \in\mathbb{C}^{n\times m}_{\rho} $$

The subspace decomposition in terms of the pseudoinverse is now explicit.

Adjoint matrix

For comparison, the adjoint matrix is $$ \begin{align} \mathbf{A}^{*} &= \mathbf{V} \, \Sigma^{T} \mathbf{U}^{*} \\ % &= % V \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % % U \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{U}_{\mathcal{N}}}^{*} \end{array} \right] \\ % \end{align} $$ $$ \mathbf{A}^{*} \in\mathbb{C}^{n\times m}_{\rho} $$


Further reading

How the pseudoinverse solution arises in least squares: How does the SVD solve the least squares problem?; Singular value decomposition proof

Variant forms of the pseudoinverse are presented in What forms does the Moore-Penrose inverse take under systems with full rank, full column rank, and full row rank?; generalized inverse of a matrix and convergence for singular matrix Note the null space relationships with the pseudoinverse.

Fundamental projectors and the pseudoinverse: Least squares solutions and the orthogonal projector onto the column space

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.