12
$\begingroup$

The pseudoinverse $A^+$ of A is the matrix for which $x = A^+Ax$ for all x in the row space of A. The nullspace of $A^+$ is the nullspace of $A^T$.

I don't understand this cause the above seems to imply that $A^+=A^T$ which doesn't make sense as $x = A^+Ax$ while $A^TA$ gives a matrix which is not an identity matrix.

Here is the source-Page 2 on "Finding the pseudo Inverse"

$\endgroup$
4
  • 1
    $\begingroup$ Use the SVD of $A$ to express the pseudoinverse of $A$. The relationship should then become clear. $\endgroup$
    – K. Miller
    Nov 24, 2016 at 13:27
  • $\begingroup$ I will get $A^+=VΣ^+U^T$ where matrix $Σ^+$ whose first r rows have $\frac{1}{σ_1}, \frac{1}{σ_2}, ..., \frac{1}{σ_r}$ on the diagonal. But I feel to see why or the intuition behind how that will equate to the transpose. $\endgroup$
    – newbie125
    Nov 24, 2016 at 14:04
  • $\begingroup$ Well $A^T = V\Sigma^TU^T$. They both have the same nullspace, namely, $U_{r+1:m}$, where $A$ is an $m\times n$ matrix. $\endgroup$
    – K. Miller
    Nov 24, 2016 at 16:37
  • $\begingroup$ So the Σ doesn't matter but it is because both consists of the matrix multiplication between V and $U^T$ right? The Σ is more like the scalar product between the two right? $\endgroup$
    – newbie125
    Nov 24, 2016 at 17:47

1 Answer 1

17
$\begingroup$

Define matrix

Start with a matrix $$ \mathbf{A} \in\mathbb{C}^{m\times n}_{\rho} $$

Fundamental Theorem of Linear Algebra

The Fundamental Theorem of Linear Algebra can be expressed as $$ \begin{align} % \mathbf{C}^{n} = \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \oplus \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \mathbf{C}^{m} = \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \oplus \color{red} {\mathcal{N} \left( \mathbf{A}^{*} \right)} % \end{align} $$

Singular value decomposition

The singular value decomposition of the matrix is $$ \begin{align} \mathbf{A} &= \mathbf{U} \, \Sigma \, \mathbf{V}^{*} \\ % &= % U \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}}} & \color{red}{\mathbf{U}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cccc|cc} \sigma_{1} & 0 & \dots & & & \dots & 0 \\ 0 & \sigma_{2} \\ \vdots && \ddots \\ & & & \sigma_{\rho} \\\hline & & & & 0 & \\ \vdots &&&&&\ddots \\ 0 & & & & & & 0 \\ \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{V}_{\mathcal{N}}}^{*} \end{array} \right] \\ % & = % U \left[ \begin{array}{cccccccc} \color{blue}{u_{1}} & \dots & \color{blue}{u_{\rho}} & \color{red}{u_{\rho+1}} & \dots & \color{red}{u_{n}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S}_{\rho\times \rho} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{v_{1}^{*}} \\ \vdots \\ \color{blue}{v_{\rho}^{*}} \\ \color{red}{v_{\rho+1}^{*}} \\ \vdots \\ \color{red}{v_{n}^{*}} \end{array} \right] % \end{align} $$ The connection to the Fundamental Theorem is intimate: $$ \begin{array}{ll} % column \ vectors & span \\\hline % \color{blue}{u_{1}} \dots \color{blue}{u_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \\ % \color{blue}{v_{1}} \dots \color{blue}{v_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{u_{\rho+1}} \dots \color{red}{u_{m}} & \color{red}{\mathcal{N} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{v_{\rho+1}} \dots \color{red}{v_{n}} & \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \end{array} $$

Pseudoinverse matrix

The least squares solution with the SVD produces the pseudoinverse:

$$ \begin{align} \mathbf{A}^{+} &= \mathbf{V} \, \Sigma^{+} \mathbf{U}^{*} \\ % &= % V \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S}^{-1} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % % U \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{U}_{\mathcal{N}}}^{*} \end{array} \right] \\ % \end{align} % $$ $$ \mathbf{A}^{+} \in\mathbb{C}^{n\times m}_{\rho} $$

The subspace decomposition in terms of the pseudoinverse is now explicit.

Adjoint matrix

For comparison, the adjoint matrix is $$ \begin{align} \mathbf{A}^{*} &= \mathbf{V} \, \Sigma^{T} \mathbf{U}^{*} \\ % &= % V \left[ \begin{array}{cc} \color{blue}{\mathbf{V}_{\mathcal{R}}} & \color{red}{\mathbf{V}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % % U \left[ \begin{array}{c} \color{blue}{\mathbf{U}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{U}_{\mathcal{N}}}^{*} \end{array} \right] \\ % \end{align} $$ $$ \mathbf{A}^{*} \in\mathbb{C}^{n\times m}_{\rho} $$


Further reading

How the pseudoinverse solution arises in least squares: How does the SVD solve the least squares problem?; Singular value decomposition proof

Variant forms of the pseudoinverse are presented in What forms does the Moore-Penrose inverse take under systems with full rank, full column rank, and full row rank?; generalized inverse of a matrix and convergence for singular matrix Note the null space relationships with the pseudoinverse.

Fundamental projectors and the pseudoinverse: Least squares solutions and the orthogonal projector onto the column space

$\endgroup$
2
  • $\begingroup$ What does "⊕" indicate? And what exactly is $V_R$ vs $V_N$? $\endgroup$ Mar 15, 2023 at 4:24
  • $\begingroup$ The symbol ⊕ represents a [direct sum] (en.wikipedia.org/wiki/Direct_sum#Examples) between range space elements and null space elements, spaces that only share the point at the origin. The row space $\mathbf{V}$ resolves into range space $\color{blue}{\mathbf{V}_{\mathcal{R}}}$ and null space $\color{red}{\mathbf{V}_{\mathcal{N}}}$ components. $\endgroup$
    – dantopa
    Mar 18, 2023 at 17:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .