Let $I_d$ denote the $d \times d$ identity matrix and for $X \in \mathbb{R}^{m\times n}$ and $\lambda>0$ I have to prove the following:

$$(X^TX+\lambda I_n)^{-1}X^T=X^T(XX^T+\lambda I_m)^{-1}$$

I think about using Singular Value Decomposition of $X$, because of the $X^TX$ term, so it holds that:

$$X = U\Sigma V^T$$

But I dont know how to applied correctly.

Thanks in advance.

  • $\begingroup$ the SVD is not a bad idea. Why don't you try some additional steps in that direction? $\endgroup$ – Exodd Jul 22 at 16:22
  • $\begingroup$ I reach this result : $(V\Sigma^T \Sigma V^T + \lambda I_n)^{-1}X^T=X^T(U\Sigma \Sigma^T U^T +\lambda I_m)^{-1}$ but i get stuck. $\endgroup$ – Hojas Jul 22 at 16:28
  • $\begingroup$ good. Now notice that $(VAV^T)^{-1} = \overline VA^{-1}\overline V^T$ and continue $\endgroup$ – Exodd Jul 22 at 16:42
  • $\begingroup$ Why $I_m$ and not $I_n$ in the RHS ? $\endgroup$ – Jean Marie Jul 22 at 16:52
  • $\begingroup$ @Exodd what $\overline{V}$ means? $\endgroup$ – Hojas Jul 22 at 16:55

$\newcommand{\diag}{\operatorname{diag}}$You are on the right track using SVD. Just need to carry some annoying algebra: $$ \begin{align} (X^TX+\lambda I_n)^{-1}X^T&=X^T(XX^T+\lambda I_m)^{-1}\\ \iff (V\Sigma^T U^T U\Sigma V^T + \lambda I_n)^{-1} V\Sigma^T U^T &= V\Sigma^T U^T (U\Sigma V^T V\Sigma^T U^T + \lambda I_m)^{-1}\\ \iff (V(\Sigma^T\Sigma + \lambda I_n)V^T)^{-1} V\Sigma^T U^T &= V\Sigma^T U^T (U(\Sigma\Sigma^T + \lambda I_m) U^T)^{-1}\\ \iff V(\Sigma^T\Sigma + \lambda I_n)^{-1}V^T V\Sigma^T U^T &= V\Sigma^T U^T U(\Sigma\Sigma^T + \lambda I_m)^{-1} U^T\\ \iff V(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T U^T &= V\Sigma^T (\Sigma\Sigma^T + \lambda I_m)^{-1} U^T\\ \end{align} $$ Now all that is left is to show that $(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T=\Sigma^T (\Sigma\Sigma^T + \lambda I_m)^{-1}$. Let us explicitly calculate these matrices:

Without loss of generality assume $m < n$, then $\Sigma^T\Sigma = \diag(\sigma_1^2,\dots,\sigma_m^2, 0,0,\dots)$ and $\Sigma\Sigma^T = \diag(\sigma_1^2,\dots,\sigma_m^2)$. Using the fact that inverse of a diagonal matrix is a diagonal matrix with it's diagonal elements inverted, we have $$(\Sigma^T\Sigma + \lambda I_n)^{-1}= \diag^{-1}(\sigma_1^2+\lambda,\dots,\sigma_m^2+\lambda, \lambda,\lambda,\dots) = \diag((\sigma_1^2+\lambda)^{-1},\dots,(\sigma_m^2+\lambda)^{-1}, \lambda^{-1},\lambda^{-1},\dots).$$ Multiplying on left by the $n\times m$ diagonal matrix $\Sigma^T$, we have $$(\Sigma^T\Sigma + \lambda I_n)^{-1}\Sigma^T= \diag_{n\times m}(\sigma_1(\sigma_1^2+\lambda)^{-1},\dots,\sigma_m(\sigma_m^2+\lambda)^{-1}).$$ Note that the extra $\lambda^{-1}$s got killed by the zero rows of $\Sigma^T$.

Similarly for the right hand side: $$(\Sigma\Sigma^T + \lambda I_m)^{-1} = \diag^{-1}(\sigma_1^2+\lambda,\dots,\sigma_m^2+\lambda) = \diag((\sigma_1^2+\lambda)^{-1},\dots,(\sigma_m^2+\lambda)^{-1}).$$ Multiplying on the left by the $n \times m$ diagonal matrix $\Sigma^T$ we have: $$\Sigma^T(\Sigma\Sigma^T + \lambda I_m)^{-1} = \diag_{n\times m}(\sigma_1(\sigma_1^2+\lambda)^{-1},\dots,\sigma_m(\sigma_m^2+\lambda)^{-1}).$$ This concludes the proof.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.