8
$\begingroup$

I do see how SVD can be understood in terms of rotating. But it is hard for me to understand SVD in terms of change of basis.

So, let me start from Diagonalization to clarify my question. When $B=\{v_1, \dots, v_n\}$ where $v_i$ are eigenvectors, $A=PDP^T$. Then $PDP^Tx$ means that $x$ becomes $[x]_B$ (that is, $P^Tx$) and scaling operates on $[x]_B$ (that is, D) and then the result is transformed into the original coordinate (that is, P).

On the other hand, understanding SVD in terms of change of basis is really hard for me. Let me say that $A=U\Sigma V^T$. Then, given $U\Sigma V^Tx$, $V^T$ transforms x into the coordinate system consisting of column vectors of V. Scaling operates in that coordinate system ($\Sigma$). But after that, how can one transform the vector under coordinate system governed by V into the original coordinate system? $U$ is different from $V$!

Thank you!

$\endgroup$
2
  • 3
    $\begingroup$ An important point is that the SVD really asks you to pick the V basis for the domain and the U basis for the codomain. This discrepancy makes sense because the SVD makes sense even for rectangular matrices. In that pair of bases, the effect of $A$ is to send the $i$th axis vector in the domain to $\sigma_i$ times the $i$th axis vector in the codomain. $\endgroup$
    – Ian
    Apr 16, 2018 at 14:12
  • $\begingroup$ @ Ian: So the coordinate regarding the basis of V is transformed into the coordinate regarding the basis of U when $\Sigma$ is applied. Thank you so much for your comment! $\endgroup$
    – 12345678
    Apr 16, 2018 at 14:36

1 Answer 1

10
$\begingroup$

I posted this to a previous related question:

Let's play with this a bit.

Let $ A \in \mathbb{R}^{m \times n} $. Then we know it has a singular value decomposition $ A = U \Sigma V^T $ where $ U \in \mathbb{R}^{m \times m} $ and $ V \in \mathbb{R}^{n \times n} $ are orthogonal ($ U^T U = U U^T = I $ and $ V^T V = V V^T = I $) and $ \Sigma $ has nonnegative diagonal elements, ordered from largest to smallest.

Now consider $ y = A x $.

$ x = I x = V V^T x $. So, $ V^T x $ gives you the coefficients of $ x $ when using the columns of $ V $ as the basis.

$ y = I y = U U^T y $. So, $ U^T y $ gives you the coefficients of $ y $ when using the columns of $ U $ as the basis.

So, $ U U^T y = A V V^T x $. Equivalently, $ (U^T y) = (U^T A V) (V^T x) = \Sigma (V^T x) $.
So, if you view $ x $ in the right orthonormal basis (given by the columns of $ V $) and $ y $ in the right orthonormal basis (given by the columns of $ U $), then the (non-square) matrix that relates them becomes diagonal.

A truly beautiful result.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .