I'm working through the textbook mathematics for machine learning and have hit a sticking point in SVD. If the matrix $A \in \mathbb{R}^{m \times n}$ then, I understand how the right and left singular vector $V$ and $U$ are derived from the diagonalization of $A^TA$ and $AA^T$. However, the book states

The last step is to link up all the parts we touched upon so far. We have an orthonormal set of right-singular vectors in $V$ . To finish the construction of the SVD, we connect them with the orthonormal vectors $U$ . To reach this goal, we use the fact the images of the $\mathbf v_i$ under $A$ have to be orthogonal, too. We can show this by using the results from Section 3.4. We require that the inner product between $A\mathbf v_i$ and $A\mathbf v_j$ must be 0 for $i \ne j$ . For any two orthogonal eigenvectors $\mathbf v_i$ , $\mathbf v_j$ , $i \ne j$ , it holds that: $$ (A\mathbf v_i)^T(A\mathbf v_j) = \mathbf v_i^T(A^T A)\mathbf v_j = \mathbf v_i^T(\lambda_j \mathbf v_j ) = \lambda_j \mathbf v_i \mathbf v_j = 0 .$$

My question is, how does the final equation go from $\mathbf v_i^T(A^T A)\mathbf v_j = \mathbf v_i^T(\lambda_j \mathbf v_j )$? How does $A^T A$ end up as a scalar value $\lambda$? Can somebody add a more intuitive explaination to the above?

  • $\begingroup$ What is $\lambda_j$ here? The eigenvalue of $A^T A$ corresponding to $\mathbf{v}_j$? $\endgroup$ – cosmic_philosopher Oct 8 '20 at 11:14

Remember how the $v$s were defined --- they're the eigenvectors of the square matrix $A^T A$, so for each $k = 1, 2, \ldots, n$, there's a number $\lambda_k$ with $$ (A^T A) v_k = \lambda_k. $$ In particular, for $k = j$, we have $$ (A^T A) v_j = \lambda_j. $$

Just to be clear, you've said that they arise from "the diagonalization of $M = A^TA$," and it may not be clear that this makes them eigenvectors. Well, suppose that $$ Q^{-1} M Q = D(\lambda_1, \ldots, \lambda_n) $$ a diagonal matrix with numbers we'll call "$\lambda$s" on the diagonal. Then multiplying through by $Q$ we get $$ MQ = QD $$ If we call the first column of $Q$ by the name $v_1$, then this can be read as saying, by equating the first column of each side, that $$ Mv_1 = Q \pmatrix{\lambda_1\\0\\ \vdots \\ 0} = \lambda_1 v_1. $$ and similarly for other columns. In other words, when we diagonalized $M$, the diagonalizing matrix $Q$ has the property that its columns (which we call $v_1, v_2, ...$) are each eigenvectors, with eigenvalues being the corresponding entries of the diagonal matrix $D$.

  • $\begingroup$ Thank you John. I'd clearly overlooked the most basic concept of $Ax = \lambda x$, which in this case is $(A^T A)x = \lambda x$ $\endgroup$ – Grant B Oct 8 '20 at 11:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.