# Intuition behind Jacobian of the SVD

I'm having a little trouble understanding the meaning behind the Jacobian of an SVD. I understand what the Jacobian is, but I don't see how you can derive a Jacobian from the SVD. To me, the SVD is just USV_transpose - I don't see how a matrix can be differentiated, since they don't really seem to be functions of anything.

http://www.ics.forth.gr/_publications/2000_eccv_SVD_jacobian.pdf

I've been looking at the above pdf just to get a better understanding, but equation (7) (the differentiation of the singular values) is really where I get lost.

-

Actually this is not very difficult. Maybe the term "Jacobian of the svd" is a bit misleading since the svd is a function that "outputs" three matrices from one matrix input. Hence the derivative of such a construct would be three four-dimensional objects. So calling this derivative a "Jacobian" is strictly speaking wrong, because the derivative is not a (two dimensional) matrix. Nevertheless, this is what the paper is about.

It can be done because the outputs can be regarded as functions of the elements $a_{nm}$ of the input matrix $\mathbf A$, e.g. for the $i,j$-th element of the $\mathbf U$ matrix you have $$u_{ij} = u_{ij}(a_{11},a_{12},a_{13},\ldots,a_{NM})$$

Hence you can also obtain all partial derivatives (if they exist!) of the output matrix elements from input elements: $$\frac{\partial u_{ij}}{\partial a_{nm}},\frac{\partial s_{ij}}{\partial a_{nm}},\frac{\partial v_{ij}}{\partial a_{nm}}$$ Unfortunately, due to the four-dimensionality a compact notation as for matrix/vector notation is not possible.

However, to make the notation a bit less complicated, the paper uses $$\frac{\partial \mathbf U}{\partial a_{nm}}$$ But it should be clear what this means.

-

Suppose $A=USV^T$ is the SVD of $A$. The Jacobian they are talking about is just the sensitivity of the singular vector matrices $U,S,V$ with respect to changes in the input matrix $A$.

It answers the question: if you change one element of the input matrix $A$ a little bit, how much will each element of the singular vector matrices change?

There is a lack of consistent notation for this sort of thing, because the input space and output space are both higher dimensional spaces of matrices - you can choose to go element by element on the input, or the output, or both, or neither, all leading to different notations. There are different notations even within that. Thus it may be helpful to try reading papers from different sources that use different notation - maybe one will make more sense than others.

I personally prefer the notation in the following paper (SVD sensitivity in section 3.2), which tries to avoid element-by-element notation as much as possible: http://people.maths.ox.ac.uk/gilesm/files/NA-08-01.pdf

Here are a few other papers/presentations using different notations for the eigenvalue sensitivity problem, which is very similar:

Full matrix notation with the matrix as a function of a scalar parameter: http://www.win.tue.nl/casa/meetings/seminar/previous/_abstract051019_files/Presentation.pdf http://alexandria.tue.nl/repository/books/616489.pdf

Element-by-element input, full matrix output notation (also includes second derivatives): http://ftp.cs.nyu.edu/cs/faculty/overton/papers/pdffiles/eighess.pdf

One of the earlier papers on the subject from 1985 which uses an archaic notation that is very confusing to me but might make mores sense to you: http://janmagnus.nl/papers/JRM011.pdf

-