# Jacobian of the normalization of vector x, with respect to vector x.

Let $$\mathbf{x}$$ $$\in \mathbb{R}^{1 \times N}$$ be a row vector of interest, and let $$\mathbf{y}$$ $$=$$ $$||\mathbf{x}||^{-1} \mathbf{x}$$ denote the normalization of $$\mathbf{x}$$ by the Euclidean / Frobenius norm.

I want to determine the gradient of $$\mathbf{y}$$ with respect to $$\mathbf{x}$$, via the Jacobian: $$\frac{\partial \text{vec}(\mathbf{y})}{\partial \text{vec}(\mathbf{x})} \in \mathbb{R}^{N \times N}$$ .

I saw this older stackexchange post that generalizes to matrices, and note that this transformation is equivalent to the norm-normalization when $$\mathbf{X}$$ is a vector instead of a matrix.

Using greg's answer in that post, I obtain:

Now I feel pretty confident in this answer, but it would be nice to have it verified. I was unable to find a recourse that discussed this Jacobian, or the Jacobian of the inverse norm of the vector.

$$\def\vec{\operatorname{vec}}$$ $$\def\qty#1{\left( #1 \right)}$$ $$\def\d{{\sf d}}$$

There is no need to write the Jacobian matrix as $$\frac{\partial \vec {\bf y}}{\partial \vec {\bf x}}$$ since both $$\bf y$$ and $$\bf x$$ are vectors.

Start by writing $${\bf y} = \lVert {\bf x} \rVert^{-1} {\bf x}$$ in terms of vector components given a basis: $$y_i = \qty{x^j x_j}^{-\frac{1}{2}} x_i.$$ Next apply the differential operator \begin{align} \d y_i & = \qty{x^j x_j}^{-\frac{1}{2}} \d x_i - \frac{1}{2} \qty{x^j x_j}^{-\frac{3}{2}} \qty{\d\qty{x^kx_k}} x_i \\ & = \qty{x^j x_j}^{-\frac{1}{2}} \delta^k_i \d x_k - \frac{1}{2} \qty{x^j x_j}^{-\frac{3}{2}} \qty{x^k \d x_k + \qty{\d x^k} x_k } x_i \\ & = \qty{x^j x_j}^{-\frac{1}{2}} \delta^k_i \d x_k - \frac{1}{2} \qty{x^j x_j}^{-\frac{3}{2}} \qty{2 x^k \d x_k } x_i \\ & = \qty{x^j x_j}^{-\frac{1}{2}} \delta^k_i \d x_k - \qty{x^j x_j}^{-\frac{3}{2}} \qty{ x^k \d x_k } x_i \end{align} Therefore the partial derivative is $$\frac{\partial y_i}{\partial x^k} = \lVert {\bf x} \rVert^{-1} \delta^k_i - \lVert {\bf x} \rVert^{-3} x_i x^k$$ or using invariants $$\frac{\partial {\bf y}}{\partial {\bf x}^\intercal} = \lVert {\bf x} \rVert^{-1} {\bf I} - \lVert {\bf x} \rVert^{-3} {\bf x} {\bf x}^\intercal$$

Using product differential rule: $$d\frac{\textbf{x}}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}}=\frac{d\textbf{x}}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}} + \textbf{x}\:d\frac{1}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}}$$ More precisely on the second term: $$d\frac{1}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}}= d\langle \textbf{x}, \textbf{x}\rangle^{-\frac12}= -\frac{1}{2}\langle \textbf{x}, \textbf{x} \rangle^{-\frac32}d\langle \textbf{x},\textbf{x}\rangle=$$ $$=-\frac{1}{2}\langle \textbf{x}, \textbf{x} \rangle^{-\frac32}\langle 2\textbf{x},d\textbf{x}\rangle= -\langle \textbf{x}, \textbf{x} \rangle^{-\frac32}\langle \textbf{x},d\textbf{x}\rangle= -\frac{\langle \textbf{x},d\textbf{x}\rangle}{\|\textbf{x}\|^3}$$ Now everything together: $$d\frac{\textbf{x}}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}}=\frac{d\textbf{x}}{\sqrt{\langle \textbf{x}, \textbf{x} \rangle}} -\textbf{x}\frac{\langle \textbf{x},d\textbf{x}\rangle}{\|\textbf{x}\|^3}= \frac{d\textbf{x}}{\|\textbf{x}\|} -\frac{\textbf{x}\textbf{x}^\top d\textbf{x}}{\|\textbf{x}\|^3}=$$ $$=\left[\frac{1}{\|\textbf{x}\|}I_n -\frac{1}{\|\textbf{x}\|^3}\textbf{x}\textbf{x}^\top\right]d\textbf{x}$$

A less mechanical approach to the problem is to construct a formula for the Jacobian by determining its eigendecomposition without constructing the matrix itself.

Let $$f(x) = \|x\|^{-1}x$$, then

1. $$D_{x}f(x) = 0$$, where $$D_x$$ is the directional derivative in the $$x$$ direction.
2. If $$y\perp x$$, then $$D_yf(x) = y/\|x\|$$. To see this, notice that $$\frac{x+hy}{\|x+hy\|} = \frac{x+hy}{\sqrt{\|x\|^2+h^2\|y\|^2}} = f(x) + h \frac{y}{\|x\|} + O(h^2).$$

Let $$J(x)$$ denote the Jacobian of $$f$$. We have now shown that $$J(x)x = 0$$ and $$J(x)y = \|x\|^{-1}y$$ for $$y\perp x$$. This implies two things:

1. $$J(x)$$ has an orthonormal diagonalizing basis $$\{x/\|x\|,y_2,\dots,y_n\}$$, where the $$y_k$$'s complete an orthonormal basis of $$\mathbb{R}^n$$. Hence, $$J(x)$$ is symmetric.

2. $$\|x\| J(x)$$ is an orthogonal projector onto the orthogonal compliment of $$\mathrm{span}\{x\}$$.

We are now done and can explicitly write $$J(x) = \frac{1}{\|x\|}P_{x^\perp} = \frac{1}{\|x\|}\left(I - \frac{xx^T}{\|x\|^2}\right).$$