# Gradient of a Mahalanobis distance

Short question:

How can I calculate $\dfrac{\partial A}{\partial L}$ where $A = \|Lx\|^2_2= x^TL^TLx$?

Is it $\dfrac{\partial A}{\partial L}=2Lx^tx$?

Long question:

I want to calculate the gradient of a Mahalanobis distance. More specifically, I like to calculate the gradient of $A$ in terms of $L$, ($\frac{\partial A}{\partial L}$).

$$A = \|Lx\|^2_2= x^TL^TLx$$

I expand the equation and calculate the gradient element by element and it seems it should be something like $\frac{\partial A}{\partial L}=2Lx^tx$. But, it's very slow! Would you please confirm the correctness of answer? and help me find a faster approach?

Thanks

-
Hint to check your solution: what happens when L is a scalar? –  leonbloy Mar 22 '12 at 18:44

## 3 Answers

Let $f(L) = Lx$ and then from the Wikipedia article on matrix calculus:

$$\frac{\partial ||Lx||^{2}}{\partial L} = 2\cdot{} ||Lx||\cdot{}\frac{\partial}{\partial L}(||f(L)||)$$ $$= 2\cdot{}||Lx||\cdot{}\biggl[\frac{\partial}{\partial f}(||f(L)||)\biggr]^{T}\cdot{}\frac{\partial f}{\partial L}$$ $$= 2\cdot{} ||Lx|| \cdot{} \biggl[\frac{(Lx)^{T}}{||Lx||}\biggr]^{T}\cdot{}x^{T}$$ $$= 2\cdot{}Lxx^{T}.$$

Edit -- I updated to include missing transpositions.

-

It's inded tedious to do it by hand.

See first that $$\frac{\partial {\bf L x }}{\partial L_{ij}}= {\bf P^{ij} x}$$

where ${\bf P^{i,j}}$ is the "singleentry matrix": it is zero everywhere except in the entry $i,j$ where it's 1. (This can be checked, an also deduced from the product rule, knowing that $\frac{\partial {\bf L }}{\partial L_{ij}}= {\bf P^{ij}}$)

And similarly: $$\frac{\partial {\bf x^t L^t }}{\partial L_{ij}}= {\bf x^t P^{ji} }$$

Then, applying the product rule:

$$\frac{\partial { \bf x^t L^t L x }}{\partial L_{ij}}= { \bf x^t L^t} {\bf P^{ij} x} + {\bf x^t P^{ji} L x } = 2 \; { \bf x^t L^t} {\bf P^{ij} x}$$

Now the Matrix Cookbook comes to the rescue ("The single-entry matrix is very useful when working with derivatives of expressions involving matrices" - page 52), by using formula 431 (page 53) we get the result:

$$\frac{\partial { \bf x^t L^t L x }}{\partial L_{ij}}= 2 ({\bf L x x^t })_{ij} \Rightarrow \frac{\partial { \bf x^t L^t L x }}{\partial {\bf L}}= 2 {\bf L x x^t }$$

BTW, this is formula 69 in the same Cookbook.

Edited: I had messed some indexes, I think it's ok now.

-

Another way is to go back to the definition of the derivative. Let us fix $x$, and let $H$ be any $n \times n$ matrix. Then:

$$A (L+H,x) = ((L+H)x,(L+H)x) = (Lx,Lx) + 2(Lx,Hx)+(Hx,Hx)$$

$$A (L+H,x) = A (L,x) +2(Lx,Hx)+o (\|H\|).$$

Hence, $\frac{\partial A}{\partial L} (L,x)$ is a linear form which maps a matrix $H$ to $2(Lx,Hx)$. Moreover, you can check that for any vectors $x$ and $y$ in $\mathbb{R}^n$ and any $n \times n$ matrix $M$,

$$(x,My)_{\mathbb{R}^n} = (x y^T,M)_{\mathcal{M}_{n,n} (\mathbb{R})},$$

where the former scalar product is taken over $\mathbb{R}^n$ and the later over $\mathcal{M}_{n,n} (\mathbb{R})$. Hence:

$$2(Lx,Hx)_{\mathbb{R}^n} = (2Lx x^T,H)_{\mathcal{M}_{n,n} (\mathbb{R})} = \left( \frac{\partial A}{\partial L},H \right)_{\mathcal{M}_{n,n} (\mathbb{R})}.$$

-