Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Short question:

How can I calculate $\dfrac{\partial A}{\partial L}$ where $A = \|Lx\|^2_2= x^TL^TLx$?

Is it $\dfrac{\partial A}{\partial L}=2Lx^tx$?

Long question:

I want to calculate the gradient of a Mahalanobis distance. More specifically, I like to calculate the gradient of $A$ in terms of $L$, ($\frac{\partial A}{\partial L}$).

$$A = \|Lx\|^2_2= x^TL^TLx$$

I expand the equation and calculate the gradient element by element and it seems it should be something like $\frac{\partial A}{\partial L}=2Lx^tx$. But, it's very slow! Would you please confirm the correctness of answer? and help me find a faster approach?

Thanks

share|improve this question
    
Hint to check your solution: what happens when L is a scalar? –  leonbloy Mar 22 '12 at 18:44

3 Answers 3

Let $f(L) = Lx $ and then from the Wikipedia article on matrix calculus:

$$ \frac{\partial ||Lx||^{2}}{\partial L} = 2\cdot{} ||Lx||\cdot{}\frac{\partial}{\partial L}(||f(L)||)$$ $$ = 2\cdot{}||Lx||\cdot{}\biggl[\frac{\partial}{\partial f}(||f(L)||)\biggr]^{T}\cdot{}\frac{\partial f}{\partial L}$$ $$ = 2\cdot{} ||Lx|| \cdot{} \biggl[\frac{(Lx)^{T}}{||Lx||}\biggr]^{T}\cdot{}x^{T} $$ $$ = 2\cdot{}Lxx^{T}.$$

Edit -- I updated to include missing transpositions.

share|improve this answer

Another way is to go back to the definition of the derivative. Let us fix $x$, and let $H$ be any $n \times n$ matrix. Then:

$$A (L+H,x) = ((L+H)x,(L+H)x) = (Lx,Lx) + 2(Lx,Hx)+(Hx,Hx)$$

$$A (L+H,x) = A (L,x) +2(Lx,Hx)+o (\|H\|).$$

Hence, $\frac{\partial A}{\partial L} (L,x)$ is a linear form which maps a matrix $H$ to $2(Lx,Hx)$. Moreover, you can check that for any vectors $x$ and $y$ in $\mathbb{R}^n$ and any $n \times n$ matrix $M$,

$$ (x,My)_{\mathbb{R}^n} = (x y^T,M)_{\mathcal{M}_{n,n} (\mathbb{R})},$$

where the former scalar product is taken over $\mathbb{R}^n$ and the later over $\mathcal{M}_{n,n} (\mathbb{R})$. Hence:

$$2(Lx,Hx)_{\mathbb{R}^n} = (2Lx x^T,H)_{\mathcal{M}_{n,n} (\mathbb{R})} = \left( \frac{\partial A}{\partial L},H \right)_{\mathcal{M}_{n,n} (\mathbb{R})}.$$

share|improve this answer

It's inded tedious to do it by hand.

See first that $$\frac{\partial {\bf L x }}{\partial L_{ij}}= {\bf P^{ij} x}$$

where ${\bf P^{i,j}}$ is the "singleentry matrix": it is zero everywhere except in the entry $i,j$ where it's 1. (This can be checked, an also deduced from the product rule, knowing that $\frac{\partial {\bf L }}{\partial L_{ij}}= {\bf P^{ij}}$)

And similarly: $$\frac{\partial {\bf x^t L^t }}{\partial L_{ij}}= {\bf x^t P^{ji} }$$

Then, applying the product rule:

$$\frac{\partial { \bf x^t L^t L x }}{\partial L_{ij}}= { \bf x^t L^t} {\bf P^{ij} x} + {\bf x^t P^{ji} L x } = 2 \; { \bf x^t L^t} {\bf P^{ij} x} $$

Now the Matrix Cookbook comes to the rescue ("The single-entry matrix is very useful when working with derivatives of expressions involving matrices" - page 52), by using formula 431 (page 53) we get the result:

$$ \frac{\partial { \bf x^t L^t L x }}{\partial L_{ij}}=  2 ({\bf L x x^t })_{ij} \Rightarrow \frac{\partial { \bf x^t L^t L x }}{\partial {\bf L}}= 2 {\bf L x x^t }$$

BTW, this is formula 69 in the same Cookbook.

Edited: I had messed some indexes, I think it's ok now.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.