Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose I have a ($n\times 1$) vector $\mathbf v$ and a ($n\times n$) matrix $\mathbf M$ and I want to compute the derivative w.r.t. some $x$. Both $\mathbf v$ and $\mathbf M$ depend on the scalar $x$.

I need to compute $\Large \frac{\partial \mathbf{v^TMv}}{\partial x}$

My initial thoughts, based on standard differentiation, is to proceed as such:

$\Large \frac{\partial \mathbf{v^TMv}}{\partial x} = \Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} + \Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x}$

However, ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = (\mathbf{M}+\mathbf{M}^T)\mathbf{v}$ yields a ($n\times1$) vector, while $\Large \frac{\partial \mathbf{v}}{\partial x}$ is a ($n\times1$) vector as well.

The problem is that I can't multiply a ($n\times1$) vector by a ($n\times1$) vector by either using outer or inner product.

What am I doing wrong?

How should I get my derivative?

PS: note that in the current font it may be hard to distinguish vectors ($\mathbf{v}$) from scalars ($x$).

share|improve this question

3 Answers 3

up vote 3 down vote accepted

Your calculation is slightly misleading. The situation becomes clear when you understand the derivative as linear map.

For example, the deriviative $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{v}$ is a linear functional $\Bbb{R}^{n} \to \Bbb{R}$ given by

$$ \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} = \left[ \mathbf{u} \mapsto \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \mathbf{u} \right] $$

In particular, we can identify $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{v}$ as $(1 \times n)$-vector, though this may obscures the true nature of the derivative. Anyway, this gives

$$ \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} = \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \frac{\partial \mathbf{v}}{\partial x}. $$

Similarly, $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{M}$ is a linear functional $\mathrm{Mat}_{n\times n}(\Bbb{R}) \to \Bbb{R}$ given by

$$ \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} = \left[ \mathbf{N} \mapsto \mathbf{v}^{T} \mathbf{N} \mathbf{v} \right]. $$

It then follows that

$$ \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x} = \mathbf{v}^{T} \frac{\partial \mathbf{M}}{\partial x} \mathbf{v}. $$

This gives

\begin{align*} \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial x} &= \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} + \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x} \\ &= \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \frac{\partial \mathbf{v}}{\partial x} + \mathbf{v}^{T} \frac{\partial \mathbf{M}}{\partial x} \mathbf{v}, \end{align*}

which can also be obtained by applying product rule (and in fact it makes calculation easier).

share|improve this answer
    
This final formula is correct. It seems to me that where I went wrong was in using the rule ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = {\color{red}{(\mathbf{M}+\mathbf{M}^T)\mathbf{v}}}$ (as stated in the Matrix Cookbook, which is otherwise a very handy and reliable tool). Actually this rule should have been ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = \mathbf{v}^T(\mathbf{M}+\mathbf{M}^T)$, I conclude from your post. –  Angelorf Feb 27 at 11:58
    
It seems to me that the chain rule is not applicable in the matrix context. Your last derivation suggests that ${\Large \frac{d \mathbf{v^TMv}}{d \mathbf{M}} \frac{d\mathbf{M}}{x}} = \mathbf{v}^T{\Large \frac{d \mathbf{M}}{d \mathbf{x}}} \mathbf{v} $, while ${\Large \frac{\partial \mathbf{v^TMv}}{d \mathbf{M}}}$ should give an $(n\times n)$ matrix and $\Large \frac{d\mathbf{M}}{x}$ should be an $(n\times n)$ matrix. Multiplying these two we would give an $(n\times n)$ matrix instead of the scalar given by $\mathbf{v}^T{\Large \frac{d\mathbf{M}}{d \mathbf{x}}} \mathbf{v}$ –  Angelorf Feb 27 at 12:09
    
@Angelorf, That's why we define derivative as linear map in multivariable calculus context. In $\Bbb{R}^{n}$, a linear functional can be identified as a row vector thanks to the existence of the inner product structure. However, in $\mathrm{Mat}_{n\times n}(\Bbb{R})$, we cannot identify the linear functional $\partial \mathrm{v}^{T} \mathrm{M}\mathrm{v} / \partial \mathrm{v}$ as a matrix unless you specify an inner product structure for $(n\times n)$ matrices. –  sos440 Feb 27 at 12:23

What you have to use is the derivative of a product, not the chain rule. So $$ \frac{d (\mathbf v^T\mathbf M\mathbf v)}{d x}=\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\frac{d (\mathbf M\mathbf v)}{d x}=\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\left(\frac{d (\mathbf M)}{d x}\mathbf v+\mathbf M\frac{d\mathbf v}{dx}\right)\\ =\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\frac{d (\mathbf M)}{d x}\mathbf v+\mathbf v^T\,\mathbf M\frac{d\mathbf v}{dx} $$

share|improve this answer
    
So why is the chain rule not applicable here? Or what else was I doing wrong? –  Angelorf Feb 27 at 10:13
    
Your final formula is correct, but it could be more compact. Also you don't explain where I went wrong. I'm sorry to say you did not provide much insight. –  Angelorf Feb 27 at 11:55

@Angelorf, you do not understand one word about this type of problem.

  1. Of course, your formula $\dfrac{∂(v^TMv)}{∂x}=\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}+\dfrac{∂(v^TMv)}{∂M}\dfrac{∂M}{∂x}$ is absolutely correct: $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂v}{∂x}=h\rightarrow h^TMv+v^TMh=v^T(M+M^T)h$ ; you can handle, in the same way, the second part of the formula.

  2. In particular the derivative of the function $f:v\rightarrow v^TMv$ is $Df_v: h\rightarrow v^T(M+M^T)h$ and not $(M+M^T)v$ or $v^T(M+M^t)$ as you seem to believe and "as stated in the Matrix Cookbook, which is otherwise a very handy and reliable tool". In fact, it is a very bad book because many people (mentioning no names) copy out the formulas without understanding.

That you saw in this book is the gradient $\nabla_v(f)$, the vector that is defined by the formula $(\nabla_v(f),h)=Df_v(h)$ where $(k,h)=k^Th$ (the standard scalar product). By identification $\nabla_v(f)=(M+M^T)v$.

  1. When one knows this type of calculation, it can be directly treated as a product of 3 terms (as Martin did). Your first comment "it could be more compact" is not serious. Your second one "I'm sorry to say you did not provide much insight" will discourage people that want to teach you Mathematics
share|improve this answer
    
I have very limited understanding of the problem at hand, indeed. It now is quite a bit clearer, thanks to your answer. It provided me with much insight. You say $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂v}{∂x}$ but shouldn't it be $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂(v^TMv)}{∂x}$? Can you maybe explain how to interpret the $h$? –  Angelorf Feb 28 at 16:08
    
Maybe I'm parsing your formula wrongly. Should it be interpreted as $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}: \left\{ x\rightarrow \dfrac{∂v}{∂x}=h \right\} \rightarrow ...$ or as $\left\{\dfrac{∂(v^TMv)}{?v}\dfrac{∂v}{∂x}: x\rightarrow \dfrac{∂v}{∂x}\right\}=\left\{h \rightarrow ...\right\}$ ? –  Angelorf Feb 28 at 16:11
1  
@Angelorf, it is the composition of $2$ applications $f\circ g$ (there are $2$ arrows) where $g:x\rightarrow \dfrac{∂v}{∂x}$ (let $h=\dfrac{∂v}{∂x}$) and $f:h\rightarrow v^T(M+M^T)h$. Thus $(f\circ g)(x)=\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}=v^T(M+M^T)\dfrac{∂v}{∂x}$. –  loup blanc Feb 28 at 18:38
    
Ah yes. Thanks. The stuff about $\nabla_v(f)$ is kind of out of my leage, but as for the rest it is quite clear how to handle derivatives in the context of matrices. Thanks for the help! –  Angelorf Mar 1 at 0:11

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.