# Derivative of matrix and vector in $\mathbf {v^TMv}$

Suppose I have a ($n\times 1$) vector $\mathbf v$ and a ($n\times n$) matrix $\mathbf M$ and I want to compute the derivative w.r.t. some $x$. Both $\mathbf v$ and $\mathbf M$ depend on the scalar $x$.

I need to compute $\Large \frac{\partial \mathbf{v^TMv}}{\partial x}$

My initial thoughts, based on standard differentiation, is to proceed as such:

$\Large \frac{\partial \mathbf{v^TMv}}{\partial x} = \Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} + \Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x}$

However, ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = (\mathbf{M}+\mathbf{M}^T)\mathbf{v}$ yields a ($n\times1$) vector, while $\Large \frac{\partial \mathbf{v}}{\partial x}$ is a ($n\times1$) vector as well.

The problem is that I can't multiply a ($n\times1$) vector by a ($n\times1$) vector by either using outer or inner product.

What am I doing wrong?

How should I get my derivative?

PS: note that in the current font it may be hard to distinguish vectors ($\mathbf{v}$) from scalars ($x$).

-

Your calculation is slightly misleading. The situation becomes clear when you understand the derivative as linear map.

For example, the deriviative $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{v}$ is a linear functional $\Bbb{R}^{n} \to \Bbb{R}$ given by

$$\frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} = \left[ \mathbf{u} \mapsto \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \mathbf{u} \right]$$

In particular, we can identify $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{v}$ as $(1 \times n)$-vector, though this may obscures the true nature of the derivative. Anyway, this gives

$$\frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} = \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \frac{\partial \mathbf{v}}{\partial x}.$$

Similarly, $\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v} / \partial \mathbf{M}$ is a linear functional $\mathrm{Mat}_{n\times n}(\Bbb{R}) \to \Bbb{R}$ given by

$$\frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} = \left[ \mathbf{N} \mapsto \mathbf{v}^{T} \mathbf{N} \mathbf{v} \right].$$

It then follows that

$$\frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x} = \mathbf{v}^{T} \frac{\partial \mathbf{M}}{\partial x} \mathbf{v}.$$

This gives

\begin{align*} \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial x} &= \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{v}} \frac{\partial \mathbf{v}}{\partial x} + \frac{\partial \mathbf{v}^{T} \mathbf{M} \mathbf{v}}{\partial \mathbf{M}} \frac{\partial \mathbf{M}}{\partial x} \\ &= \mathbf{v}^{T} (\mathbf{M} + \mathbf{M}^{T}) \frac{\partial \mathbf{v}}{\partial x} + \mathbf{v}^{T} \frac{\partial \mathbf{M}}{\partial x} \mathbf{v}, \end{align*}

which can also be obtained by applying product rule (and in fact it makes calculation easier).

-
This final formula is correct. It seems to me that where I went wrong was in using the rule ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = {\color{red}{(\mathbf{M}+\mathbf{M}^T)\mathbf{v}}}$ (as stated in the Matrix Cookbook, which is otherwise a very handy and reliable tool). Actually this rule should have been ${\Large \frac{\partial \mathbf{v^TMv}}{\partial \mathbf{v}}} = \mathbf{v}^T(\mathbf{M}+\mathbf{M}^T)$, I conclude from your post. – Angelorf Feb 27 '14 at 11:58
It seems to me that the chain rule is not applicable in the matrix context. Your last derivation suggests that ${\Large \frac{d \mathbf{v^TMv}}{d \mathbf{M}} \frac{d\mathbf{M}}{x}} = \mathbf{v}^T{\Large \frac{d \mathbf{M}}{d \mathbf{x}}} \mathbf{v}$, while ${\Large \frac{\partial \mathbf{v^TMv}}{d \mathbf{M}}}$ should give an $(n\times n)$ matrix and $\Large \frac{d\mathbf{M}}{x}$ should be an $(n\times n)$ matrix. Multiplying these two we would give an $(n\times n)$ matrix instead of the scalar given by $\mathbf{v}^T{\Large \frac{d\mathbf{M}}{d \mathbf{x}}} \mathbf{v}$ – Angelorf Feb 27 '14 at 12:09
@Angelorf, That's why we define derivative as linear map in multivariable calculus context. In $\Bbb{R}^{n}$, a linear functional can be identified as a row vector thanks to the existence of the inner product structure. However, in $\mathrm{Mat}_{n\times n}(\Bbb{R})$, we cannot identify the linear functional $\partial \mathrm{v}^{T} \mathrm{M}\mathrm{v} / \partial \mathrm{v}$ as a matrix unless you specify an inner product structure for $(n\times n)$ matrices. – Sangchul Lee Feb 27 '14 at 12:23

What you have to use is the derivative of a product, not the chain rule. So $$\frac{d (\mathbf v^T\mathbf M\mathbf v)}{d x}=\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\frac{d (\mathbf M\mathbf v)}{d x}=\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\left(\frac{d (\mathbf M)}{d x}\mathbf v+\mathbf M\frac{d\mathbf v}{dx}\right)\\ =\frac{d \mathbf v^T}{d x}\,\mathbf M\mathbf v+\mathbf v^T\,\frac{d (\mathbf M)}{d x}\mathbf v+\mathbf v^T\,\mathbf M\frac{d\mathbf v}{dx}$$

-
So why is the chain rule not applicable here? Or what else was I doing wrong? – Angelorf Feb 27 '14 at 10:13
Your final formula is correct, but it could be more compact. Also you don't explain where I went wrong. I'm sorry to say you did not provide much insight. – Angelorf Feb 27 '14 at 11:55

1. Of course, your formula $\dfrac{∂(v^TMv)}{∂x}=\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}+\dfrac{∂(v^TMv)}{∂M}\dfrac{∂M}{∂x}$ is absolutely correct: $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂v}{∂x}=h\rightarrow h^TMv+v^TMh=v^T(M+M^T)h$ ; you can handle, in the same way, the second part of the formula.
2. In particular the derivative of the function $f:v\rightarrow v^TMv$ is $Df_v: h\rightarrow v^T(M+M^T)h$ and not $(M+M^T)v$ or $v^T(M+M^t)$ as you seem to believe and "as stated in the Matrix Cookbook, which is otherwise a very handy and reliable tool". In fact, it is a very bad book because many people (mentioning no names) copy out the formulas without understanding.
That you saw in this book is the gradient $\nabla_v(f)$, the vector that is defined by the formula $(\nabla_v(f),h)=Df_v(h)$ where $(k,h)=k^Th$ (the standard scalar product). By identification $\nabla_v(f)=(M+M^T)v$.
I have very limited understanding of the problem at hand, indeed. It now is quite a bit clearer, thanks to your answer. It provided me with much insight. You say $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂v}{∂x}$ but shouldn't it be $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}:x\rightarrow \dfrac{∂(v^TMv)}{∂x}$? Can you maybe explain how to interpret the $h$? – Angelorf Feb 28 '14 at 16:08
Maybe I'm parsing your formula wrongly. Should it be interpreted as $\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}: \left\{ x\rightarrow \dfrac{∂v}{∂x}=h \right\} \rightarrow ...$ or as $\left\{\dfrac{∂(v^TMv)}{?v}\dfrac{∂v}{∂x}: x\rightarrow \dfrac{∂v}{∂x}\right\}=\left\{h \rightarrow ...\right\}$ ? – Angelorf Feb 28 '14 at 16:11
@Angelorf, it is the composition of $2$ applications $f\circ g$ (there are $2$ arrows) where $g:x\rightarrow \dfrac{∂v}{∂x}$ (let $h=\dfrac{∂v}{∂x}$) and $f:h\rightarrow v^T(M+M^T)h$. Thus $(f\circ g)(x)=\dfrac{∂(v^TMv)}{∂v}\dfrac{∂v}{∂x}=v^T(M+M^T)\dfrac{∂v}{∂x}$. – loup blanc Feb 28 '14 at 18:38
Ah yes. Thanks. The stuff about $\nabla_v(f)$ is kind of out of my leage, but as for the rest it is quite clear how to handle derivatives in the context of matrices. Thanks for the help! – Angelorf Mar 1 '14 at 0:11