Let $w$ be a $n \times 1$ vector and $\Phi$ be a $n \times n$ matrix. Then the following differentiation rule holds:

$\frac{\delta}{\delta w} w^T \Phi^T \Phi w = 2 \Phi^T \Phi w $

Is there some general principle of vector differentiation that implies this rule? I can verify the rule by writing out the whole matrix product and collecting the terms, but I get no intuitive understanding from doing this. The rule seems very similar to the single variable derivative rule $\frac{d}{dx} ax^2 = 2ax$, which comes from the more general rule $\frac{d}{dx} x^n = nx^{n-1}$. Is there an analogous general rule for vector functions? Is there a way to "just see" the solution without writing out the matrix products?


For several cases, you can follow the steps described in the books:

a) Matrix Differential Calculus with Applications in Statistics and Econometrics (W.E. Shewhart and S. S. Wilks);

b) Complex-Valued Matrix Derivatives (Are Hjørungnes).

For a function $f: \mathbb{R}^n \rightarrow \mathbb{R}$

1) Calculate $\Delta f = f(\mathbf{x}+\mathbf{\Delta x})-f(\mathbf{x})$;

2) Keep only the first order terms (in $\Delta \mathbf{x}$) of $\Delta f$. Here, you will have been constructing the differential of $f$, namely, $∂f$.

3) Change, thus, $\Delta \mathbf{x}$ by $\partial \mathbf{x}$ and you'll have $\partial f = (\cdot)\partial \mathbf{x}$. The quantity $(\cdot)$ is the derivative $\frac{\partial f}{\partial\mathbf{x}}$.

For your example, $f(\mathbf{w}) = \mathbf{w}^{T}\Phi^{T}\Phi\mathbf{w}$. Thus,

1) $\Delta f = (\mathbf{w}+\Delta \mathbf{w})^T\Phi^{T}\Phi(\mathbf{w}+\Delta \mathbf{w}) - \mathbf{w}^{T}\Phi^{T}\Phi\mathbf{w} = \mathbf{w}^T\Phi^{T}\Phi\Delta \mathbf{w} + (\Delta \mathbf{w})^T\Phi^{T}\Phi\mathbf{w} + (\Delta \mathbf{w})^T\Phi^{T}\Phi\Delta \mathbf{w};$

2)$\partial f = \mathbf{w}^T\Phi^{T}\Phi\Delta \mathbf{w} + (\Delta \mathbf{w})^T\Phi^{T}\Phi\mathbf{w};$

3)$\partial f = \mathbf{w}^T\Phi^{T}\Phi\partial \mathbf{w} + (\partial \mathbf{w})^T\Phi^{T}\Phi\mathbf{w}$.

Notice that $\mathbf{w}^T\Phi^{T}\Phi\partial \mathbf{w} = (\partial \mathbf{w})^T\Phi^{T}\Phi\mathbf{w}$. Thus, $\frac{\partial f}{\partial \mathbf{w}} = 2\mathbf{w}^T\Phi^{T}\Phi.$

Actually, your result is the gradient $\nabla_\mathbf{w} f = (\frac{\partial f}{\partial \mathbf{w}})^T$.

  • $\begingroup$ Thank you, this is very helpful, though I must admit I was hoping that the solution would be even simpler :). I'll take a look at your references to learn more. $\endgroup$ – jnalanko Dec 4 '14 at 18:20
  • $\begingroup$ There are more complicated derivatives for which these steps are not enough. Within the books, you will find other techniques for these cases, like the chain rule, similar to the case with one variable. :) $\endgroup$ – Alex Silva Dec 4 '14 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.