being $\mathbf{w}$ a vector, how do I calculate the derivative of $\mathbf{w}^T\mathbf{w}$?

Let's say that I have a vector $\mathbf{w}$.

How can I calculate the derivative in the following expression?

$\frac{\mathrm{d}}{\mathrm{d}\mathbf{w}}\mathbf{w}^T\mathbf{w}$

Update: found these useful definitions

• Start with a simple example and build from there. The pattern should be clear. Sep 14, 2014 at 20:46
• What's the meaning of $\dfrac{d}{d\mathbf{w}}$? Sep 14, 2014 at 20:47
• Traditionally, it means "a vector whose elements are the derivative of what follows with respect to each coordinate." For instance, if $\mathbf{w} = (x,\ y)$, then $\frac{d}{d\mathbf{w}} \left( x^2+y^2\right) = (2x,\ 2y)$. Sep 14, 2014 at 20:50
• @egreg, I am pretty sure this is common notation in optimization, various computer-related fields. Meant o be a directional derivative, so here it would be $w \cdot \nabla f,$ with $f(w) = w \cdot w$ this time. Sep 14, 2014 at 20:52
• Yes, in essence. Which makes sense. Because $w^Tw$ is the closest thing you can get to $x^2$ in a vector sense, and hence you would expect the derivative to look something like $2x$, as well. Sep 14, 2014 at 20:57

In order to find the derivative let us calculate the h-linear term, $$$$\begin{split} (w+h)^T(w+h)-w^Tw&=(w^T+h^T)(w+h)-w^Tw\approx\\ &\approx h^Tw+w^Th= (h^Tw)^T+w^Th=2w^Th \end{split}$$$$ Hence the derivative is: $2w^T$. Rather than $2w$. Indeed the gradient is not really a column- but a row-vector (or covector, or dual vector).

It is easier to see it in component form.

Let $\hat{x_i}$ be the unit vector in the $i$-th direction, then we can express a vector as

$$\mathbf{w}=\sum_{i=1}^{n}w_i \hat{x_i} \tag{1}$$

And

$$\frac{d}{d\mathbf{w}}=\sum_{i=1}^{n}\hat{x}^T_i \frac{d}{dw_i} \tag{2}$$

So

$$\mathbf{w}^T \mathbf{w}=\sum_{i=1}^{n}w_i^2 \tag{3}$$

$$\frac{d}{d\mathbf{w}}(\mathbf{w}^T \mathbf{w})=\sum_{i=1}^{n}2w_i\hat{x}^T_i =2\mathbf{w}^T\tag{4}$$

EDIT: I made a minor correction ($\hat{x}_i$ to $\hat{x}^T_i$)in (2) and (4) based on rych's suggestion. Now the final results is $2\mathbf{w}^T$

• should be $2w^T$, no?
– rych
Oct 8, 2014 at 7:00
• I think that you are right!
– mike
Oct 8, 2014 at 8:31
• @rych Where is the final transpose coming from? Nov 5, 2014 at 1:54