1
$\begingroup$

Let's say that I have a vector $\mathbf{w}$.

How can I calculate the derivative in the following expression?

$\frac{\mathrm{d}}{\mathrm{d}\mathbf{w}}\mathbf{w}^T\mathbf{w}$

Update: found these useful definitions

$\endgroup$
7
  • $\begingroup$ Start with a simple example and build from there. The pattern should be clear. $\endgroup$
    – Emily
    Sep 14, 2014 at 20:46
  • $\begingroup$ What's the meaning of $\dfrac{d}{d\mathbf{w}}$? $\endgroup$
    – egreg
    Sep 14, 2014 at 20:47
  • 1
    $\begingroup$ Traditionally, it means "a vector whose elements are the derivative of what follows with respect to each coordinate." For instance, if $\mathbf{w} = (x,\ y)$, then $\frac{d}{d\mathbf{w}} \left( x^2+y^2\right) = (2x,\ 2y)$. $\endgroup$
    – Emily
    Sep 14, 2014 at 20:50
  • $\begingroup$ @egreg, I am pretty sure this is common notation in optimization, various computer-related fields. Meant o be a directional derivative, so here it would be $w \cdot \nabla f,$ with $f(w) = w \cdot w$ this time. $\endgroup$
    – Will Jagy
    Sep 14, 2014 at 20:52
  • 1
    $\begingroup$ Yes, in essence. Which makes sense. Because $w^Tw$ is the closest thing you can get to $x^2$ in a vector sense, and hence you would expect the derivative to look something like $2x$, as well. $\endgroup$
    – Emily
    Sep 14, 2014 at 20:57

2 Answers 2

3
$\begingroup$

In order to find the derivative let us calculate the h-linear term, $$\begin{equation} \begin{split} (w+h)^T(w+h)-w^Tw&=(w^T+h^T)(w+h)-w^Tw\approx\\ &\approx h^Tw+w^Th= (h^Tw)^T+w^Th=2w^Th \end{split} \end{equation}$$ Hence the derivative is: $2w^T$. Rather than $2w$. Indeed the gradient is not really a column- but a row-vector (or covector, or dual vector).

$\endgroup$
1
$\begingroup$

It is easier to see it in component form.

Let $\hat{x_i}$ be the unit vector in the $i$-th direction, then we can express a vector as

$$\mathbf{w}=\sum_{i=1}^{n}w_i \hat{x_i} \tag{1}$$

And

$$\frac{d}{d\mathbf{w}}=\sum_{i=1}^{n}\hat{x}^T_i \frac{d}{dw_i} \tag{2}$$

So

$$\mathbf{w}^T \mathbf{w}=\sum_{i=1}^{n}w_i^2 \tag{3}$$

$$\frac{d}{d\mathbf{w}}(\mathbf{w}^T \mathbf{w})=\sum_{i=1}^{n}2w_i\hat{x}^T_i =2\mathbf{w}^T\tag{4}$$

EDIT: I made a minor correction ($\hat{x}_i$ to $\hat{x}^T_i$)in (2) and (4) based on rych's suggestion. Now the final results is $2\mathbf{w}^T$

$\endgroup$
4
  • 1
    $\begingroup$ should be $2w^T$, no? $\endgroup$
    – rych
    Oct 8, 2014 at 7:00
  • 1
    $\begingroup$ I think that you are right! $\endgroup$
    – mike
    Oct 8, 2014 at 8:31
  • $\begingroup$ @rych Where is the final transpose coming from? $\endgroup$
    – alonso s
    Nov 5, 2014 at 1:54
  • $\begingroup$ @alonsos. I made a minor change in my answer. Please take a look. Thanks $\endgroup$
    – mike
    Nov 5, 2014 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.