Let's say that I have a vector $\mathbf{w}$.
How can I calculate the derivative in the following expression?
$\frac{\mathrm{d}}{\mathrm{d}\mathbf{w}}\mathbf{w}^T\mathbf{w}$
Update: found these useful definitions
$\begingroup$
$\endgroup$
7
-
$\begingroup$ Start with a simple example and build from there. The pattern should be clear. $\endgroup$– EmilySep 14, 2014 at 20:46
-
$\begingroup$ What's the meaning of $\dfrac{d}{d\mathbf{w}}$? $\endgroup$– egregSep 14, 2014 at 20:47
-
1$\begingroup$ Traditionally, it means "a vector whose elements are the derivative of what follows with respect to each coordinate." For instance, if $\mathbf{w} = (x,\ y)$, then $\frac{d}{d\mathbf{w}} \left( x^2+y^2\right) = (2x,\ 2y)$. $\endgroup$– EmilySep 14, 2014 at 20:50
-
$\begingroup$ @egreg, I am pretty sure this is common notation in optimization, various computer-related fields. Meant o be a directional derivative, so here it would be $w \cdot \nabla f,$ with $f(w) = w \cdot w$ this time. $\endgroup$– Will JagySep 14, 2014 at 20:52
-
1$\begingroup$ Yes, in essence. Which makes sense. Because $w^Tw$ is the closest thing you can get to $x^2$ in a vector sense, and hence you would expect the derivative to look something like $2x$, as well. $\endgroup$– EmilySep 14, 2014 at 20:57
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
In order to find the derivative let us calculate the h-linear term, $$\begin{equation} \begin{split} (w+h)^T(w+h)-w^Tw&=(w^T+h^T)(w+h)-w^Tw\approx\\ &\approx h^Tw+w^Th= (h^Tw)^T+w^Th=2w^Th \end{split} \end{equation}$$ Hence the derivative is: $2w^T$. Rather than $2w$. Indeed the gradient is not really a column- but a row-vector (or covector, or dual vector).
$\begingroup$
$\endgroup$
4
It is easier to see it in component form.
Let $\hat{x_i}$ be the unit vector in the $i$-th direction, then we can express a vector as
$$\mathbf{w}=\sum_{i=1}^{n}w_i \hat{x_i} \tag{1}$$
And
$$\frac{d}{d\mathbf{w}}=\sum_{i=1}^{n}\hat{x}^T_i \frac{d}{dw_i} \tag{2}$$
So
$$\mathbf{w}^T \mathbf{w}=\sum_{i=1}^{n}w_i^2 \tag{3}$$
$$\frac{d}{d\mathbf{w}}(\mathbf{w}^T \mathbf{w})=\sum_{i=1}^{n}2w_i\hat{x}^T_i =2\mathbf{w}^T\tag{4}$$
EDIT: I made a minor correction ($\hat{x}_i$ to $\hat{x}^T_i$)in (2) and (4) based on rych's suggestion. Now the final results is $2\mathbf{w}^T$
