Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I am reading a section of a book regarding linear regression and came across a derivation that I could not follow.

It starts with a loss function:

$\mathcal{L}(\textbf{w},S) = (\textbf{y}-\textbf{X}\textbf{w})^\top(\textbf{y}-\textbf{X}\textbf{w})$

and then states that "We can seek the optimal $\textbf{w}$ by taking the derivatives of the loss with respect to $\textbf{w}$ and setting them to the zero vector"

$\frac{\partial\mathcal{L}(\textbf{w},S)}{\partial\textbf{w}} = -2\textbf{X}^{\top}\textbf{y} + 2\textbf{X}^\top\textbf{X}\textbf{w} = \textbf{0}$

How is this derivative being calculated? I find that I have no idea how to take the derivative of vector or matrix valued functions, especially when the derivative is with respect to a vector, however I found a pdf ( http://orion.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf ) which appears to address some of my questions, yet my attempts at taking the derivative of the loss function seem to be missing a transpose and thus does not reduce as nicely as the books result.

share|improve this question
add comment

2 Answers

The definition of the derivative can be found at http://en.wikipedia.org/wiki/Fr%C3%A9chet_derivative.

In this case, the derivative can be computed directly by expanding the function: $$\mathcal{L}(w+\delta,S)= \langle y -X(w+\delta), y -X(w + \delta) \rangle = \mathcal{L}(w,S)+2 \langle y -Xw,-X\delta\rangle+ || X \delta||^2.$$ The second term can be written as $2 \langle -X^T(y -Xw),\delta\rangle = \langle -2X^Ty +2X^TXw,\delta\rangle$, from which it follows that the (Fréchet) derivative is $ \frac{\partial \mathcal{L}(w,S)}{\partial w} = (-2X^Ty +2X^TXw)^T$.

The derivative can also be computed componentwise, but requires more bookkeeping.

The expression you have for the partial is missing a transpose.

share|improve this answer
add comment

The notion of derivative seen in single-variable calculus is really just a special case of the more general notion of derivative. Given two Banach spaces (complete normed linear spaces) $X$ and $Y$, we can define the derivative of a function $f:X\to Y$ at a point $a\in X$ as the linear map $A:X\to Y$ such that $$\lim\limits_{x\to a}\frac{\|f(x)-f(a)-A(x-a)\|_Y}{\|x-a\|_X}=0$$ where $\|\cdot \|_X$ is the norm on $X$ and $\|\cdot \|_Y$ is the norm on $Y$. It is a theorem that $A$, if it exists, is unique. If we let $X=Y=\mathbb R$, and note that linear maps $\mathbb R\to\mathbb R$ are of the form $cx$ for some constant $c$, by taking this constant $c$ we get the derivative from single-variable calculus. It can be shown that this general derivative behaves in much the same way as the derivative from SVC, obeying product rule, chain rule, etc.

It's worth noting that the function $\mathcal L$ you have is actually real-valued, not matrix-valued, since it is the dot product of two vectors. The derivative with respect to $w$ at $w$ is the unique linear map $A:\mathbb R^n\to \mathbb R$ such that $$\lim\limits_{x\to w}\frac{|(y-Xx)^T(y-Xx)-(y-Xw)^T(y-Xw)-A(x-w)|}{\|x-w\|}=0$$ which you can verify is what you are given (you are given a vector, which can be considered a map from $\mathbb R^n$ to $\mathbb R$ by taking the dot product of it with other vectors). Of course, guess-and-check is not a good way to do computations. You can compute the derivative for yourself rather than just verifying it by differentiating the functions $w\mapsto y-Xw$ and $w\mapsto (y-Xw)^T$ and applying the product rule, but this is somewhat conceptually difficult because it involves taking derivatives linear-function-valued functions.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.