In Deep Learning (adapted from page 108), explaining linear regression as a machine learning algorithm, there is a passage for the solution of this expression:

To minimize $MSE$, we can simply solve for where its gradient is $0$: $$\nabla_{\mathbf w}MSE = 0$$

In addition, $\hat{\mathbf{y}}$ is defined as the prediction of the linear regression (also defined as $\mathbf X \mathbf w$, where $\mathbf{X}$ is the matrix of inputs and $\mathbf{w}$ is the weights vector), while $\mathbf{y}$ is defined as the real output value.

The solution follows this path:

$$\nabla_{\mathbf w}MSE = 0$$ $$\Rightarrow \nabla_{\mathbf w}\frac{1}{m}\lvert\lvert \hat {\mathbf{y}}-{\mathbf{y}}\rvert\rvert_2^2= 0$$ $$\Rightarrow \frac{1}{m} \nabla_{\mathbf w}\lvert\lvert {\mathbf{X}}\mathbf w -{\mathbf{y}}\rvert\rvert_2^2= 0$$ $$\Rightarrow \nabla_{\mathbf w} ( {\mathbf{X}}\mathbf w -{\mathbf{y}} )^{T} ( {\mathbf{X}}\mathbf w -{\mathbf{y}} ) = 0$$ $$\Rightarrow \nabla_{\mathbf w} ( \mathbf{w}^T{\mathbf{X}}^{T}{\mathbf{X}}\mathbf w - 2\mathbf{w}^T{\mathbf{X}}^{T}{\mathbf{y}} + {\mathbf{y}}^{T}{\mathbf{y}} ) = 0$$

Now, the subsequent step is:

$$\Rightarrow ( 2{\mathbf{X}}^{T}{\mathbf{X}}\mathbf w - 2{\mathbf{X}}^{T}{\mathbf{y}}) = 0$$

I think to understand that it takes the vector derivative with respect to $\mathbf{w}$, however I could not find the exact term of this derivative and consequently its rules to carry out the derivative myself (in particular, how to deal with transposed vectors and matrices).

  • $\begingroup$ Well this is a scalar valued function of the weight vector. Look at atmos.washington.edu/~dennis/MatrixCalculus.pdf and in particular propositions 7, 8, 9. Recall that the transpose of a product is equal to the product of the transposes. $\endgroup$ – James Lea Mar 17 '17 at 20:36

Using matrix transpose notation with vectors often confuses me. So I prefer to expand the norm using an explicit dot product instead $$\eqalign{ \|z\|^2_2 &= z\cdot z \cr }$$ In this form, finding the differential and the gradient of the norm is straightforward $$\eqalign{ d\|z\|^2_2 &= 2z\cdot dz \cr \frac{\partial\|z\|^2_2}{\partial z} &= 2z \cr\cr }$$ Now repeat the calculation for $\,\,z=(X\cdot w-y)$ $$\eqalign{ d\|z\|^2_2 &= 2z\cdot dz \cr &= 2z\cdot (X\cdot dw) \cr &= 2(X^T\cdot z)\cdot dw \cr \cr \frac{\partial\|z\|^2_2}{\partial w} &= 2X^T\cdot z \cr &= 2X^T\cdot(X\cdot w-y) \cr\cr }$$


Greg's answer already solved the problem. However, I want to specifically address the question about the notation $\nabla_{w}f$.

If you are using "numerator layout" (for me the most logical), $\nabla_{w}f$ simply refers to $\frac{\partial f}{\partial w^{T}}$. $$\frac{\partial}{\partial w^{T}}\left(w^{T}X^{T}Xw - 2w^{T}X^{T}y + y^{T}y\right)\\ =\frac{\partial w^{T}}{\partial w^{T}}X^{T}Xw + w^{T}X^{T}X\frac{\partial w}{\partial w^{T}} - 2\frac{\partial w^{T}}{\partial w^{T}}X^{T}y\\ =X^{T}Xw + (w^{T}X^{T}X)^{T} - 2X^{T}y\\ =2X^{T}Xw - 2X^{T}y$$

If you are using "denominator layout" then $\nabla_w f$ is $\frac{\partial f}{\partial w}$, but the result is the same.

You can read more about layouts at the wikipedia article on matrix calculus.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.