# How to apply gradient with respect to a vector

In Deep Learning (adapted from page 108), explaining linear regression as a machine learning algorithm, there is a passage for the solution of this expression:

To minimize $MSE$, we can simply solve for where its gradient is $0$: $$\nabla_{\mathbf w}MSE = 0$$

In addition, $\hat{\mathbf{y}}$ is defined as the prediction of the linear regression (also defined as $\mathbf X \mathbf w$, where $\mathbf{X}$ is the matrix of inputs and $\mathbf{w}$ is the weights vector), while $\mathbf{y}$ is defined as the real output value.

The solution follows this path:

$$\nabla_{\mathbf w}MSE = 0$$ $$\Rightarrow \nabla_{\mathbf w}\frac{1}{m}\lvert\lvert \hat {\mathbf{y}}-{\mathbf{y}}\rvert\rvert_2^2= 0$$ $$\Rightarrow \frac{1}{m} \nabla_{\mathbf w}\lvert\lvert {\mathbf{X}}\mathbf w -{\mathbf{y}}\rvert\rvert_2^2= 0$$ $$\Rightarrow \nabla_{\mathbf w} ( {\mathbf{X}}\mathbf w -{\mathbf{y}} )^{T} ( {\mathbf{X}}\mathbf w -{\mathbf{y}} ) = 0$$ $$\Rightarrow \nabla_{\mathbf w} ( \mathbf{w}^T{\mathbf{X}}^{T}{\mathbf{X}}\mathbf w - 2\mathbf{w}^T{\mathbf{X}}^{T}{\mathbf{y}} + {\mathbf{y}}^{T}{\mathbf{y}} ) = 0$$

Now, the subsequent step is:

$$\Rightarrow ( 2{\mathbf{X}}^{T}{\mathbf{X}}\mathbf w - 2{\mathbf{X}}^{T}{\mathbf{y}}) = 0$$

I think to understand that it takes the vector derivative with respect to $\mathbf{w}$, however I could not find the exact term of this derivative and consequently its rules to carry out the derivative myself (in particular, how to deal with transposed vectors and matrices).

• Well this is a scalar valued function of the weight vector. Look at atmos.washington.edu/~dennis/MatrixCalculus.pdf and in particular propositions 7, 8, 9. Recall that the transpose of a product is equal to the product of the transposes. – James Lea Mar 17 '17 at 20:36

Using matrix transpose notation with vectors often confuses me. So I prefer to expand the norm using an explicit dot product instead \eqalign{ \|z\|^2_2 &= z\cdot z \cr } In this form, finding the differential and the gradient of the norm is straightforward \eqalign{ d\|z\|^2_2 &= 2z\cdot dz \cr \frac{\partial\|z\|^2_2}{\partial z} &= 2z \cr\cr } Now repeat the calculation for $\,\,z=(X\cdot w-y)$ \eqalign{ d\|z\|^2_2 &= 2z\cdot dz \cr &= 2z\cdot (X\cdot dw) \cr &= 2(X^T\cdot z)\cdot dw \cr \cr \frac{\partial\|z\|^2_2}{\partial w} &= 2X^T\cdot z \cr &= 2X^T\cdot(X\cdot w-y) \cr\cr }

Greg's answer already solved the problem. However, I want to specifically address the question about the notation $\nabla_{w}f$.

If you are using "numerator layout" (for me the most logical), $\nabla_{w}f$ simply refers to $\frac{\partial f}{\partial w^{T}}$. $$\frac{\partial}{\partial w^{T}}\left(w^{T}X^{T}Xw - 2w^{T}X^{T}y + y^{T}y\right)\\ =\frac{\partial w^{T}}{\partial w^{T}}X^{T}Xw + w^{T}X^{T}X\frac{\partial w}{\partial w^{T}} - 2\frac{\partial w^{T}}{\partial w^{T}}X^{T}y\\ =X^{T}Xw + (w^{T}X^{T}X)^{T} - 2X^{T}y\\ =2X^{T}Xw - 2X^{T}y$$

If you are using "denominator layout" then $\nabla_w f$ is $\frac{\partial f}{\partial w}$, but the result is the same.

You can read more about layouts at the wikipedia article on matrix calculus.