So I've tried to solve this problem where we are asked to solve the partial derivative of function $\sum(y - Xw)^2$ or $\|y-Xw\|^2$ and then minimize it. I've never done any linear algebra aside some really basic stuff and I can't seem to find any information how to take partial derivative of such a function. I know how to take partial derivative of simple function but not functions with $\||x||^2$ notation.

In this case it would probably help to denote some variable e.g $\ z=y-Xw $ that way we get $\|z\|^2 $. Is this even a right approach? I have no clue what to do after this.

  • $\begingroup$ Write it out explicitly in two dimensions with $y=(y_1,y_2)$ and $w=(w_1,w_2)$ and just take partial derivatives. $\endgroup$ Nov 6 '17 at 18:46
  • $\begingroup$ I don't know if I understand completely what you mean. Could you give some kind of steps how to do this? Thanks a lot. $\endgroup$
    – user499999
    Nov 6 '17 at 19:18
  • $\begingroup$ If $X$ is the identity matrix you'll get $(y_1-w_1)^2 + (y_2-w_2)^2$. If the matrix $X$ is in there, you'll get $\big(y_1-(x_{11}w_1 + x_{12}w_2)\big)^2 + \big(y_2-(x_{21}w_1 + x_{22}w_2)\big)^2$. :) $\endgroup$ Nov 6 '17 at 19:23

An easy way to do it is write it like scalar product and then use the properties of the scalar product. I assume you are working with real matrix and vectors (it really seems that you are dealing with Ordinary Least Squares). \begin{align} \|y-Xw\|^2 &=\langle y-Xw,y-Xw \rangle \\ &=y^Ty-2w^TX^Ty+w^TX^TXw. \end{align} I've just used the fact that the scalar product is bilinear and symmetric. Now just take the derivative and use the product rule: $$\implies -2X^Ty+2X^TXw=0 \\ \implies X^Ty=X^TXw \\ \implies w=(X^TX)^{-1}X^Ty.$$ If you have problem with the derivation in general just write it down the two dimensional case.

  • $\begingroup$ When you use the < and > signs, are you detailing the inner product of the elements contained? $\endgroup$ Sep 18 '18 at 17:05
  • 2
    $\begingroup$ yes, i like that notation. $\endgroup$
    – chak
    Sep 25 '18 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy