I have this function of $\beta$: $$f(\beta)=\left(\textbf{y}-\textbf{X}\beta\right)^T\left(\textbf{y}-\textbf{X}\beta\right)$$


  • $\textbf{y}$ is a $N \times 1$ column vector.
  • $\textbf{X}$ is a $N \times p$ matrix.
  • Therefore $\beta$ is a $p \times 1$ column vector.

I'm asked to differentiate $f(\beta)$ with respect to $\beta$, but I've never worked with matrices when it comes to differentiating, I find it a bit difficult.

I looked for help on some books I have and on the internet, and found these expressions:

  • $\left(\partial/\partial_{\textbf{x}}\right)\textbf{x}^T\textbf{y}=\left(\partial/\partial_{\textbf{x}}\right)\textbf{y}^T\textbf{x}=\textbf{y}$

  • $\left(\partial/\partial_{\textbf{x}}\right)\textbf{x}^TA\textbf{y}=\left(\partial/\partial_{\textbf{x}}\right)\textbf{y}^TA^T\textbf{x}=A\textbf{y}$

But I can't seem to be able to apply them in my case. Any help is more than appreciated, I'm still in my learning stage with mathematics.


I've tried to apply the chain rule to no avail, apparently, because it doesn't match with the final solution given by the book I took this problem from: $$\dfrac{\partial f(\beta)}{\partial \beta}=-\textbf{X}^T\left(\textbf{y}-\textbf{X}\beta\right)-\left(\textbf{y}-\textbf{X}\beta\right)^T\textbf{X}$$


Just play a Taylor series trick. Recall, Taylor series tell us that $$ f(x+\partial x) = f(x) + f^\prime(x)\partial x + o(\|\partial x\|) $$ Hence, we can see that \begin{align*} f(\beta+\partial \beta) =&(y-X(\beta+\partial \beta))^T(y-X(\beta+\partial \beta))\\ =&(y-X\beta+X\partial \beta)^T(y-X\beta+X\partial \beta)\\ =&\underbrace{(y-X\beta)^T(y-X\beta)}_{f(\beta)}\underbrace{-(X\partial \beta)^T(y-X\beta) - (y-X\beta)^T(X\partial \beta)}_{f^\prime(\beta)\partial \beta}+\underbrace{(X\partial \beta)^T(X\partial\beta)}_{o(\|\partial \beta\|}\\ \end{align*} Basically, we just expand $f(\beta + \partial \beta)$, regroup terms, and Taylor's theorem tells us which one the derivative is. This gives \begin{align*} f^\prime(\beta)\partial \beta =&-(X\partial \beta)^T(y-X\beta) - (y-X\beta)^T(X\partial \beta)\\ =&-\partial \beta^TX^T(y-X\beta) - (y-X\beta)^TX\partial \beta\\ =&-(X^T(y-X\beta))^T\partial \beta - (y-X\beta)^TX\partial \beta\\ =&-2(X^T(y-X\beta))^T\partial \beta\\ \end{align*}

From the Riesz representation theorem we get that $\langle \nabla f(\beta),\partial \beta\rangle = f^\prime(\beta)\partial \beta$ or that $\nabla f(\beta)^T\partial\beta = f^\prime(\beta)\partial \beta$. Matching terms, we get the result we want which is $$ \nabla f(\beta) = -2X^T(y-X\beta). $$

  • $\begingroup$ Thank you for your detailed answer, though this is way out of my basic algebra skillset. Isn't there an easier way to work out a solution? (i.e. your last equation is the correct expression I'm trying to reach) $\endgroup$ – Jose Lopez Garcia May 6 '17 at 16:12
  • $\begingroup$ Also, this is part of an optimization problem, so I'll eventually set the derivative equal to 0 and solve for $\beta$, if that simplifies the process? $\endgroup$ – Jose Lopez Garcia May 6 '17 at 16:13
  • $\begingroup$ In truth, for this kind of problem, the Taylor Series trick really is the easiest way to derive things. Basically, just expand $f(\beta + \partial \beta)$ and regroup things to look like a Taylor series. This gives the directional derivative. To get the gradient, isolate $\partial \beta$. What remains is the gradient. Technically, you can do all of this variable by variable using sums, but I find it ugly and don't like that derivation. From an optimization point of view, the vectorized form above is what you want since these operations can be implemented more efficiently than with sums. $\endgroup$ – wyer33 May 7 '17 at 5:31
  • $\begingroup$ I understand it now @wyer33, thank you! (However I'm in trouble trying to reach the third expression for $f(\beta+\partial \beta)$, but I'll keep trying). $\endgroup$ – Jose Lopez Garcia May 7 '17 at 15:47
  • 1
    $\begingroup$ No problem! For the third line, I just expanded the multiplication using FOIL. I grouped $y-X\beta$ together and then $X\partial\beta$ by itself. You can multiply out each of the elements individually, but then you'll need to recombine them later, so it's a bit uglier.. $\endgroup$ – wyer33 May 8 '17 at 6:37

Define a new vector
$$\eqalign{ w &= X\beta-y \cr}$$ Then write the function in terms of the inner/Frobenius product (denoted by a colon) and this new variable. In this new form, finding the differential and gradient is straightforward. $$\eqalign{ f &= w:w \cr\cr df &= 2w:dw \cr &= 2w:X\,d\beta \cr &= 2X^Tw:d\beta \cr\cr \frac{\partial f}{\partial\beta} &= 2X^Tw \cr &= 2X^T(X\beta-y) \cr\cr }$$ Don't be put-off by the Frobenius product, it's merely a convenient infix notation for the trace $$\eqalign{A:B={\rm tr}(A^TB)\cr}$$

  • $\begingroup$ Thank you so much for showing me a new way to get to the final result :) $\endgroup$ – Jose Lopez Garcia May 7 '17 at 15:48

Expanding the expression gives us $$f(\beta) = (y-X\beta)^T(y-X\beta) = y^Ty - \beta^TX^Ty - y^TX\beta + \beta^T X^T X \beta$$ Therefore, $$\frac{\partial f}{\partial \beta} = 0-X^Ty-y^TX + \frac{\partial }{\partial \beta} \beta^T A \beta$$ where $A = X^TX$. Note that $$\beta^T A\beta = \sum_{k=1}^p \sum_{\ell=1}^p \beta_k\beta_\ell A_{k\ell}$$ and so $$\frac{\partial}{\partial \beta_i} \beta^T A \beta = \sum_{k=1}^p \sum_{\ell=1}^p \frac{\partial}{\partial \beta_i}\beta_k\beta_\ell A_{k\ell} = \sum_{k=1}^p \sum_{\ell=1}^pA_{k\ell}\beta_\ell \frac{\partial \beta_k}{\partial \beta_i}+A_{k\ell}\beta_k\frac{\partial \beta_\ell}{\partial \beta_i}$$ $$= \sum_{k=1}^p \sum_{\ell=1}^pA_{k\ell}\beta_\ell \delta_{ki}+A_{k\ell}\beta_k\delta_{\ell i} = \sum_{k=1}^p A_{ki}\beta_k + \sum_{\ell=1}^p A_{i\ell}\beta_\ell = (\beta^TA+A\beta)_i$$ Thus, $$\frac{\partial f}{\partial \beta} = \beta^T X^TX + X^TX\beta - X^Ty-y^TX$$ which is equivalent to your book's solution.

Also, the formulas you gave are incorrect, since the dimensions don't work out. You should have $$\frac{\partial }{\partial x} y^Tx = y^T$$ $$\frac{\partial}{\partial x} y^TA^Tx = y^TA^T$$

  • $\begingroup$ Wow, advanced maths there. I didn't understand much, sadly, would you point me to an article where I can learn your process or the sort? $\endgroup$ – Jose Lopez Garcia May 6 '17 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.