Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to compute the gradient of the following function with respect to $\beta$

$$L(\beta) = \sum_{i=1}^n (y_i - \phi(x_i)^T \cdot \beta)^2$$

Where $\beta$, $y_i$ and $x_i$ are vectors. The $\phi(x_i)$ simply adds additional coefficients, with the result that $\beta$ and $\phi(x_i)$ are both $\in \mathbb{R}^d$

Here is my approach so far:

\begin{align*} \frac{\partial}{\partial \beta} L(\beta) &= \sum_{i=1}^n ( \frac{\partial}{\partial \beta} y_i - \frac{\partial}{\partial \beta}( \phi(x_i)^T \cdot \beta))^2\\ &= \sum_{i=1}^n ( 0 - \frac{\partial}{\partial \beta}( \phi(x_i)^T \cdot \beta))^2\\ &= - \sum_{i=1}^n ( \partial \phi(x_i)^T \cdot \beta + \phi(x_i)^T \cdot \partial \beta))^2\\ &= - \sum_{i=1}^n ( 0 \cdot \beta + \phi(x_i)^T \cdot \textbf{I}))^2\\ &= - \sum_{i=1}^n ( \phi(x_i)^T \cdot \textbf{I}))^2\\ \end{align*}

But what to do with the power of two? Have I made any mistakes? Because $\phi(x_i)^T \cdot \textbf I$ seems to be $\in \mathbb{R}^{1 \times d}$

$$= - 2 \sum_{i=1}^n \phi(x_i)^T\\$$

share|cite|improve this question
Do you know about the chain rule? – Chris Taylor May 4 '12 at 13:16
@ChrisTaylor Sure, I added something to the end of my question. – Mahoni May 4 '12 at 13:22
up vote 0 down vote accepted

Vector differentiation can be tricky when you're not used to it. One way to get around that is to use summation notation until you're confident enough to perform the derivatives without it.

To begin with, let's define $X_i=\phi(x_i)$ since it will save some typing, and let $X_{ni}$ be the $n$th component of the vector $X_i$.

Using summation notation, you have

$$\begin{align} L(\beta) & = (y_i-X_{mi}\beta_m)(y_i - X_{ni}\beta_n) \\ & = y_i y_i - 2 y_i X_{mi} \beta_m + X_{mi}X_{ni}\beta_m\beta_n \end{align}$$

To take the derivative with respect to $\beta_k$, you do

$$\begin{align} \frac{\partial}{\partial\beta_k} L(\beta) & = -2 y_i X_{mi} \delta_{km} + 2 X_{mi} X_{ni} \beta_n \delta_{km} \\ & = -2 X_{mi} \delta_{km} (y_i - X_{ni} \beta_n) \\ & = -2 X_{ki} (y_i - X_{ni} \beta_n) \end{align}$$

which you can then translate back into vector notation:

$$ \frac{\partial}{\partial\beta} L(\beta) = -2 \sum_{i=1}^n X_i (y_i - X_i^T\beta) $$

Does that help?

share|cite|improve this answer
Yes indeed, thanks for the explanation in an alternative notation, this will help me in the future a lot. – Mahoni May 4 '12 at 13:29
I have redone the derivation in my original notation, and I found it quite easy, by just computing the derivative of $(y - x^T \cdot \beta) = -x$ and then applying the product rule which gets me to the same result as your final line. – Mahoni May 4 '12 at 14:58
Glad to hear it! – Chris Taylor May 4 '12 at 15:08

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.