Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Considering a binary classification problem with data $D = \{(x_i,y_i)\}_{i=1}^n$, $x_i \in \mathbb{R}^d$ and $y_i \in \{0,1\}$. Given the following definitions:

$f(x) = x^T \beta$

$p(x) = \sigma(f(x))$ with $\sigma(z) = 1/(1 + e^{-z})$

$$L(\beta) = \sum_{i=1}^n \Bigl[ y_i \log p(x_i) + (1 - y_i) \log [1 - p(x_i)] \Bigr]$$

where $\beta \in \mathbb{R}^d$ is a vector. $p(x)$ is a short-hand for $p(y = 1\ |\ x)$.

The task is to compute the derivative $\frac{\partial}{\partial \beta} L(\beta)$. A tip is to use the fact, that $\frac{\partial}{\partial z} \sigma(z) = \sigma(z) (1 - \sigma(z))$.

So here is my approach so far:

\begin{align*} L(\beta) & = \sum_{i=1}^n \Bigl[ y_i \log p(x_i) + (1 - y_i) \log [1 - p(x_i)] \Bigr]\\ \frac{\partial}{\partial \beta} L(\beta) & = \sum_{i=1}^n \Bigl[ \Bigl( \frac{\partial}{\partial \beta} y_i \log p(x_i) \Bigr) + \Bigl( \frac{\partial}{\partial \beta} (1 - y_i) \log [1 - p(x_i)] \Bigr) \Bigr]\\ \end{align*}

\begin{align*} \frac{\partial}{\partial \beta} y_i \log p(x_i) &= (\frac{\partial}{\partial \beta} y_i) \cdot \log p(x_i) + y_i \cdot (\frac{\partial}{\partial \beta} p(x_i))\\ &= 0 \cdot \log p(x_i) + y_i \cdot (\frac{\partial}{\partial \beta} p(x_i))\\ &= y_i \cdot (p(x_i) \cdot (1 - p(x_i))) \end{align*}

\begin{align*} \frac{\partial}{\partial \beta} (1 - y_i) \log [1 - p(x_i)] &= (1 - y_i) \cdot (\frac{\partial}{\partial \beta} \log [1 - p(x_i)])\\ & = (1 - y_i) \cdot \frac{1}{1 - p(x_i)} \cdot p(x_i) \cdot (1 - p(x_i))\\ & = (1 - y_i) \cdot p(x_i) \end{align*}

$$\frac{\partial}{\partial \beta} L(\beta) = \sum_{i=1}^n \Bigl[ y_i \cdot (p(x_i) \cdot (1 - p(x_i))) + (1 - y_i) \cdot p(x_i) \Bigr]$$

So basically I used the product and chain rule to compute the derivative. I am afraid, that my solution is wrong, because in Hasties The Elements of Statistical Learning on page 120 it says the gradient is:

$$\sum_{i = 1}^N x_i(y_i - p(x_i;\beta))$$

I don't know what could have possibly gone wrong, any advices on this?

share|cite|improve this question
Did you mean $p(x)=\sigma(p(x))$ ? Because I don't see you using $f$ anywhere. – Raskolnikov Apr 28 '12 at 7:08
Also, note your final line can be simplified to: $\sum_{i=1}^n \Bigl[ p(x_i) (y_i - p(x_i)) \Bigr]$. – Raskolnikov Apr 28 '12 at 7:12
Yes, absolutely, thanks for pointing out, it is indeed $p(x) = \sigma(p(x))$. But isn't the simplification term: $\sum_{i=1}^n [p(x_i) ( 1 - y \cdot p(x_i)]$ ? – Mahoni Apr 28 '12 at 7:16
Maybe, but I just noticed another mistake: when you compute the derivative of the first term in $L(\beta)$. – Raskolnikov Apr 28 '12 at 7:18
Ah, are you sure about the relation being $p(x)=\sigma(f(x))$? Because if that's the case, then I can see why you don't arrive at the correct result. – Raskolnikov Apr 28 '12 at 7:21
up vote 1 down vote accepted

So, if $p(x)=\sigma(f(x))$ and $\frac{d}{dz}\sigma(z)=\sigma(z)(1-\sigma(z))$, then

$$\frac{d}{dz}p(z) = p(z)(1-p(z)) f'(z) \; .$$

This changes everyting and you should arrive at the correct result this time.

In particular,

$$\frac{d}{dz}\log p(z) = (1-p(z)) f'(z)$$


$$\frac{d}{dz}\log (1-p(z)) = -p(z) f'(z) \; .$$

share|cite|improve this answer
Of course, I ignored the chain rule for that one! – Mahoni Apr 28 '12 at 7:26
Also be careful because your $\beta$ is a vector, so is $x$. So you should really compute a gradient when you write $\partial/\partial \beta$. – Raskolnikov Apr 28 '12 at 7:29

In your third line, while differentiating you missed out $1/p(x_i)$ which is the derivative of $\log(p(x_i))$.

\begin{eqnarray} d/db(y_i \cdot \log p(x_i)) &=& \log p(x_i) \cdot 0 + y_i \cdot(d/db(\log p(x_i))\\ &=& y_i \cdot 1/p(x_i) \cdot d/db(p(x_i)) \end{eqnarray}

Note that $d/db(p(xi)) = p(x_i)\cdot {\bf x_i} \cdot (1-p(x_i))$ and not just $p(x_i) \cdot(1-p(x_i))$.

Also in 7th line you missed out the $-$ sign which comes with the derivative of $(1-p(x_i))$.

share|cite|improve this answer

The classification problem data can be captured in one matrix and one vector, i.e. $\{X,y\}$.

Then the relevant quantities are the vectors $$\eqalign{ f &= X^T\beta \cr p &= \sigma(f) \cr }$$ and their differentials and logarithmic differentials $$\eqalign{ df &= X^Td\beta \cr dp &= p\circ(1-p)\circ df \cr\cr d\log(p) &= \frac{dp}{p} \,=\, (1-p)\circ df \cr d\log(1-p) &= \frac{-dp}{1-p} \,=\, -p\circ df \cr }$$ where $(g\circ h)$ and $\big(\frac{g}{h}\big)$ denote element-wise (aka Hadamard) multiplication and division.

The likelihood function is a scalar which can be written in terms of Frobenius products $$\eqalign{ L &= y:\log(p) + (1-y):\log(1-p) \cr }$$ whose differential is $$\eqalign{ dL &= y:d\log(p) + (1-y):d\log(1-p) \cr &= y:(1-p)\circ df - (1-y):p\circ df \cr &= (y-p):df \cr &= \big(y-p\big):X^Td\beta \cr &= X\,\big(y-p\big):d\beta \cr }$$ Yielding the gradient as $$\eqalign{ \frac{\partial L}{\partial\beta} &= X\,(y-p) \cr }$$

share|cite|improve this answer
This is the matrix form of the gradient, which appears on page 121 of Hastie's book. – greg Nov 14 '15 at 19:27

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.