Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Considering a binary classification problem with data $D = \{(x_i,y_i)\}_{i=1}^n$, $x_i \in \mathbb{R}^d$ and $y_i \in \{0,1\}$. Given the following definitions:

$f(x) = x^T \beta$

$p(x) = \sigma(f(x))$ with $\sigma(z) = 1/(1 + e^{-z})$

$$L(\beta) = \sum_{i=1}^n \Bigl[ y_i \log p(x_i) + (1 - y_i) \log [1 - p(x_i)] \Bigr]$$

where $\beta \in \mathbb{R}^d$ is a vector. $p(x)$ is a short-hand for $p(y = 1\ |\ x)$.

The task is to compute the derivative $\frac{\partial}{\partial \beta} L(\beta)$. A tip is to use the fact, that $\frac{\partial}{\partial z} \sigma(z) = \sigma(z) (1 - \sigma(z))$.

So here is my approach so far:

\begin{align*} L(\beta) & = \sum_{i=1}^n \Bigl[ y_i \log p(x_i) + (1 - y_i) \log [1 - p(x_i)] \Bigr]\\ \frac{\partial}{\partial \beta} L(\beta) & = \sum_{i=1}^n \Bigl[ \Bigl( \frac{\partial}{\partial \beta} y_i \log p(x_i) \Bigr) + \Bigl( \frac{\partial}{\partial \beta} (1 - y_i) \log [1 - p(x_i)] \Bigr) \Bigr]\\ \end{align*}

\begin{align*} \frac{\partial}{\partial \beta} y_i \log p(x_i) &= (\frac{\partial}{\partial \beta} y_i) \cdot \log p(x_i) + y_i \cdot (\frac{\partial}{\partial \beta} p(x_i))\\ &= 0 \cdot \log p(x_i) + y_i \cdot (\frac{\partial}{\partial \beta} p(x_i))\\ &= y_i \cdot (p(x_i) \cdot (1 - p(x_i))) \end{align*}

\begin{align*} \frac{\partial}{\partial \beta} (1 - y_i) \log [1 - p(x_i)] &= (1 - y_i) \cdot (\frac{\partial}{\partial \beta} \log [1 - p(x_i)])\\ & = (1 - y_i) \cdot \frac{1}{1 - p(x_i)} \cdot p(x_i) \cdot (1 - p(x_i))\\ & = (1 - y_i) \cdot p(x_i) \end{align*}

$$\frac{\partial}{\partial \beta} L(\beta) = \sum_{i=1}^n \Bigl[ y_i \cdot (p(x_i) \cdot (1 - p(x_i))) + (1 - y_i) \cdot p(x_i) \Bigr]$$

So basically I used the product and chain rule to compute the derivative. I am afraid, that my solution is wrong, because in Hasties The Elements of Statistical Learning on page 120 it says the gradient is:

$$\sum_{i = 1}^N x_i(y_i - p(x_i;\beta))$$

I don't know what could have possibly gone wrong, any advices on this?

share|improve this question
    
Did you mean $p(x)=\sigma(p(x))$ ? Because I don't see you using $f$ anywhere. –  Raskolnikov Apr 28 '12 at 7:08
    
Also, note your final line can be simplified to: $\sum_{i=1}^n \Bigl[ p(x_i) (y_i - p(x_i)) \Bigr]$. –  Raskolnikov Apr 28 '12 at 7:12
    
Yes, absolutely, thanks for pointing out, it is indeed $p(x) = \sigma(p(x))$. But isn't the simplification term: $\sum_{i=1}^n [p(x_i) ( 1 - y \cdot p(x_i)]$ ? –  Mahoni Apr 28 '12 at 7:16
    
Maybe, but I just noticed another mistake: when you compute the derivative of the first term in $L(\beta)$. –  Raskolnikov Apr 28 '12 at 7:18
    
Ah, are you sure about the relation being $p(x)=\sigma(f(x))$? Because if that's the case, then I can see why you don't arrive at the correct result. –  Raskolnikov Apr 28 '12 at 7:21
show 1 more comment

2 Answers

up vote 1 down vote accepted

So, if $p(x)=\sigma(f(x))$ and $\frac{d}{dz}\sigma(z)=\sigma(z)(1-\sigma(z))$, then

$$\frac{d}{dz}p(z) = p(z)(1-p(z)) f'(z) \; .$$

This changes everyting and you should arrive at the correct result this time.

In particular,

$$\frac{d}{dz}\log p(z) = (1-p(z)) f'(z)$$

and

$$\frac{d}{dz}\log (1-p(z)) = -p(z) f'(z) \; .$$

share|improve this answer
    
Of course, I ignored the chain rule for that one! –  Mahoni Apr 28 '12 at 7:26
    
Also be careful because your $\beta$ is a vector, so is $x$. So you should really compute a gradient when you write $\partial/\partial \beta$. –  Raskolnikov Apr 28 '12 at 7:29
add comment

In your third line, while differentiating you missed out $1/p(x_i)$ which is the derivative of $\log(p(x_i))$.

\begin{eqnarray} d/db(y_i \cdot \log p(x_i)) &=& \log p(x_i) \cdot 0 + y_i \cdot(d/db(\log p(x_i))\\ &=& y_i \cdot 1/p(x_i) \cdot d/db(p(x_i)) \end{eqnarray}

Note that $d/db(p(xi)) = p(x_i)\cdot {\bf x_i} \cdot (1-p(x_i))$ and not just $p(x_i) \cdot(1-p(x_i))$.

Also in 7th line you missed out the $-$ sign which comes with the derivative of $(1-p(x_i))$.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.