I am using logistic in classification task. The task equivalents with find $\omega, b$ to minimize loss function: enter image description here

That means we will take derivative of L with respect to $\omega$ and $b$ (assume y and X are known). Could you help me develop that derivation . Thank you so much

  • 1
    $\begingroup$ In this setup, I believe the $y_i =$ 1 or -1. Otherwise the loss for control class would be constant. $\endgroup$ Jan 10, 2018 at 22:15

2 Answers 2


I will ignore the sum because of the linearity of differentiation [1]. And I will ignore the bias because I think the derivation for $w$, which I will show, is sufficiently similar. For what it's worth, I think the key is to really understand the chain rule [2]. You might also find these rules helpful. Let's first compute the derivatives of each of the functions separately:

$$ l(a) = \ln(a) = z $$ $$ l^{\prime}(a) = \frac{\partial z}{\partial a} = \frac{1}{\ln(e)(a)} = \frac{1}{a} $$

$$ f(b) = 1 + e^b = v $$

$$ f^{\prime}(b) = \frac{\partial v}{\partial b} = e^b $$

$$ g(c) = -yc = u $$

$$ g^{\prime}(c) = \frac{\partial u}{\partial c} = -y $$

$$ h(w) = wx = t $$

$$ h^{\prime}(w) = \frac{\partial t}{\partial w} = x $$

Composing these functions:

$$ l(f(g(h(w)))) = \ln(1 + e^{-y(wx)}) $$

$$ l^{\prime}(f(g(h(w)))) = \frac{\partial z}{\partial v} \frac{\partial v}{\partial u} \frac{\partial u}{\partial t} \frac{\partial t}{\partial w} = \frac{1}{1+e^{-y(wx)}} \times e^{-y(wx)} \times -y \times x = \frac{-yxe^{-y(wx)}}{1+e^{-y(wx)}} $$


First it is : $ \frac{d}{dx}\sum_{i=1}^n f_i(x) =\sum_{i=1}^n \frac{d}{dx} f_i (x)$

So you can derive every individual summand.

And the derivation of $log(f(x))$ is $\frac{1}{f(x)} \cdot f'(x)$, by using the chain rule.

The third point, which might help you is, that the derivation of $e^{g(x)}$ is $g'(x) \cdot e^{g(x)}$.

If you derive a function of two variables, than pretend one of the variables as a constant:



$\frac{\partial f}{\partial x}=2xy+3y^2$

$\frac{\partial f}{\partial y}=x^2+6yx$

  • $\begingroup$ Can you apply for my formula. Your formula is very basic $\endgroup$
    – John
    Jul 22, 2014 at 7:27
  • $\begingroup$ The derivative of summand i with respect to b is $y_i\cdot e^{y_i \cdot (w^Tx_i+b) } \bigg / \left( 1+e^{y_i \cdot (w^Tx_i+b)} \right) $ $\endgroup$ Jul 22, 2014 at 8:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .