# Derivative of logistic loss function

I am using logistic in classification task. The task equivalents with find $$\omega, b$$ to minimize loss function: That means we will take derivative of L with respect to $$\omega$$ and $$b$$ (assume y and X are known). Could you help me develop that derivation . Thank you so much

• In this setup, I believe the $y_i =$ 1 or -1. Otherwise the loss for control class would be constant. Jan 10, 2018 at 22:15

I will ignore the sum because of the linearity of differentiation . And I will ignore the bias because I think the derivation for $w$, which I will show, is sufficiently similar. For what it's worth, I think the key is to really understand the chain rule . You might also find these rules helpful. Let's first compute the derivatives of each of the functions separately:

$$l(a) = \ln(a) = z$$ $$l^{\prime}(a) = \frac{\partial z}{\partial a} = \frac{1}{\ln(e)(a)} = \frac{1}{a}$$

$$f(b) = 1 + e^b = v$$

$$f^{\prime}(b) = \frac{\partial v}{\partial b} = e^b$$

$$g(c) = -yc = u$$

$$g^{\prime}(c) = \frac{\partial u}{\partial c} = -y$$

$$h(w) = wx = t$$

$$h^{\prime}(w) = \frac{\partial t}{\partial w} = x$$

Composing these functions:

$$l(f(g(h(w)))) = \ln(1 + e^{-y(wx)})$$

$$l^{\prime}(f(g(h(w)))) = \frac{\partial z}{\partial v} \frac{\partial v}{\partial u} \frac{\partial u}{\partial t} \frac{\partial t}{\partial w} = \frac{1}{1+e^{-y(wx)}} \times e^{-y(wx)} \times -y \times x = \frac{-yxe^{-y(wx)}}{1+e^{-y(wx)}}$$

First it is : $\frac{d}{dx}\sum_{i=1}^n f_i(x) =\sum_{i=1}^n \frac{d}{dx} f_i (x)$

So you can derive every individual summand.

And the derivation of $log(f(x))$ is $\frac{1}{f(x)} \cdot f'(x)$, by using the chain rule.

The third point, which might help you is, that the derivation of $e^{g(x)}$ is $g'(x) \cdot e^{g(x)}$.

If you derive a function of two variables, than pretend one of the variables as a constant:

Example:

$f(x,y)=x^2y+3y^2x$

$\frac{\partial f}{\partial x}=2xy+3y^2$

$\frac{\partial f}{\partial y}=x^2+6yx$

• Can you apply for my formula. Your formula is very basic
– John
Jul 22, 2014 at 7:27
• The derivative of summand i with respect to b is $y_i\cdot e^{y_i \cdot (w^Tx_i+b) } \bigg / \left( 1+e^{y_i \cdot (w^Tx_i+b)} \right)$ Jul 22, 2014 at 8:15