Derivative of logistic loss function

Question

I am using logistic in classification task. The task equivalents with find $\omega, b$ to minimize loss function:

That means we will take derivative of L with respect to $\omega$ and $b$ (assume y and X are known). Could you help me develop that derivation . Thank you so much

Answer 1

I will ignore the sum because of the linearity of differentiation [1]. And I will ignore the bias because I think the derivation for $w$, which I will show, is sufficiently similar. For what it's worth, I think the key is to really understand the chain rule [2]. You might also find these rules helpful. Let's first compute the derivatives of each of the functions separately:

$$ l(a) = \ln(a) = z $$ $$ l^{\prime}(a) = \frac{\partial z}{\partial a} = \frac{1}{\ln(e)(a)} = \frac{1}{a} $$

---

$$ f(b) = 1 + e^b = v $$

$$ f^{\prime}(b) = \frac{\partial v}{\partial b} = e^b $$

---

$$ g(c) = -yc = u $$

$$ g^{\prime}(c) = \frac{\partial u}{\partial c} = -y $$

---

$$ h(w) = wx = t $$

$$ h^{\prime}(w) = \frac{\partial t}{\partial w} = x $$

---

Composing these functions:

$$ l(f(g(h(w)))) = \ln(1 + e^{-y(wx)}) $$

$$ l^{\prime}(f(g(h(w)))) = \frac{\partial z}{\partial v} \frac{\partial v}{\partial u} \frac{\partial u}{\partial t} \frac{\partial t}{\partial w} = \frac{1}{1+e^{-y(wx)}} \times e^{-y(wx)} \times -y \times x = \frac{-yxe^{-y(wx)}}{1+e^{-y(wx)}} $$

Answer 2

First it is : $ \frac{d}{dx}\sum_{i=1}^n f_i(x) =\sum_{i=1}^n \frac{d}{dx} f_i (x)$

So you can derive every individual summand.

And the derivation of $log(f(x))$ is $\frac{1}{f(x)} \cdot f'(x)$, by using the chain rule.

The third point, which might help you is, that the derivation of $e^{g(x)}$ is $g'(x) \cdot e^{g(x)}$.

If you derive a function of two variables, than pretend one of the variables as a constant:

Example:

$f(x,y)=x^2y+3y^2x$

$\frac{\partial f}{\partial x}=2xy+3y^2$

$\frac{\partial f}{\partial y}=x^2+6yx$