2
$\begingroup$

I want to compute the derivative of binary cross entropy loss w.r.t to the input of the sigmoid function and was wondering if there's a closed form expression? I've seen derivations of binary cross entropy loss with respect to model weights/parameters (derivative of cost function for Logistic Regression) as well as derivations of the sigmoid function w.r.t to its input (Derivative of sigmoid function $\sigma (x) = \frac{1}{1+e^{-x}}$), but nothing that combines the two. I would greatly appreciate any help with this.

There's also a post that computes the derivative of categorical cross entropy loss w.r.t to pre-softmax outputs (Derivative of Softmax loss function). I am looking for something similar in the binary case (perhaps this generalizes to the binary case, but not sure).

$\endgroup$

1 Answer 1

4
$\begingroup$

Use properties of logarithms to simplify as much as possible before taking the derivative.

Let $0 \leq p \leq 1$. We want to compute the derivative of the function \begin{align} L(u) &= -p \log(\sigma(u)) - (1-p)\log(1 - \sigma(u)) \\ &= -p\log( \frac{e^u}{1+e^u} ) - (1-p) \log( \frac{1}{1+e^u}) \\ &= -pu +\log(1 + e^u). \end{align}

Look how much $L(u)$ simplified! Sigmoid and binary cross-entropy are a match made in heaven.

It is now easy to take the derivative of $L$: $$ L’(u) = \sigma(u) - p. $$

This formula has a nice interpretation. If the predicted probability $\sigma(u)$ agrees perfectly with the ground truth probability $p$, then the derivative of $L$ is $0$ — suggesting that we do not need to make any change to the value of $u$.

$\endgroup$
2
  • $\begingroup$ Thanks for this clean and simple derivation! Makes a lot of sense. One quick question: should the e^u in the sigmoid function be e^-u? Regardless I don't think that should impact the final answer. $\endgroup$
    – Jane Sully
    Aug 20, 2021 at 15:31
  • $\begingroup$ @JaneSully Note that $\frac{e^u}{1 + e^u} = \frac{1}{1 + e^{-u}}$, so the formula $\sigma(u) = \frac{1}{1 + e^{-u}}$ that you used is equivalent to the formula $\sigma(u) = \frac{e^u}{1 + e^u}$ that I'm using here. $\endgroup$
    – littleO
    Aug 21, 2021 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.