# Derivative of Binary Cross Entropy - why are my signs not right?

I'm trying to derive formulas used in backpropagation for a neural network that uses a binary cross entropy loss function. When I perform the differentiation, however, my signs do not come out right:

Binary cross entropy loss function: $$J(\hat y) = \frac{-1}{m}\sum_{i=1}^m y_i\log(\hat y_i)+(1-y_i)(\log(1-\hat y)$$

where

$m =$ number of training examples
$y =$ true y value
$\hat y =$ predicted y value

When I attempt to differentiate this for one training example, I do the following process:

Product rule: $$\frac{dJ}{d\hat y_i} = -1(\frac{d}{d\hat y_i}(y_i\log(\hat y_i)+(1-y_i)(\log(1-\hat y))))$$

Sum rule: $$= -1(\frac{d}{d\hat y_i}y_i\log(\hat y_i)+\frac{d}{d\hat y_i}(1-y_i)(\log(1-\hat y)))$$

Product rule, deriv of constant (treating $y$ as a constant) and deriv of natural log: $$= -1(\frac{y_i}{\hat y_i} + \frac{1-y_i}{1 - \hat y_i})$$

However, this is different from the expected result: $$\frac{dJ}{d\hat y_i} = -1(\frac{y_i}{\hat y_i} - \frac{1-y_i}{1 - \hat y_i})$$

Not sure what's going wrong. I'm sure I'm doing something incorrectly, but I can't figure out what it is. Any help is appreciated!

• $\frac{d}{dx}\log(1-x)\neq \frac{1}{1-x}$. You need the chain rule, which will give you the required negative sign. Nov 3 '17 at 20:44
• Thanks @AlexR. I lost many hours to this. Nov 3 '17 at 21:22

$$\mathbf{h} = \mathbf{w}^T \mathbf{X}$$

$$\mbox{Logistic regression: }\mathbf{z} = \sigma(\mathbf{h}) = \frac{1}{1 + e^{-\mathbf{h}}}$$

$$\mbox{Cross-entropy loss: } J(\mathbf{w}) = -(\mathbf{y} log(\mathbf{z}) + (1 - \mathbf{y})log(1 - \mathbf{z}))$$ $$\mbox{Use chain rule: } \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}} = \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{z}}} \frac{\partial{\mathbf{z}}}{\partial{\mathbf{h}}} \frac{\partial{\mathbf{h}}}{\partial{\mathbf{\mathbf{w}}}}$$

$$\frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{z}}} = -(\frac{\mathbf{y}}{\mathbf{z}} - \frac{1-\mathbf{y}}{1-\mathbf{z}}) = \frac{\mathbf{z} - \mathbf{y}}{\mathbf{z}(1-\mathbf{z})}$$

$$\frac{\partial{\mathbf{z}}}{\partial{\mathbf{h}}} = \mathbf{z}(1-\mathbf{z})$$

$$\frac{\partial{\mathbf{h}}}{\partial{\mathbf{\mathbf{w}}}} = \mathbf{X}$$

$$\frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}} = \mathbf{X}^T (\mathbf{z}-\mathbf{y})$$

$$\mbox{Gradient descent: } \mathbf{w} = \mathbf{w} - \alpha \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}}$$

• dJ/dw is derivative of sigmoid binary cross entropy with logits, binary cross entropy is dJ/dz where z can be something else rather than sigmoid May 28 '20 at 20:20
• I just noticed that this derivation seems to apply for gradient descent of the last layer's weights only. I'm wondering how backpropagation to previous layer's weight matrices works. I've been studying the algorithm from chapter 2 of this book, which takes a very different approach: neuralnetworksanddeeplearning.com/… Jun 8 '20 at 0:44
• Is it possible to find a close form ? if yes, I cannot find it Oct 17 '20 at 18:05

Let's denote the inner/Frobenius product by $a:b= a^Tb$
and the elementwise/Hadamard product by $a\odot b$
and elementwise/Hadamard division by $\frac{a}{b}$
and note that the $\log$ function is to be applied elementwise.

For convenience, let's use a modified loss function $$L=-mJ$$ Then the differential and gradient of $L$ can be calculated as \eqalign{ L &= y:\log({\hat y}) + (1-y):\log(1-{\hat y}) \cr \cr dL &= y:d\log({\hat y}) + (1-y):d\log(1-{\hat y}) \cr &= \frac{y}{{\hat y}}:d{\hat y} + \frac{1-y}{1-{\hat y}}:d(1-{\hat y}) \cr &= \Big(\frac{y}{{\hat y}} - \frac{1-y}{1-{\hat y}}\Big):d{\hat y} \cr &= \Big(\frac{y-{\hat y}}{{\hat y}-{\hat y}\odot{\hat y}}\Big):d{\hat y} \cr \cr \frac{\partial L}{\partial{\hat y}} &= \frac{y-{\hat y}}{{\hat y}-{\hat y}\odot{\hat y}} \cr \cr } And the gradient of the original cost function is \eqalign{ \frac{\partial J}{\partial{\hat y}} &= -\frac{1}{m}\frac{\partial L}{\partial{\hat y}} = \frac{{\hat y}-y}{m\,({\hat y}-{\hat y}\odot{\hat y})} \cr }

• that should be the trace of A.T*b, to indicate the frobenius product yes? Aug 11 '20 at 4:02

Your answer is almost correct except for the second term. While taking derivative of $(1-y_i)(\log(1-\hat y))$ w.r.t $\hat y$, using product rule, $= (1-y_i)(\frac {1} {1-\hat y_i})* \frac {d(1-\hat y_i)}{d\hat y_i}$ $= (1-y_i)(\frac {1} {1-\hat y_i})*-1$ $= - (\frac {1-y_i} {1-\hat y_i})$

• IMO best explained! Oct 21 at 7:28