I'm trying to derive formulas used in backpropagation for a neural network that uses a binary cross entropy loss function. When I perform the differentiation, however, my signs do not come out right:

Binary cross entropy loss function: $$J(\hat y) = \frac{-1}{m}\sum_{i=1}^m y_i\log(\hat y_i)+(1-y_i)(\log(1-\hat y)$$


$m = $ number of training examples
$y = $ true y value
$\hat y = $ predicted y value

When I attempt to differentiate this for one training example, I do the following process:

Product rule: $$ \frac{dJ}{d\hat y_i} = -1(\frac{d}{d\hat y_i}(y_i\log(\hat y_i)+(1-y_i)(\log(1-\hat y)))) $$

Sum rule: $$ = -1(\frac{d}{d\hat y_i}y_i\log(\hat y_i)+\frac{d}{d\hat y_i}(1-y_i)(\log(1-\hat y))) $$

Product rule, deriv of constant (treating $y$ as a constant) and deriv of natural log: $$ = -1(\frac{y_i}{\hat y_i} + \frac{1-y_i}{1 - \hat y_i})$$

However, this is different from the expected result: $$ \frac{dJ}{d\hat y_i} = -1(\frac{y_i}{\hat y_i} - \frac{1-y_i}{1 - \hat y_i}) $$

Not sure what's going wrong. I'm sure I'm doing something incorrectly, but I can't figure out what it is. Any help is appreciated!

  • 6
    $\begingroup$ $\frac{d}{dx}\log(1-x)\neq \frac{1}{1-x}$. You need the chain rule, which will give you the required negative sign. $\endgroup$
    – Alex R.
    Nov 3, 2017 at 20:44
  • $\begingroup$ Thanks @AlexR. I lost many hours to this. $\endgroup$
    – Murcielago
    Nov 3, 2017 at 21:22

3 Answers 3


$$\mathbf{h} = \mathbf{w}^T \mathbf{X} $$

$$\mbox{Logistic regression: }\mathbf{z} = \sigma(\mathbf{h}) = \frac{1}{1 + e^{-\mathbf{h}}}$$

$$\mbox{Cross-entropy loss: } J(\mathbf{w}) = -(\mathbf{y} log(\mathbf{z}) + (1 - \mathbf{y})log(1 - \mathbf{z})) $$ $$ \mbox{Use chain rule: } \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}} = \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{z}}} \frac{\partial{\mathbf{z}}}{\partial{\mathbf{h}}} \frac{\partial{\mathbf{h}}}{\partial{\mathbf{\mathbf{w}}}}$$

$$\frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{z}}} = -(\frac{\mathbf{y}}{\mathbf{z}} - \frac{1-\mathbf{y}}{1-\mathbf{z}}) = \frac{\mathbf{z} - \mathbf{y}}{\mathbf{z}(1-\mathbf{z})}$$

$$\frac{\partial{\mathbf{z}}}{\partial{\mathbf{h}}} = \mathbf{z}(1-\mathbf{z}) $$

$$\frac{\partial{\mathbf{h}}}{\partial{\mathbf{\mathbf{w}}}} = \mathbf{X} $$

$$\frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}} = \mathbf{X}^T (\mathbf{z}-\mathbf{y})$$

$$\mbox{Gradient descent: } \mathbf{w} = \mathbf{w} - \alpha \frac{\partial{J(\mathbf{w})}}{\partial{\mathbf{w}}} $$

  • 1
    $\begingroup$ dJ/dw is derivative of sigmoid binary cross entropy with logits, binary cross entropy is dJ/dz where z can be something else rather than sigmoid $\endgroup$ May 28, 2020 at 20:20
  • 1
    $\begingroup$ I just noticed that this derivation seems to apply for gradient descent of the last layer's weights only. I'm wondering how backpropagation to previous layer's weight matrices works. I've been studying the algorithm from chapter 2 of this book, which takes a very different approach: neuralnetworksanddeeplearning.com/… $\endgroup$ Jun 8, 2020 at 0:44
  • $\begingroup$ Is it possible to find a close form ? if yes, I cannot find it $\endgroup$
    – lalaland
    Oct 17, 2020 at 18:05
  • $\begingroup$ @rocksNwaves you're right. The above derivation is for the last layer only which outputs a scalar. To compute gradient in the previous layer's weight matrices, you need to compute jacobian matrix since this is vector-vector transformation. Not vector-scalar like above. $\endgroup$
    – Giang Tran
    Jan 14 at 8:04
  • $\begingroup$ @lalaland stats.stackexchange.com/questions/949/… check this post out $\endgroup$
    – Giang Tran
    Jan 14 at 8:06

Let's denote the inner/Frobenius product by $a:b= a^Tb$
and the elementwise/Hadamard product by $a\odot b$
and elementwise/Hadamard division by $\frac{a}{b}$
and note that the $\log$ function is to be applied elementwise.

For convenience, let's use a modified loss function $$L=-mJ$$ Then the differential and gradient of $L$ can be calculated as $$\eqalign{ L &= y:\log({\hat y}) + (1-y):\log(1-{\hat y}) \cr \cr dL &= y:d\log({\hat y}) + (1-y):d\log(1-{\hat y}) \cr &= \frac{y}{{\hat y}}:d{\hat y} + \frac{1-y}{1-{\hat y}}:d(1-{\hat y}) \cr &= \Big(\frac{y}{{\hat y}} - \frac{1-y}{1-{\hat y}}\Big):d{\hat y} \cr &= \Big(\frac{y-{\hat y}}{{\hat y}-{\hat y}\odot{\hat y}}\Big):d{\hat y} \cr \cr \frac{\partial L}{\partial{\hat y}} &= \frac{y-{\hat y}}{{\hat y}-{\hat y}\odot{\hat y}} \cr \cr }$$ And the gradient of the original cost function is $$\eqalign{ \frac{\partial J}{\partial{\hat y}} &= -\frac{1}{m}\frac{\partial L}{\partial{\hat y}} = \frac{{\hat y}-y}{m\,({\hat y}-{\hat y}\odot{\hat y})} \cr }$$

  • $\begingroup$ that should be the trace of A.T*b, to indicate the frobenius product yes? $\endgroup$ Aug 11, 2020 at 4:02

Your answer is almost correct except for the second term. While taking derivative of $(1-y_i)(\log(1-\hat y))$ w.r.t $ \hat y$, using product rule, $= (1-y_i)(\frac {1} {1-\hat y_i})* \frac {d(1-\hat y_i)}{d\hat y_i} $ $ = (1-y_i)(\frac {1} {1-\hat y_i})*-1 $ $ = - (\frac {1-y_i} {1-\hat y_i})$

  • $\begingroup$ IMO best explained! $\endgroup$
    – D.Samchuk
    Oct 21, 2021 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.