as far as I know, cross entropy of two distributions is: $$ C(p,q) = -\sum_{s \in classes}p(s)\log(q(s)) $$ however, the loss function for logistic regression (called "crossentropy loss") it's defined as: $$ J(\theta) = -\frac{1}{m}\sum_{i=1}^my^{(i)} \:\cdot\: log(h_\theta(x^{(i)}))+(1-y^{(i)})\:\cdot\:log(1-h_\theta(x^{(i)})) $$ as far as I know, $y$ and $(1-y)$ are just term to handle the error in case of miss-classification in both ways, and this makes me really miss the connection with the crossentropy definition, ergo, I don't see why it's called like that, even though i don't see any connection between the 2 formulas


1 Answer 1


In the crossentropy loss, $y$ is either $0$ or $1$. It can be seen as the exact probability of an example belongs to class 1 or class 2 So $\mathbb{P}(w=1|\mathbf{x}) = y$ and $\mathbb{P}(w=2|\mathbf{x}) = 1-y$

Identically $h_\theta(\mathbf{x})$ is the predicted probability of an example belonging to class 1 or class 2. So $\mathbb{P}(w=1) = h_\theta(\mathbf{x})$ and $\mathbb{P}(w=2) = 1-h_\theta(\mathbf{x})$.

This should make clear the connection with the definition of cross-entropy. Note that your second definition is a kind of average cross-entropy (you are summung over examples, but your indices seem wrong)

  • $\begingroup$ sure, I was missing the indices.... but I don't get which are 2 distributions on which we are calculating the crossentropy ($p,q$) $\endgroup$
    – anon
    Commented Jan 30, 2022 at 14:23
  • $\begingroup$ p,q are probability mass functions that give the probability that a discrete random variable (the class) is exactly 1 or 2 (these numbers are arbitrary) $\endgroup$
    – Steph
    Commented Jan 30, 2022 at 16:26
  • $\begingroup$ So our $\frac{1}{m}\sum...$ is just a "estimation" of the $p(t)$ of the first formula (ergo, the probability is the proportion of elements)? $\endgroup$
    – anon
    Commented Jan 30, 2022 at 21:35

You must log in to answer this question.