37
$\begingroup$

I found that Kullback-Leibler loss, log-loss or cross-entropy is the same loss function. Is the logistic-loss function used in logistic regression equivalent to the cross-entropy function? If yes, can anybody explain how they are related?

Thanks

$\endgroup$
45
$\begingroup$

The relationship between Cross-entropy, logistic loss and K-L divergence is quite natural and immersed in the definition itself.

Cross-entropy is defined as: \begin{equation} H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q)=-\sum_x p(x)\log q(x) \end{equation} Where, $p$ and $q$ are two distributions and using the definition of K-L divergence. $H(p)$ is the entropy of p. Now if $p \in \{y,1-y\}$ and $q \in \{\hat{y}, 1-\hat{y}\}$, we can re-write cross-entropy as: \begin{equation} H(p, q) = -\sum_x p_x \log q_x =-y\log \hat{y}-(1-y)\log (1-\hat{y}) \end{equation} which is nothing but logistic loss. Further, log loss is also related to logistic loss and cross-entropy as follows:

Expected Log loss is defined as follows: \begin{equation} E[-\log q] \end{equation} Note the above loss function used in logistic regression where q is a sigmoid function. Excess risk for the above loss function is defined as follows: \begin{equation} E[\log p - \log q ]=E[\log\frac{p}{q}]=D_{KL}(p||q) \end{equation} Notice that the K-L divergence is nothing but the excess risk of the log loss and K-L differs from Cross-entropy by a constant factor (see the first definition). One important thing to remember is that we usually minimize the log loss instead of the cross-entropy in logistic regression which is not perfectly OK but it is in practice.

$\endgroup$
6
  • 1
    $\begingroup$ Sorry if this a dum question, but what is $H(p)$? $\endgroup$ – Charlie Parker Jan 19 '18 at 20:23
  • 1
    $\begingroup$ @CharlieParker $H(p)$ is the entropy of $p$. $\endgroup$ – CKM Jan 22 '18 at 3:35
  • $\begingroup$ @chandresh in logistic regression we have true probabilities {1,0}, so log_loss is equivalent to cross-entropy. Why it is not perfectly OK to optimize log_loss instead of the cross-entropy? $\endgroup$ – Vadym B. Jun 5 '18 at 10:47
  • 1
    $\begingroup$ @VadymB. If you see my the very first and the last eq, you will realize that two losses differ by a constant (here $H(p)$). Hence, it's not perfectly OK. $\endgroup$ – CKM Jun 5 '18 at 12:42
  • 1
    $\begingroup$ Just to clarify...since "perfectly OK" is vague. They have exactly the same minimum, so it is perfectly okay to ignore the constant offset during minimization, it is just not okay if you were to try and compare losses across different datasets (which might have different H(p) values). $\endgroup$ – slund Nov 11 '18 at 4:56
4
$\begingroup$

yes they are related.
the cross entropy used in logistic regression is derived from the Maximum Likelihood principle (or equivalently minimise (- log(likelihood))). see section 28.2.1 Kullback-Liebler divergence:

Suppose ν and µ are the distributions of two probability models, and ν << µ. Then the cross-entropy is the expected negative log-likelihood of the model corresponding to ν, when the actual distribution is µ

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.