Cross-entropy is a good perspective to understand logistic regression, but I have the following question:

the objective function of LR: $$\max L(\theta) = \max \sum_{i=1}^N y_i \log \hat y_i + (1-y_i) \log (1- \hat y_i)$$ where $y_i$ is the probability of true label,$\hat y_i$ is the probability of predicted label. Now if we have $p \in \{y,1-y\}$ and $q \in \{\hat{y}, 1-\hat{y}\}$,then the cross-entropy of $p,q$ has the following two formulations: $$H_p(q)=\sum_x q\log \frac{1}{p}$$ and $$H_q(p)=\sum_x p\log \frac{1}{q}$$ According to the information theory, we know that these two formulations are not symmetry, that is they are not equal to each other. Usually, we may use the second formulation to understand the logistic regression and my question is that why not use the first one?

In my opinion, the first one may be more suitable for the thought of LR. From the view of information theory, since $p$ is the baseline distribution, we may use the optimal code of $p$ ($\log\frac{1}{p}$ represents the optimal code) to value the average code for distribution $q$, then we may try to minimize $H_p(q)$ in order to make $q$ more suitable for the optimal code. Then for LR, we are also looking forward to have a distribution $q$ which is similar to the true distribution $p$. I'm not sure if I have made a mistake.

two formulations of cross-entropy

related question


The notion of cross entropy is related to KL-divergence and entropy: $$ H_q(p)=\sum p\log\frac 1q=\sum p\log\frac pq+\sum p\log \frac 1p=-D(p||q)+H(p). $$ Maximizing the cross entropy over $q$ is equivalent to minimizing KL-divergence. Since the KL-divergence is non-negative, the maximum cross entropy is $H(p)$ and it is achieved by choosing $p=q$.

In other words, in logistic regression the goal is to bring $q$ as close as possible to $p$.

Now if you use the other notion of cross entropy, this interpretation does not work. What is missing from your "optimal code" interpretation is the notion of error. Without consideration for average error, maximizing $H_p(q)$ tends to concentrate $q$ on those $x$ with smallest probability which is very bad in terms of probability of error, again from coding perspective.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.