What does the $-\log[P(X)]$ term mean in the calculation of entropy? The entropy (self information) of a discrete random variable X is calculated as:
$$
H(x)=E(-\log[P(X)])
$$
What does the $-\log[P(X)]$ mean? It seems to be something like ""the self information of each possible outcome of the random variable X".
And why do we use log function to calculate it?
ADD 1
Well, below is my reasoning:
The root motivation is to quantify/measure the uncertainty contained in a random variable.
Intuitively, people tend to agree that there's some connection between uncertainty and probability. And still intuitively, people shall agree that:

*

*the more probability an outcome has, the less uncertainty it has.

*thus, the less probability an outcome has, the more uncertainty it has.

So, I think if we want to measure the uncertainty for an outcome of a random variable, the measure function should satisfy:

*

*the value of uncertainty measure should be positive (human instinct when counting)

*the value of this measure for the uncertainty of an outcome should be monotonic decreasing function of the probability of that outcome.

*for outcomes of independent experiments, the uncertainty should be additive. That is for P(A)*P(B), the total uncertainty should be the sum of A's and B's. (This is kind of instinctive, too.)

Then I come to the choice of -log[p(i)] as the measure of uncertainty of each possible outcome, or self-information of each outcome.
Then I treat the entropy as the weighted average of the self-information of all possible outcomes.
I just read the book <Information Theory, Inference and Learning Algorithms> by MacKay. The author indeed gives a similar explanation to mine. And he name it the information content of each outcome. It is not difficult to see that entropy better describes a random variable than the information content.
And it is coincidental that the formula we intuitively found to measure the average information content of a random variable has a similar form to the one of entropy in thermodynamics. Thus comes the name information entropy...
BTW I want to quote some words from Einstein...

"It is not so important where one settles down. The best thing is to
follow your instincts without too much reflection."
--Einstein to Max Born, March 3, 1920. AEA 8-146

ADD 2
Following my above reasoning, I tried to derive the calculation of entropy for a continuous random variable Y in a similar way. But I was blocked. Details below.
Let Y's p.d.f be: $$f(y)$$
Then, if we strictly follow my previous reasoning, then we should pick up a small interval of I, and the probability of Y within interval I is given by: $$P(y\ within\ I)=\int_If(y)dy$$Then the measure of uncertainty for Y to fall in interval I should be: $$m(y\ within\ I) = -log\int_If(y)dy$$ Then, to get the entropy, we should get the expectation/average of this measure m, which is essentially: $$E[m(y\ within\ I)]$$ and it can be expanded as below:
$$
\int{P(y\ within\ I)*m(y\ within\ I)}dI
=\int{(\int_I{f(y)dy}*{(-log\int_If(y)dy)})dI}
$$
I found myself stuck here because the interval I is not strictly defined.
Then I find from here the authoritative definition of entropy of continuous random variable:
$$
H(Y)=-\int{f(y)log[f(y)]dy}
$$
The p.d.f. $f(y)$ can certainly be $> 1$, so the $H(Y)$ can be negative, while in discrete scenario, the $H(X)$ is always non-negative.
I cannot explain the why this in-consistence is happening. For now, I can only consider it as a philosophical difficulty regarding continuity and discreteness.
Some of my personal feeling (can be safely ignored):

In the discrete scenario, the concrete countable outcome provide the
foothold for us to carry out our calculation. But in the continuous
scenario, there's no such ready-made foothold (unless we can somehow
make one). Without such foothold, it feels like we just keep falling
into the endless hollowness of mind.

Anyone could shed some light?
ADD 3 - 4:23 PM 2/21/2022
We created mathematics to quantify the world. And here in probability we even try to quantify our mentality, while our mentality created mathematics in the first place. It's like an endless recursive fall. And it's really hard for one to settle down...
 A: In his 1948 paper Claude Shannon introduced the entropy $H$ of a discrete random variable $X$ with probabilities $p_1, \dots, p_n$ as a function which satisfied three requirements, which should provide a measure of the information contained in $X$:

  
*
  
*$H$ should be continuous in the $p_i$.
  
*If all the $p_i$ are equal, $p_i = \frac{1}{n}$, then $H$ should be a monotonic increasing function of $n$. With equally likely events there is more choice, or uncertainty, when there are more possible events.
  
*If a choice be broken down into two successive choices, the original $H$
  should be the weighted sum of the individual values of $H$. 
  

He further explains what property 3 means with a nice example. Then, in appendix 2, he shows that only a function of the form $$K \sum_{i=1}^n p_i \log(p_i)$$ can satisfy all these three requirements, where $K$ is some multiplicative constant.
A: Assume that one repeatedly draws values from a finite set $S$ of size $|S|$ according to a distribution $p=(p_x)_{x\in S}$. After one draw, there are $|S|$ possible results, after two draws there are $|S|^2$, and so on, so one can get the impression that after $n$ draws, the resulting distribution is spread out on the Cartesian product $S^n$, whose size is $|S|^n$. And indeed it is, but this view is deceptive because the distribution is extremely unevenly spread out on $S^n$. Actually:

There exists a subset $T_n\subset S^n$, often much smaller than $S^n$, on which nearly all the distribution of the sample of size $n$ is concentrated. And in this "vanishingly small" subset $T_n$, the weight of each element is roughly the same...

In other words, everything happens as if the combined result of the $n$ first draws was chosen uniformly randomly in $T_n$. What connects the dots is that the size of $T_n$ is $\mathrm e^{nH}$ for some deterministic finite number $H$. (Actually, the size of $T_n$ is $\mathrm e^{nH+o(n)}$.) Surely you recognized that $H$ is the entropy of the distribution according to which one is drawing the values from $S$, that is,
$$
H=-\sum_{x\in S}p_x\log p_x=-E[\log p_X],
$$
where $X$ is any random variable with distribution $p$.
This surprisingly general phenomenon, related to what is called concentration of measure, quantifies $\mathrm e^H$ as the (growth of the) effective size of the sample space. As direct consequences, $0\leqslant H\leqslant\log|S|$, $H=0$ if and only if $p$ is a Dirac measure and $H=\log|S|$ if and only if $p$ is uniform.
A: Easy illustrative example:
Take a fair coin. $P({\rm each\ result})=1/2$. By independence, $P({\rm each\ result\ in\ n\ tosses})=1/2^n$. The surprise in each coin toss is the same. The surprise in $n$ tosses is $n\times$(surprise in one toss). The $\log$ makes the trick. And the entropy is the mean surprise.
