The entropy (self information) of a discrete random variable X is calculated as:

$$ H(x)=E(-\log[P(X)]) $$

What does the $-\log[P(X)]$ mean? It seems to be something like ""the self information of each possible outcome of the random variable X".

And why do we use log function to calculate it?


Well, below is my reasoning:

The root motivation is to quantify/measure the uncertainty contained in a random variable.

Intuitively, people tend to agree that there's some connection between uncertainty and probability. And still intuitively, people shall agree that:

  • the more probability an outcome has, the less uncertainty it has.
  • thus, the less probability an outcome has, the more uncertainty it has.

So, I think if we want to measure the uncertainty for an outcome of a random variable, the measure function should satisfy:

  • the value of uncertainty measure should be positive (human instinct when counting)
  • the value of this measure for the uncertainty of an outcome should be monotonic decreasing function of the probability of that outcome.
  • for outcomes of independent experiments, the uncertainty should be additive. That is for P(A)*P(B), the total uncertainty should be the sum of A's and B's. (This is kind of instinctive, too.)

Then I come to the choice of -log[p(i)] as the measure of uncertainty of each possible outcome, or self-information of each outcome.

Then I treat the entropy as the weighted average of the self-information of all possible outcomes.

I just read the book <Information Theory, Inference and Learning Algorithms> by MacKay. The author indeed gives a similar explanation to mine. And he name it the information content of each outcome. It is not difficult to see that entropy better describes a random variable than the information content.

And it is coincidental that the formula we intuitively found to measure the average information content of a random variable has a similar form to the one of entropy in thermodynamics. Thus comes the name information entropy...

BTW I want to quote some words from Einstein...

"It is not so important where one settles down. The best thing is to follow your instincts without too much reflection."

--Einstein to Max Born, March 3, 1920. AEA 8-146


Following my above reasoning, I tried to derive the calculation of entropy for a continuous random variable Y in a similar way. But I was blocked. Details below.

Let Y's p.d.f be: $$f(y)$$

Then, if we strictly follow my previous reasoning, then we should pick up a small interval of I, and the probability of Y within interval I is given by: $$P(y\ within\ I)=\int_If(y)dy$$Then the measure of uncertainty for Y to fall in interval I should be: $$m(y\ within\ I) = -log\int_If(y)dy$$ Then, to get the entropy, we should get the expectation/average of this measure m, which is essentially: $$E[m(y\ within\ I)]$$ and it can be expanded as below:

$$ \int{P(y\ within\ I)*m(y\ within\ I)}dI =\int{(\int_I{f(y)dy}*{(-log\int_If(y)dy)})dI} $$

I found myself stuck here because the interval I is not strictly defined.

Then I find from here the authoritative definition of entropy of continuous random variable:

$$ H(Y)=-\int{f(y)log[f(y)]dy} $$

The p.d.f. $f(y)$ can certainly be $> 1$, so the $H(Y)$ can be negative, while in discrete scenario, the $H(X)$ is always non-negative.

I cannot explain the why this in-consistence is happening. For now, I can only consider it as a philosophical difficulty regarding continuity and discreteness.

Some of my personal feeling (can be safely ignored):

In the discrete scenario, the concrete countable outcome provide the foothold for us to carry out our calculation. But in the continuous scenario, there's no such ready-made foothold (unless we can somehow make one). Without such foothold, it feels like we just keep falling into the endless hollowness of mind.

Anyone could shed some light?


Easy illustrative example:

Take a fair coin. $P({\rm each\ result})=1/2$. By independence, $P({\rm each\ result\ in\ n\ tosses})=1/2^n$. The surprise in each coin toss is the same. The surprise in $n$ tosses is $n\times$(surprise in one toss). The $\log$ makes the trick. And the entropy is the mean surprise.

  • 2
    $\begingroup$ It's an interesting/intuitive explanation when you mention "the surprise in n tosses is n×(surprise in one toss)" $\endgroup$ – smwikipedia Feb 4 '14 at 14:43

In his 1948 paper Claude Shannon introduced the entropy $H$ of a discrete random variable $X$ with probabilities $p_1, \dots, p_n$ as a function which satisfied three requirements, which should provide a measure of the information contained in $X$:

  1. $H$ should be continuous in the $p_i$.
  2. If all the $p_i$ are equal, $p_i = \frac{1}{n}$, then $H$ should be a monotonic increasing function of $n$. With equally likely events there is more choice, or uncertainty, when there are more possible events.
  3. If a choice be broken down into two successive choices, the original $H$ should be the weighted sum of the individual values of $H$.

He further explains what property 3 means with a nice example. Then, in appendix 2, he shows that only a function of the form $$K \sum_{i=1}^n p_i \log(p_i)$$ can satisfy all these three requirements, where $K$ is some multiplicative constant.

  • $\begingroup$ From the perspective of probability, the entropy is just an expectation of -log[P(i)], so -log[P(i)] must have some solid meaning. So what is it? $\endgroup$ – smwikipedia Feb 4 '14 at 14:55
  • $\begingroup$ Unfortunately, I cannot offer some intuitive explanation to this expectations, and I see that this answer is not fully satisfying. $\endgroup$ – Roland Feb 4 '14 at 14:57
  • $\begingroup$ It does reveal something. Thanks. Actually, since the entropy/self-information is used to quantify/measure the uncertainty of a random variable, I did think about selecting a mathematical function according to a set of preferred properties. These properties are based on some common sense about the relation between uncertainty and information gain. It seems Shannon also took this approach. $\endgroup$ – smwikipedia Feb 4 '14 at 15:01
  • 1
    $\begingroup$ There is a more satisfying theorem than Shannon's, due to Faddeev (The notion of entropy of finite probabilistic schemes (Russian), Uspekhi Mat. Nauk 11 (1956), 15-19.) It replaces condition 2 by the weaker and quite natural condition that H is a symmetric function of its arguments. It's interesting that in the proof, to prove $H(1/n,1/n,...,1/n) = -k\log(n)$ for some $k > 0$ he needs to use the infinitude of the primes. $\endgroup$ – KCd Feb 5 '14 at 8:58

Assume that one repeatedly draws values from a finite set $S$ of size $|S|$ according to a distribution $p=(p_x)_{x\in S}$. After one draw, there are $|S|$ possible results, after two draws there are $|S|^2$, and so on, so one can get the impression that after $n$ draws, the resulting distribution is spread out on the Cartesian product $S^n$, whose size is $|S|^n$. And indeed it is, but this view is deceptive because the distribution is extremely unevenly spread out on $S^n$. Actually:

There exists a subset $T_n\subset S^n$, often much smaller than $S^n$, on which nearly all the distribution of the sample of size $n$ is concentrated. And in this "vanishingly small" subset $T_n$, the weight of each element is roughly the same...

In other words, everything happens as if the combined result of the $n$ first draws was chosen uniformly randomly in $T_n$. What connects the dots is that the size of $T_n$ is $\mathrm e^{nH}$ for some deterministic finite number $H$. (Actually, the size of $T_n$ is $\mathrm e^{nH+o(n)}$.) Surely you recognized that $H$ is the entropy of the distribution according to which one is drawing the values from $S$, that is, $$ H=-\sum_{x\in S}p_x\log p_x=-E[\log p_X], $$ where $X$ is any random variable with distribution $p$.

This surprisingly general phenomenon, related to what is called concentration of measure, quantifies $\mathrm e^H$ as the (growth of the) effective size of the sample space. As direct consequences, $0\leqslant H\leqslant\log|S|$, $H=0$ if and only if $p$ is a Dirac measure and $H=\log|S|$ if and only if $p$ is uniform.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.