4
$\begingroup$

I am a layman interested in understanding why the foundation of Shannon's entropy is logarithmic.

To that end I've read the answers here, at the Cross Validated Stack, but I'm not technical enough to infer the basic idea from the math.

https://stats.stackexchange.com/questions/87182/what-is-the-role-of-the-logarithm-in-shannons-entropy

But in trying to make an semi-educated guess I infer the following and am asking here if this is a reasonable explanation.

Shannon's entropy is logarithmic because the CHANCES of multiple information events occurring simultaneously are multiplied ... but should all those events occur the total VALUE of those events are summed.

For example, betting on coin flips. The chance of heads is $1/2$ while the chances for three heads in a row is $1/8 = (1/2)^3$ as each flip is independent.

Assuming the bet for each flip is also independent, the total won for winning each of the three flips is $B_1+B_2+B_3$.

This is a logarithmic relationship because we can express:

  • (a) An exponent as a log - the chance being 'to the power of'
  • (b) The result of said event as a sum. In other words, logarithms let you express multiplications as sums and vice versa.

Thanks.

$\endgroup$
9
  • $\begingroup$ What do you mean by "total won"? $\endgroup$
    – J.G.
    Commented Apr 26, 2022 at 22:01
  • $\begingroup$ the total won for winning three bets in a row is B1+B2+B3. $\endgroup$ Commented Apr 26, 2022 at 22:20
  • 2
    $\begingroup$ I would say you understand the basic reason, except instead of "total won" I think it's more about "total information". The information contained in the outcome of three coin tosses is three times the information contained in one coin toss, assuming they are fair and independent. $\endgroup$
    – Joe
    Commented Apr 26, 2022 at 22:31
  • $\begingroup$ Yes, ok, that's fine. Total won = information value...e.g, "three dollars" vs. "thirty dollars". The reasoning for the logarithm is what I want to verify because it seems it have great consequences for productivity and habits ... that it's far more value to be consistent than to perform perfectly. (Thank you for the edit Axel.) $\endgroup$ Commented Apr 26, 2022 at 22:38
  • $\begingroup$ I’m voting to close this question because Piotr Migdal's answer in OP's link explains clearly that Shannon entropy uses the logarithm to make it grow linearly with system size. What more does one want of an entropy ? $\endgroup$
    – Kurt G.
    Commented Apr 27, 2022 at 9:28

2 Answers 2

1
$\begingroup$

There are many possible characterizations of entropy, including Shannon, Renyi and other entropies. For example John Baez discusses some of this in terms of information loss

Regarding Shannon entropy specifically, it was first axiomatically characterized by Khinchin, see Halizi's notes for example. Basically assuming a finite range of cardinality $k$ for the random variable $X$, say $X\in \{1,2,\ldots,k\},$ if you want that

  1. $H(X)$ should depend only on the probability distribution of $X$ not on the values it takes (labels are unimportant to uncertainty)
  2. For a given $k,$ $H(X)$ should be maximum when $X$ is uniform with probability $1/k$ everywhere
  3. Adding points to the range with zero probability should not change the entropy
  4. There should be a chain rule $H(X,Y)=H(X)+H(Y|X)$ connecting the joint entropy of two random variables via a conditional entropy $H(X|Y)$ (uncertainty in both is uncertainty in the first plus uncertainty in the second given the first)

then the only function satisfying these axioms is the entropy denoted as Shannon entropy $$ H(X)=\sum_{i=1}^k P(X=i) \log (1/P(X=i)) $$ up to a choice of the base of logarithm.

Prior to Shannon and Khinchin, Hartley had defined entropy only with respect to uniform distributions, actually just by looking at the number of points and not worrying about the probability distribution, so just as $\log k$ which is of course the maximum entropy for the general case.

$\endgroup$
1
$\begingroup$

Short version: What you said:

the chances of multiple information events occurring simultaneously are multiplied ... but should all those events occur the total value of those events are summed

is pretty much correct.

Suppose you are going to send a message $M_1$ and then a message $M_2$. Let's try to invent a measure of the information content of the two messages and write $I(M_1)$ and $I(M_2)$ for the information content of the two messages. What properties would we like this “information content” to have?

Suppose we consider the act of sending $M_1$ and then $M_2$ as a single communication, which we could call $M_1\oplus M_2$. Whatever $I$ means, we ought to have $$I(M_1\oplus M_2) = I(M_1) + I(M_2)$$

which says that the amount of information in the combined message is equal to the sum of the information in the two messages sent separately.

Now suppose $M_1$ could consist of any of $n_1$ distinguishable signals and similarly $M_2$ will be one of $n_2$ distinguishable signals. It makes sense that the information content is related to the number of possible signals: if I could send you one of a million possible messages, the message I send will carry more information than if there are only two possible messages I could have sent. Let's see what happens if we try to express $I$ only in terms of $n_1$ and $n_2$.

Since there are $n_1$ was to send $M_1$, and $n_2$ ways to send $M_2$, there are just $n_1\cdot n_2\text{ }$ways to send $M_1\oplus M_2$. So if $I$ can be expressed in terms of the number of possible messages that could be sent, we must have $$I(n_1\cdot n_2) = I(n_1)+I(n_2)$$ which tells us that $I$ must be logarithmic in the number of possible messages.

The choice of which logarithm we use isn't important; it is simply a change of units, like the choice of whether to measure distance in feet or meters. One common choice is a base-2 logarithm, in which case we get a unit called a bit; sometimes we use a base-$e$ logarithm and get a unit called a nat which is 44% larger. In the past a base-10 logarithm was sometimes used, producing a unit called a hartley which is around 3.32 bits.

$\endgroup$
3
  • $\begingroup$ This is excellent. Thank you. Re: feet vs. meters ... are there still situations where base-10 in info-entropy related calculations. $\endgroup$ Commented Apr 27, 2022 at 17:10
  • $\begingroup$ "if I could send you one of a million possible messages, the message I send will carry more information than if there are only two possible messages I could have sent." Why does a message depend on the size of the group it came from? You use the words "Message" and "Communication" and "Information". Which are the containers and which are the information? You say "Combined Message" ... does that mean 'interlaced' somehow or 'in a package' like eggs in a carton? $\endgroup$ Commented May 1, 2022 at 9:33
  • $\begingroup$ I think the whole thing is a formalized expression of powers (exponents, logs) with information as the nouns. If I pick a number between 1 and 256 it would take you no more than 8 guesses to figure it out as 2^8 is 256. The number of guesses, the power, grows linearly as the system (the numbers, the information) grows exponentially. Yes or no. $\endgroup$ Commented May 1, 2022 at 19:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .