Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using the standard Shannon entropy formula for calculating the entropy of a system at different states. The system has a different number of possible outcomes at each state, in other words the alphabet of the discrete random variable has a different size at each state. The maximum entropy at each state is $\log N$, where $N$ is the number of possible outcomes. This means that for every state I get an entropy value for a different range ($[0,\log N]$). Is it reasonable to (linearly) map entropy values to a common range (e.g. $[0,1]$) so that I can get the entropy difference between states or even calculate the average entropy of all states?

To give a simple but illustrative example, let's say that someone can play a number of games each day. On Monday she can play basketball or football, so $N=2$ for Monday, and on Tuesday basketball or tennis or cricket, so $N=3$ for Tuesday. Then if you collect data about the actual outcomes for several weeks and calculate the entropy for each day, 'Monday's entropy' will be in the range $[0,\log 2]$ and 'Tuesday's entropy' in the range $[0, \log 3]$. What would be the best way to compare these values? Is linear mapping to a common range correct?

share|cite|improve this question
Suppose $N = 2$ on Monday and $N = 1000$ on Tuesday. Would you still think it's reasonable to scale the entropies or would you regard the second number as being larger than the first for a good reason? – Qiaochu Yuan Sep 22 '11 at 4:23
Thanks for your answer, and sorry for my late reply. Let's see. If we collect data for 1000 weeks, and we have an equal number of different outcomes including all possible outcomes for both days, then entropy will be maximum for both days. But it is indeed easier to predict the outcome of a Monday than the outcome of a Tuesday, so the number of possible outcomes must obviously play a role. Is any comparison between them possible at all? For example, can we compare them on the basis of how predictable each day is with respect to its own possibilities? Or is this just a play of words? – Orestis Tsinalis Sep 25 '11 at 23:06

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.