# Shannon's formula

Shannon formally deﬁned the amount of information in a message as a function of the probability of the occurrence of each possible message[1]. Given a universe of messages $\mathbf{M} = \{ m_1,m_2,..,m_n \}$ and a probability $\mathbf{p(m_i)}$ for the occurrence of each message, the information content of a message in $\mathbf{M}$ is given by:

$$\sum\limits_{i=1}^{n} -p(m_{i})\log_{2}(p(m_{i}))$$

How is this formula derived? Why did Shannon add $\log_2$?

[1]C.E. Shannon. A mathematical theory of communication. Bell Systems Technical Journal, 27:379–423, 623–656, 1948.

-
Since you mention the paper, I'd have assumed that you read it. But then you ask that question... Here, help yourself. – Raskolnikov Mar 6 '11 at 20:18
The $\log_2$ is just a convention, since in computer science people tend to work with bits, and you want the amount of information in one bit (with $p(m=0) = p(m=1) = 1/2$) to be equal to 1. Physicists tend use the natural logarithm, for example. – Gerben Mar 6 '11 at 22:06
bits, nats, whatever – yoyo Mar 6 '11 at 22:28

Shannon proves in his paper that in order to code $n$ messages you need roughly $nH(M)$ bits. You can take that as the definition of entropy, and then derive Shannon's formula.
There are also axiomatic derivations of $H(M)$ - you start with some expected properties of the entropy function, and you end up with exactly that formula. I think that's also in Shannon's paper.
The logarithm comes in basically since to represent $T$ different messages you need at least $\log_2 T$ bits.
Toss a coin $n$ times. The probability of the outcome you get is $p=(1/2)^n$. The number of bits of information is $n$, which is $-\log_2 p$.