Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Shannon's "self-information" of the specific outcome "A" is given as: -log(Pr(A)), and the entropy is the expectation of the "self-information" of all the outcomes of the random variable.

When the base of the log is 2, the units of information/entropy are called "bits".

What is the best explanation the following simple question:

Why do these information units are called "bits"?

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

One good reason to call them bits is that this is the number of bits that you need on average to encode an outcome. Some Wikipedia articles you might want to take a look at are Huffman coding, arithmetic coding, entropy encoding and Shannon's source coding theorem.

To give a simple example, say outcome A has probability $1/2$ and outcomes B and C have probabilities $1/4$ each. Then you can encode A by $0$, B by $10$ and C by $11$. This is an optimal prefix-free code; the expected number of bits required to encode an outcome is $\frac12\cdot1+\frac14\cdot2+\frac14\cdot2=\frac32$, and since the number of bits in each code is the self-information (to base $2$) of the outcome it encodes, this expected number of bits is the entropy of the distribution.

share|improve this answer
    
Thank you for the answer. –  Michael Mar 20 '12 at 19:53
    
@Michael: You're welcome. –  joriki Mar 20 '12 at 19:58
    
I'm familiar with information theory and know the source and channel coding approaches. From this point of view I completely understand the usage of a term "bit". But what is the meaning, for example, of the "self-information" of 10 bits of some event? –  Michael Mar 20 '12 at 20:05
    
@Michael: I'm not sure I understand the question. As my example illustrates, the self-information of an event is the number of bits in the code used to encode it in an optimal prefix-free code. What other meaning are you looking for? (By the way, it would have been a good idea to mention in the question that you're familiar with information theory.) –  joriki Mar 20 '12 at 20:10
    
Maybe I'm digging to much in it... But it is not always possible to achieve exactly the "self-information" with a symbol code (only in the case when the probabilities are the powers of 2). It is a nice explanation with symbol codes but how you explain students that the units are "bits" on the first lesson? –  Michael Mar 20 '12 at 20:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.