Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

While I was watching a lecture on Information theory, I found that entropy of an information source is the average amount of information that it provides in terms of bits (or nats, decits or whatever), which's actually the weighted average of information contained in all the symbols that source provides (weighted by probabilities of individual symbols).. I found a striking similarity between this & concept of Expectation of a random variable which has similar stuff in it's explanation. Am I right on this ? I mean is their any intuitive connection between the two concepts? The explanation of both the concepts taking into consideration their similarity (if find any) is also welcome. Thank you.

share|improve this question
    
Am I right with the assumption that you see a similarity in the sense that both objects are weighted averages? –  Fabian Apr 3 '11 at 15:37
    
& also both seem to be probabilistically weighted averages. –  Amit L Apr 3 '11 at 15:51
    
Expectation is a very basic concept in probability theory, and everything is connected to it. –  Yuval Filmus Apr 3 '11 at 16:11

1 Answer 1

up vote 4 down vote accepted

The entropy is defined using an expectation. If you have a random variable $X$ whose pdf is $P$ then its entropy is $$H(X) = \mathbb{E}(-\log P(X)).$$ Another connection is through your definition above. For every $n$, one can find a uniquely decodable code $C_n$ that encodes $n$-tuples of values of $X$ in binary, minimizing the expected length $L_n$. The entropy is the limit of $L_n/n$.

share|improve this answer
    
can you explain the part after the formula of H(x) ? –  Amit L Apr 4 '11 at 17:14
1  
Optimally coding $n$ copies of $X$ (using a uniquely decodable code) costs (in expectation) about $nH(X)$ bits. Since you can define this cost without mentioning entropy, you get another definition of entropy. –  Yuval Filmus Apr 4 '11 at 17:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.