While I was watching a lecture on Information theory, I found that entropy of an information source is the average amount of information that it provides in terms of bits (or nats, decits or whatever), which's actually the weighted average of information contained in all the symbols that source provides (weighted by probabilities of individual symbols).. I found a striking similarity between this & concept of Expectation of a random variable which has similar stuff in it's explanation. Am I right on this ? I mean is their any intuitive connection between the two concepts? The explanation of both the concepts taking into consideration their similarity (if find any) is also welcome. Thank you.

  • $\begingroup$ Am I right with the assumption that you see a similarity in the sense that both objects are weighted averages? $\endgroup$ – Fabian Apr 3 '11 at 15:37
  • $\begingroup$ & also both seem to be probabilistically weighted averages. $\endgroup$ – amitlan Apr 3 '11 at 15:51
  • $\begingroup$ Expectation is a very basic concept in probability theory, and everything is connected to it. $\endgroup$ – Yuval Filmus Apr 3 '11 at 16:11

The entropy is defined using an expectation. If you have a random variable $X$ whose pdf is $P$ then its entropy is $$H(X) = \mathbb{E}(-\log P(X)).$$ Another connection is through your definition above. For every $n$, one can find a uniquely decodable code $C_n$ that encodes $n$-tuples of values of $X$ in binary, minimizing the expected length $L_n$. The entropy is the limit of $L_n/n$.

| cite | improve this answer | |
  • $\begingroup$ can you explain the part after the formula of H(x) ? $\endgroup$ – amitlan Apr 4 '11 at 17:14
  • 1
    $\begingroup$ Optimally coding $n$ copies of $X$ (using a uniquely decodable code) costs (in expectation) about $nH(X)$ bits. Since you can define this cost without mentioning entropy, you get another definition of entropy. $\endgroup$ – Yuval Filmus Apr 4 '11 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.