This question already has an answer here:

The explanations about Shannon's theorem speed up all of a sudden when they should tell us why he introduced the log term. They usually range from 'it's useful' to 'it's because there are two choices, then log2 is suitable'. Am I right in thinking that what the log does is 'give a context' to the probability you are measuring the entropy of? If 0.125 was one of two outcomes it's different than if it was one out of ten. Hence, choosing different logs puts this specific option (0.125) in the right ballpark. Am I correct? And is there a more formal proof for this (maybe for the use of log in probability in general). thanks!


marked as duplicate by leonbloy, Namaste, Claude Leibovici, Lord Shark the Unknown, Leucippus Jul 16 '17 at 5:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Are you asking why a logarithmic function, or why the choice of log base 2? $\endgroup$ – fourierwho Jul 15 '17 at 16:51
  • $\begingroup$ why the function. thanks $\endgroup$ – matteoeoeo Jul 15 '17 at 16:58
  • 2
    $\begingroup$ Have you seen these previous questions? What is the role of the logarithm in Shannon's entropy?, Intuitive explanation of entropy? I don't know of any interpretation in terms of contexts or ballparks -- a probability of 0.125 contributes the same term $-0.125\log 0.125$ to the entropy no matter how many other outcomes there are -- so I don't think you are on the right track. $\endgroup$ – Rahul Jul 15 '17 at 17:37

Why are there squares in Pythagoras' theorem? Long and deep thinking led Shannon to this formula. The proof is in the pudding: You could, in the heuristic process, try other functions, like $\arctan$ or something, but only with the correct "Ansatz" you obtain the mighty theorem.


Not the answer you're looking for? Browse other questions tagged or ask your own question.