This question already has an answer here:
- Intuitive explanation of entropy 13 answers
The explanations about Shannon's theorem speed up all of a sudden when they should tell us why he introduced the log term. They usually range from 'it's useful' to 'it's because there are two choices, then log2 is suitable'. Am I right in thinking that what the log does is 'give a context' to the probability you are measuring the entropy of? If 0.125 was one of two outcomes it's different than if it was one out of ten. Hence, choosing different logs puts this specific option (0.125) in the right ballpark. Am I correct? And is there a more formal proof for this (maybe for the use of log in probability in general). thanks!