In Information Theory, entropy is defined as:


where $-P_ilog(P_i)$ looks like this (using log base 2):

enter image description here

From just a generic English definition of entropy, meaning lack of predictability, I don't find this particularly intuitive. Would you not have the most entropy (be the most uncertain) when $P_i=0.5$? In other words, would it not make more sense to use a measure like this:


where $-4P_i(P_i-1)$ looks like this:

enter image description here

instead? What is the advantage of using $P_ilog(P_i)$?

  • $\begingroup$ Please disregard my now-deleted comments. Your definition of entropy corresponds to what is known as Tsallis entropy with entropy-index 2. This is a generalization of Shannon entropy, and in fact, Shannon entropy is exactly Tsallis entropy with entropy-index 1. In particular, they share many properties. But Shannon entropy $H$ enjoys an additional property called additivity, which says that $H(X,Y)=H(X)+H(Y)$ for independent $X$ and $Y$. $\endgroup$ – Sangchul Lee Sep 10 at 1:49
  • 1
    $\begingroup$ The sum that you are ignoring is also important. That is, if you are flipping a fair coin (i.e. your $P_i=0.5$), it's also important that you sum over the two states (heads and tails). In both formulas you give, you'll find the maximum entropy of $P_1 = p, P_2 = 1-p$ will occur when $p=0.5$ $\endgroup$ – Brian Moehring Sep 10 at 1:58
  • 5
    $\begingroup$ Your first graph is wrong, what you should have plotted was $\text{H}(x)=-x\log(x) - (1-x)\log(1-x)$, which does in fact have its maxima at $x=0.5$. $\endgroup$ – Thoth Sep 10 at 2:00
  • $\begingroup$ @Thoth, his first graph is a plot of an individual additive term in the overall entropy ... and I believe it was indicated as such $\endgroup$ – phdmba7of12 Sep 12 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.