# Definition of Entropy (Information Theory)

In Information Theory, entropy is defined as:

$$-\sum_{i}P_ilog(P_i)$$

where $$-P_ilog(P_i)$$ looks like this (using log base 2):

From just a generic English definition of entropy, meaning lack of predictability, I don't find this particularly intuitive. Would you not have the most entropy (be the most uncertain) when $$P_i=0.5$$? In other words, would it not make more sense to use a measure like this:

$$-\sum_{i}4P_i(P_i-1)$$

where $$-4P_i(P_i-1)$$ looks like this:

instead? What is the advantage of using $$P_ilog(P_i)$$?

• Please disregard my now-deleted comments. Your definition of entropy corresponds to what is known as Tsallis entropy with entropy-index 2. This is a generalization of Shannon entropy, and in fact, Shannon entropy is exactly Tsallis entropy with entropy-index 1. In particular, they share many properties. But Shannon entropy $H$ enjoys an additional property called additivity, which says that $H(X,Y)=H(X)+H(Y)$ for independent $X$ and $Y$. – Sangchul Lee Sep 10 at 1:49
• The sum that you are ignoring is also important. That is, if you are flipping a fair coin (i.e. your $P_i=0.5$), it's also important that you sum over the two states (heads and tails). In both formulas you give, you'll find the maximum entropy of $P_1 = p, P_2 = 1-p$ will occur when $p=0.5$ – Brian Moehring Sep 10 at 1:58
• Your first graph is wrong, what you should have plotted was $\text{H}(x)=-x\log(x) - (1-x)\log(1-x)$, which does in fact have its maxima at $x=0.5$. – Thoth Sep 10 at 2:00
• @Thoth, his first graph is a plot of an individual additive term in the overall entropy ... and I believe it was indicated as such – phdmba7of12 Sep 12 at 16:14