0
$\begingroup$

If I choose a random alphabet from (a-z), $26$ characters, what is the entropy?

Shannon's formula:

$$H = - \sum p \log_2(p) = - (1/26)\log_2(1/26) = 0.18$$ bits.

However, other formulas on the Internet use:

$$H = \log_2 (N^L) = \log_2(26) = 4.7$$

bits.

Which one is correct?

$\endgroup$
2
  • $\begingroup$ You only got one term of the sum, not all $26$ of them $\endgroup$ Aug 9, 2019 at 16:35
  • $\begingroup$ @RossMillikan Thanks for the clarification! I had wrongly thought that the sum was for the number of characters in the string. Turns out it's for the number of possible characters in the character set :) $\endgroup$ Aug 9, 2019 at 17:17

1 Answer 1

2
$\begingroup$

The internet is correct.

The sum is over all possible outcomes, and not just one. So you should get:

$H = \sum_{x = 'a'}^{'z'} - p(x) \log_2(p(x)) = \sum_{x = 'a'}^{'z'} - \frac{1}{26} \log_2(\frac{1}{26} = 26 * (-\frac{1}{26} \log_2(\frac{1}{26})) = -\log_2(\frac{1}{26}) = \log_2(26) = 4.70043971814...$

$\endgroup$
2
  • 1
    $\begingroup$ Put another way, Shannon's formula $H = - \sum p \log_2 p$ is correct as written, it's just that the expansion in OP's question of that sum is incorrect. $\endgroup$ Aug 9, 2019 at 16:11
  • $\begingroup$ Understood thanks! I mistakenly thought that the sum was over the number of characters in the string, not the character set itself. $\endgroup$ Aug 9, 2019 at 17:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .