I'm trying to understand Shannon's formula for information as entropy in some detail. (I'm obviously not a trained mathematician, so please bear with me).
Here's what I understand and then following are two questions.
Suppose I have a message composed of "a" where "a" is a symbol in a set X of 32 symbols. My understanding is that this has an entropy of 0.15625 = (1/32 * log2 (32))
And correspondingly if change the the number of symbols in set X the entropy changes.
where X = 16, H = 0.25 = 1/16 * log2(16)
where X = 8, H = 0.375 = 1/8 * log2(8)
where X = 4, H = 0.5 = 1/4 * log2(4)
But here's my first question, I would expect that where X = 2, H should equal 1 but where X = 2, H actually equals 0.5 = 1/2 * log2(2)
My second question. I'm trying to explain to other non-mathematicians what "H" means. After some reading, I'm under the impression that Entropy or H should correlate with the amount of uncertainty in a message, or the number of yes or no questions it should on average take to, guess the message.
(Would this be: 1/H = number of guesses needed?)
Hence my confusion in question number. In a set of only two members, "ab", it should only take one guess to identify the message, but the result of H as still .5, means that 1/H (1/.5) = 2, suggesting I still need two guesses.
However, in the case of a set X=4, "abcd" this seems to work out. In the set composed of "abcd", I need two guesses to figure out the correct letter. For example: Guess 1: is it a or b, if yes, Guess 2: is it a, if yes, I found it, if not, I know it is b.
But this doesn't seem to work out so well in a set of 8 or 16 or 32. In the case of a set of 8 "abcdefgh" 1/H would be 1/.375 = 2.66.
The simply log2 of the size of set X, seems to give me what I would have expected H to be, set X=8, log2(8)=3, therefore it takes three guess.
So perhaps I'm not accurately understanding what H is supposed to represent or perhaps I'm doing something wrong in the calculation of H.
Any thoughts, pointers, or tips would be much appreciated.