1
$\begingroup$

Right now I'm confused on how Information is even different than entropy.

But let's say we got an information about the result of a coin toss. How much information would that give us? In bits?

I straight up went to calculating Entropy and got 1 bit, which is easily backed up by logic, since 1 bit tells us that a value is either 1 or 0(head or tails).

How about information of a die toss result?

I'm guessing it should carry a lot more information. Using the same process as before i got 6* (1/6 * log2(1/6)) which leaves me with 2.58 bits of information. This could be right but I'm confused. Am I not just calculating regular entropy?

If by some miracle my thinking is correct for a die toss result, how much information would the result of a two-die toss result bring us? My logic would be since each die is bringing us 2.58bits of information, the sum of these should tell us how much information in an two-die-toss. What do you think?

Let's take it a bit further. If we're just randomly guessing 5 numbers out of 50, how much information would the result of that one bring us?

$\endgroup$
  • $\begingroup$ For three bits, we would need $8$ possible results. So, a die result can only give $2$ bits. But if we throw two times, we already have $36$ possible results, giving us $5$ bits.If we throw many times, we have actually about $n\cdot \log_2(6)$ bits, when $n$ is the number of throws, so the number $\log_2(6)$ in fact is the average information for a die result. The reason is that the number of bits is $$\lfloor \frac{\log(6^n)} {log(2)} \rfloor=\lfloor n\cdot log_2(6) \rfloor$$ $\endgroup$ – Peter Nov 12 '17 at 16:35
  • $\begingroup$ What are the precise definitions of "information" and "entropy" that you're using, and what makes you think they're not the same? $\endgroup$ – Ilmari Karonen Nov 12 '17 at 16:39
  • $\begingroup$ @IlmariKaronen I'm not using any defition for Information. What makes me think they're not the same is that they're named differently nothing else to be honest. $\endgroup$ – Nephilim Nov 12 '17 at 17:28
0
$\begingroup$

which leaves me with 2.58 bits of information. This could be right but I'm confused. Am I not just calculating regular entropy?

Yes. Actually, entropy is the average information.

In your examples, all events (coin and dice) are equiprobable, so each ocurrence carries the same information. But that's not necessary. Asumme you have a biased coin, with $p(Head)=0.9 $ and $p(Tail)=0.1$. In that case, the information you get each time a Head occurs is $-\log(0.9)=0.152$ bits; each time a Tail occurs you get an information $-\log(0.1)=3.322$ bits (less probable events are "more informative"). In average you have $0.152 \times 0.9 + 3.322 \times 0.1=0.469$ bits - which is the entropy of the coin.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.