# Entropy of a random vector: 3 coins

I have a sample question for an exam and would like some help understanding if I'm approaching it correctly. The question:

We toss three fair (unbiased) coins.
For each of them, heads and tails have the identical probability 1/2.
The resulting random value is a vector hx1, x2, x3i where xi ∈ {H, T}.
1. Compute entropy of this random variable.
2. How does the entropy change if one of the three coins is unfair (biased) and it always lands with heads up?


With a fair coin there are 8 possible outcomes/vectors: HHH, HHT, HTH, HTT, TTT, TTH, THT, THH Since the coins are fair, each is equally likely, so we have $$H(X) = log_28 = 3$$

Then if one coin is biased to heads, the outcomes are HHH, HHT, HTH, HTT, TTH, THT, THH
and the calculation of the entropy changes to $$H(X) = log_27 ≈ 2.81$$, so the entropy decreases slightly.

Is this understanding correct?

• No, if one coin always comes up heads then there are only 4 possible outcomes and the entropy is 2 bits. Intuitively each fair coin provides 1 bit of entropy and the bad coin provides no entropy because it's deterministic.
– Karl
Commented May 30, 2023 at 15:47
• @Karl but you don't know what is position of bad coin. Commented May 30, 2023 at 18:02
• @mihalid We don't know the index of the bad coin, but (as I interpret the given wording) the bad coin still has a fixed index. Your interpretation (that the experiment includes randomizing the coin order) makes it more interesting, though.
– Karl
Commented May 30, 2023 at 19:52
• I was not sure if the ordering matters, as it is represented as a random vector. I was working as if the order was irrelevant, i.e. 8 outcomes. Commented May 30, 2023 at 20:15

For example, outcome $$HTT$$ has probability $$1/12$$: it requires the first coin to be bad ($$1/3$$) and rolling tale twice ($$1/4$$). However, outcome $$HHT$$ has probability $$2/12$$ (it allows two positions for bad coin).
Overall, we have $$3$$ outcomes with probability $$1/12$$, $$3$$ outcomes with probability $$2/12$$ and one outcome ($$HHH$$) with probability $$3/12$$, for total entropy
$$-\frac{3}{12} \log_2 \frac{1}{12} -\frac{6}{12} \log_2 \frac{2}{12} - \frac{3}{12} \log_2 \frac{3}{12} \approx 2.69$$