1
$\begingroup$

I have a sample question for an exam and would like some help understanding if I'm approaching it correctly. The question:

We toss three fair (unbiased) coins. 
For each of them, heads and tails have the identical probability 1/2.
The resulting random value is a vector hx1, x2, x3i where xi ∈ {H, T}.
    1. Compute entropy of this random variable.
    2. How does the entropy change if one of the three coins is unfair (biased) and it always lands with heads up?

With a fair coin there are 8 possible outcomes/vectors: HHH, HHT, HTH, HTT, TTT, TTH, THT, THH Since the coins are fair, each is equally likely, so we have $H(X) = log_28 = 3$

Then if one coin is biased to heads, the outcomes are HHH, HHT, HTH, HTT, TTH, THT, THH
and the calculation of the entropy changes to $H(X) = log_27 ≈ 2.81$, so the entropy decreases slightly.

Is this understanding correct?

$\endgroup$
4
  • 1
    $\begingroup$ No, if one coin always comes up heads then there are only 4 possible outcomes and the entropy is 2 bits. Intuitively each fair coin provides 1 bit of entropy and the bad coin provides no entropy because it's deterministic. $\endgroup$
    – Karl
    Commented May 30, 2023 at 15:47
  • 1
    $\begingroup$ @Karl but you don't know what is position of bad coin. $\endgroup$
    – mihaild
    Commented May 30, 2023 at 18:02
  • 1
    $\begingroup$ @mihalid We don't know the index of the bad coin, but (as I interpret the given wording) the bad coin still has a fixed index. Your interpretation (that the experiment includes randomizing the coin order) makes it more interesting, though. $\endgroup$
    – Karl
    Commented May 30, 2023 at 19:52
  • $\begingroup$ I was not sure if the ordering matters, as it is represented as a random vector. I was working as if the order was irrelevant, i.e. 8 outcomes. $\endgroup$
    – Reccho
    Commented May 30, 2023 at 20:15

1 Answer 1

2
$\begingroup$

This is not correct, because outcomes (assuming the position of bad coin is chosen uniformly) do not have equal probability.

For example, outcome $HTT$ has probability $1/12$: it requires the first coin to be bad ($1/3$) and rolling tale twice ($1/4$). However, outcome $HHT$ has probability $2/12$ (it allows two positions for bad coin).

Overall, we have $3$ outcomes with probability $1/12$, $3$ outcomes with probability $2/12$ and one outcome ($HHH$) with probability $3/12$, for total entropy

$$-\frac{3}{12} \log_2 \frac{1}{12} -\frac{6}{12} \log_2 \frac{2}{12} - \frac{3}{12} \log_2 \frac{3}{12} \approx 2.69$$

$\endgroup$
2
  • 1
    $\begingroup$ I think I see your point. Either way the entropy decreases it seems, the degree is what differs. $\endgroup$
    – Reccho
    Commented May 30, 2023 at 20:17
  • 1
    $\begingroup$ Yes, uniform distribution has maximal entropy, so any deviation from it decreases the entropy. $\endgroup$
    – mihaild
    Commented May 31, 2023 at 8:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .