# Coin toss bias estimation problem

Alice and Bob are given a set of five biased coins. They both estimate the probability that each coin will show a head when flipped, and each coin is then flipped once. These are the estimates and values observed:

Coin $1$: Alice's estimate $0.4$, Bob's estimate $0.2$, Observed - Heads

Coin $2$: Alice's estimate $0.7$, Bob's estimate $0.8$, Observed - Heads

Coin $3$: Alice's estimate $0.2$, Bob's estimate $0.3$, Observed - Tails

Coin $4$: Alice's estimate $0.9$, Bob's estimate $0.6$, Observed - Tails

Coin $5$: Alice's estimate $0.4$, Bob's estimate $0.3$, Observed - Heads

Whom would you say is better at estimating the bias of the coins, and why?

I would like guidance on how to approach this question.

• What is the question? – OFRBG Oct 8 '16 at 10:22
• @OFRBG Question added – Hogg Oct 8 '16 at 10:24
• What results do you expect for someone who is good at predicting bias? What would you expect from someone who is worse at that? – Graham Kemp Oct 8 '16 at 10:31
• I was thinking that the person with the higher expectation would be better, i.e. $E(\text{Alice's estimate of outcome})=0.4+0.7+0.8+0.1+0.4=2.4$, $E(\text{Bob's estimate of outcome})=0.2+0.8+0.7+0.4+0.3=2.4$, but they both add up to the same value. – Hogg Oct 8 '16 at 10:46
• @Hogg - those additions are not probabilities if they exceed $1$. – Henry Oct 8 '16 at 10:49

One approach is to look at the sum of the squared deviations from the observed outcome ($1$ if heads, $0$ if tails):

$$A: \sqrt{0.6^2+0.3^2+0.2^2+0.9^2+0.6^2}= 1.288\dots$$ $$B: \sqrt{0.8^2+0.2^2+0.3^2+0.6^2+0.7^2}=1.273\dots$$

So, Bob wins, but not significantly!

Another is to count who came the closest the most times, which results in $3:2$ in favor of Alice. This, however, has the same "drawbacks" as for instance the scoring system in Tennis, where one player can actually win more games (which for our example would be equivalent to being closer overall), but still loose the match.

• How about squaring the error? The error would go down, so maybe square rooting the abs? – OFRBG Oct 8 '16 at 13:52
• @OFRBG Thanks for the suggestion, edited! – Bobson Dugnutt Oct 8 '16 at 13:55

There are several possibilities. Here is one, which uses multiplication rather than your suggestion of addition:

Alice's estimated likelihood of the observed outcome is $0.4 \times 0.7 \times 0.8 \times 0.1 \times 0.4 = 0.00896$

Bob's estimated likelihood of the observed outcome is $0.2 \times 0.8 \times 0.7 \times 0.4 \times 0.3 = 0.01344$

So Bob's estimated likelihood is $1.5$ times Alice's estimated likelihood, suggesting Bob's estimates may have been slightly better. Alice's estimate for Coin $4$ was particularly unfortunate.

But somebody else who thought there was no predictable bias might say $0.5$ each time and have an overall likelihood for the outcome of $0.5^5=0.03125$, more than double Bob's figure, suggesting that neither were particularly good.

• If you consider the coin toss estimation of Coin $4$ as an anomalous result and ignore it, we have that Alice's estimated likelihood of the observed outcome is $0.4\times 0.7 \times 0.8 \times 0.4=0.0896$ and Bob's estimated likelihood of the observed outcome is $0.2\times 0.8 \times 0.7 \times 0.3=0.0336$, this would make Alice's estimated likelihood $2.67$ times Bob's estimated likelihood. – Hogg Oct 8 '16 at 11:11
• This approach is called Bayesian Model Comparison. See en.wikipedia.org/wiki/Bayes_factor – awkward Oct 8 '16 at 13:54