Biased coin: Proof that heads has two third probability for second toss when first toss was heads

From the new book Weird Math:

While the Bayesian approach may seem subjective, it can be made rigorous in an abstract sense. For example, suppose you had a coin that was biased. It could be biased by any amount from 0 percent heads to 100 percent heads, with each value equally likely. You toss it once, and it comes up heads. It’s possible to prove that the probability of a head on the second toss is two out of three using Bayesian probability.

My question
How do you prove that? Could the same result be reached with the frequentist approach?

• HINT: Let the bias be $p$ which is uniformly distributed in $[0,1]$. Then $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1)$. Then evaluate each (the numerator, the denominator, separately) using the law of total probability conditioned on $p$. – antkam Apr 20 '18 at 18:05
• @antkam: Thank you. Could you form an answer for the Bayesian and the frequentist approach? – vonjd Apr 20 '18 at 18:07
• Ha, that is a philosophical question. I am in no way an expert on this but my impression is that a frequentist might say, if you already know the coin could be biased, then 1 sample is not enough to make any useful claim about $P(H_2)$. Basically this frequentist is hedging and saying, "I don't know enough about $p$ after 1 sample, and I'm not willing to assume something like $p \sim U(0,1)$..." Another more reckless frequentist might say "Hey 1 sample is fine, and $P(H_2) = 1/1 = 100\%$!" But I could be completely mischaracterizing the frequentist philosophy here. Sorry! :) – antkam Apr 20 '18 at 18:33
• @antkam: I understand... could you do the Bayesian part? – vonjd Apr 20 '18 at 20:17

Let $p=P(H)$ and assume that $p$ is uniformly distributed within $[0,1]$. I.e. the PDF is $f(p) = 1$ for $p \in [0,1]$ (and zero everywhere else).

Let $H_i$ be the event that the $i$th flip is Head. We want to calculate $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1)$.

• Numerator: $P(H_2 \cap H_1) = \int^1_0 P(H_2 \cap H_1 | p)\ f(p)\ dp = \int^1_0 p^2 \cdot 1\ dp = \left[\frac{p^3}{3}\right]^1_0 = \frac{1}{3}$.

• Note: for any specific value of $p, P(H_2 \cap H_1 | p) = p^2$ because flips are independent.
• Denominator: $P(H_1) = \int^1_0 P(H_1 | p)\ f(p)\ dp = \int^1_0 p \cdot 1\ dp = \left[\frac{p^2}{2}\right]^1_0 = \frac{1}{2}$.

• So $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1) = (\frac{1}{3}) / (\frac{1}{2}) = \frac{2}{3}$.

Note 1: the distribution of the bias itself, $p \sim U(0,1)$, is important. If other models are assumed, the result will be different. E.g. if $p$ is one of the three values $\{0, \frac{1}{2}, 1\}$ each with probability $\frac{1}{3}$ (i.e. "First randomly pick one of 3 coins in a bag..."), then $P(H_2 \cap H_1) = \frac{1}{3}(0 + \frac{1}{4} + 1) = \frac{5}{12}$ instead.

Note 2: $P(H_1) = 1/2$ could have been argued based on the symmetry of the bias itself, i.e. based on $f(p)$ being symmetric about $1/2$. In fact, $P(H_1) = \int^1_0 p\ f(p)\ dp = \mathbb{E}[p]$.

Note 3: For a fair coin $p = 1/2$ and obviously $P(H_2 \cap H_1) = 1/4$. But despite $f(p)$ being symmetric about $1/2$, i.e. the bias distribution being "fair" (equally likely to be biased either way, to equal amounts), nevertheless $P(H_2 \cap H_1) > 1/4$. This is where the "Bayesian"-ness comes in.

BTW, you may want to work through the example of 3 coins, one with both Heads, one with both Tails, and one fair. The answer is $5/6$ but more importantly the answer can be calculated with just combinatorics. The Bayesian idea is to have (or to assume) some prior "bag of (potentially uncountably infinitely many) coins".