From the new book Weird Math:

While the Bayesian approach may seem subjective, it can be made rigorous in an abstract sense. For example, suppose you had a coin that was biased. It could be biased by any amount from 0 percent heads to 100 percent heads, with each value equally likely. You toss it once, and it comes up heads. It’s possible to prove that the probability of a head on the second toss is two out of three using Bayesian probability.

My question
How do you prove that? Could the same result be reached with the frequentist approach?

  • 2
    $\begingroup$ HINT: Let the bias be $p$ which is uniformly distributed in $[0,1]$. Then $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1)$. Then evaluate each (the numerator, the denominator, separately) using the law of total probability conditioned on $p$. $\endgroup$ – antkam Apr 20 '18 at 18:05
  • 1
    $\begingroup$ @antkam: Thank you. Could you form an answer for the Bayesian and the frequentist approach? $\endgroup$ – vonjd Apr 20 '18 at 18:07
  • 2
    $\begingroup$ Ha, that is a philosophical question. I am in no way an expert on this but my impression is that a frequentist might say, if you already know the coin could be biased, then 1 sample is not enough to make any useful claim about $P(H_2)$. Basically this frequentist is hedging and saying, "I don't know enough about $p$ after 1 sample, and I'm not willing to assume something like $p \sim U(0,1)$..." Another more reckless frequentist might say "Hey 1 sample is fine, and $P(H_2) = 1/1 = 100\%$!" But I could be completely mischaracterizing the frequentist philosophy here. Sorry! :) $\endgroup$ – antkam Apr 20 '18 at 18:33
  • $\begingroup$ @antkam: I understand... could you do the Bayesian part? $\endgroup$ – vonjd Apr 20 '18 at 20:17

Let $p=P(H)$ and assume that $p$ is uniformly distributed within $[0,1]$. I.e. the PDF is $f(p) = 1$ for $p \in [0,1]$ (and zero everywhere else).

Let $H_i$ be the event that the $i$th flip is Head. We want to calculate $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1)$.

  • Numerator: $P(H_2 \cap H_1) = \int^1_0 P(H_2 \cap H_1 | p)\ f(p)\ dp = \int^1_0 p^2 \cdot 1\ dp = \left[\frac{p^3}{3}\right]^1_0 = \frac{1}{3}$.

    • Note: for any specific value of $p, P(H_2 \cap H_1 | p) = p^2$ because flips are independent.
  • Denominator: $P(H_1) = \int^1_0 P(H_1 | p)\ f(p)\ dp = \int^1_0 p \cdot 1\ dp = \left[\frac{p^2}{2}\right]^1_0 = \frac{1}{2}$.

  • So $P(H_2|H_1) = P(H_2 \cap H_1) / P(H_1) = (\frac{1}{3}) / (\frac{1}{2}) = \frac{2}{3}$.

Note 1: the distribution of the bias itself, $p \sim U(0,1)$, is important. If other models are assumed, the result will be different. E.g. if $p$ is one of the three values $\{0, \frac{1}{2}, 1\}$ each with probability $\frac{1}{3}$ (i.e. "First randomly pick one of 3 coins in a bag..."), then $P(H_2 \cap H_1) = \frac{1}{3}(0 + \frac{1}{4} + 1) = \frac{5}{12}$ instead.

Note 2: $P(H_1) = 1/2$ could have been argued based on the symmetry of the bias itself, i.e. based on $f(p)$ being symmetric about $1/2$. In fact, $P(H_1) = \int^1_0 p\ f(p)\ dp = \mathbb{E}[p]$.

Note 3: For a fair coin $p = 1/2$ and obviously $P(H_2 \cap H_1) = 1/4$. But despite $f(p)$ being symmetric about $1/2$, i.e. the bias distribution being "fair" (equally likely to be biased either way, to equal amounts), nevertheless $P(H_2 \cap H_1) > 1/4$. This is where the "Bayesian"-ness comes in.

BTW, you may want to work through the example of 3 coins, one with both Heads, one with both Tails, and one fair. The answer is $5/6$ but more importantly the answer can be calculated with just combinatorics. The Bayesian idea is to have (or to assume) some prior "bag of (potentially uncountably infinitely many) coins".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.