# Two weighted coins, determining which has a higher probability of landing heads

A friend of mine asked me the following question, and I am not sure how to solve it:

You are given two weighted coins, $C_1$ and $C_2$. Coin $C_1$ has probability $p_1$ of landing heads and $C_2$ has probability $p_2$ of landing heads. The following experiment is preformed:

Coin $C_1$ is flipped 3 times, and lands heads 3 times.

Coin $C_2$ is flipped 10 times, and lands heads 7 times.

Based on this experiment, choose the coin which is more likely to have a higher probability of being heads. In other words, which is more likely: $p_1>p_2$ or $p_2>p_1$.

Intuition tells me coin $C_1$ is the better choice, but this could be wrong, and I am wondering how do you solve this in general. Consider the experiment, $C_1$ is flipped $n_1$ times and lands heads $m_1$ times, $C_2$ is flipped $n_2$ times and lands heads $m_2$ times.

Thanks for the help,

Edit: I think this might answer some questions: Suppose that the probabilities of the coins, $p_1$ and $p_2$ are chosen uniformly from $[0,1]$.

-
You really need a prior distribution for how "weighted" the coins may be, and then use Bayesian techniques. – Henry Apr 18 '11 at 0:18
I would be very interested in seeing a "prior-free", possibly frequentist, approach to this problem, if there is such a thing. I feel like Bayesian techniques get all the press these days. – Rahul Apr 18 '11 at 0:20
I wonder if you can do this with nonparameteric statistics. The only test I can think of requires that the two sequences have the same length, i.e. $n_1=n_2$ and that's pretty dull here. – Carl Brannen Apr 18 '11 at 0:49
@Rahul: The question "which is more likely: $p_1 > p_2$ or $p_2 > p_2$" cries out for a solution which treats both as uncertain. – Henry Apr 18 '11 at 1:15
@Eric: Yes - you get a figure of about 0.758 for the chance $p_1 > p_2$ – Henry Apr 18 '11 at 8:58

Given a specific probability $p$ of heads, the probability of getting $h$ heads and $t$ tails is just the binomial distribution: $P(H = h, T = t | n, p) = p^h (1-p)^t {n \choose h}$ (though we consider $p$ as varying, rather than $n$ and $h$.

With a uniform prior $g(p) = 1$, that probability is just the weight. The integral of this is $\frac{1}{1 + h + t}$, giving a probability density of $(1 + h + t) p^h (1-p)^t {n \choose h}$. The mean is the integral of $p$ times this, is $(1 + h + t) \int p^{(h+1)} (1-p)^t {n \choose h} dp$. Reusing the result from last time, we know that this must be $(1+h+t) {n \choose h}/{n+1 \choose h + 1}/(2+h+t) = (1+n)(h+1)/(n+1)(n+2) = (h+1)/(n+2)$. This is slightly "hedged toward the center" from the naïve estimator $h/n$ (which is the peak of the distribution).

Another common prior that a Bayesian might use is the beta distribution. It's handy because it is a conjugate prior for the binomial distribution. After collecting data generated by the binomial distribution, the probability is still in the form of a beta distribution. In fact, the uniform prior is just the beta distribution with $\alpha = \beta = 1$. Heads and tail each just add one to the parameters $\alpha$ and $\beta$ respectively. The integrals were essentially worked out above -- factorials generalize to $\Gamma(x+1)$. It's often considered that this case of $\alpha = \beta = 1$ is too conservative, and that $\alpha = \beta = 1/2$ "assumes less" and "lets the data speak more".

With the Uniform Prior (Beta(1,1)), $\overline{p} = (h+1)/(n+2)$:
$C_1$, 3 heads, 0 tails: $\overline{p} = 4/5$
$C_2$, 7 heads, 3 tails: $\overline{p} = 8/12 = 2/3$

$C_1$ is expected to do better.

With the Beta(1/2, 1/2) prior, $\overline{p} = (h+1/2)/(n+1) = (2h+1)/(2n+2)$:
$C_1$, 3 heads, 0 tails: $\overline{p} = 7/8$
$C_2$, 7 heads, 3 tails: $\overline{p} = 15/22$

$C_1$ is expected to do better.

Actually calculating the chances of $C_1$ being better than $C_2$ involve a rather nasty integral, but the calculated $p$ value is enough to tell you which is the better bet.

-
How do you prove that it's ok to compare estimators for $p_1$ and $p_2$ as you did here, instead of actually testing (or finding the probability of) the hypothesis that $p_1 > p_2$? It's intuitive, but does it not require proof? – ShreevatsaR Jun 2 '11 at 3:26
Huh, you're right @ShreevatsaRt. This only gets $E[p_1 - p_2] > 0$, not $P(p_1 > p_2) > 1/2$. For most purposes what you want is the first, as payoffs are linear in $p$. – wnoise Jun 2 '11 at 13:45

As Henry mentions, I think one needs some information about the prior distributions of the weights.

Denote by $r$ the weight of a coin. Suppose that the weight has some prior distribution $g(r)$. Let $f(r|H=h, T=t)$ be the posterior probability density function of $r$ having observed $h$ heads and $t$ tails tossed. Bayes' Theorem tells us that: $$f(r|H=h, T=t) = \frac{Pr(H=h|r, N = h+t)g(r)}{\int_0^1 Pr(H=h|r, N = h+t)g(r)\ dr}.$$

This should allow you to answer your question. Once you have the posterior pdf for each coin, just find their respective expected weights.

-

You could do the integration which wnoise is talking about approximately using the following R code, and it is easily adapted to other cases:

> n <- 1000000                      # number of cases to simulate for integration
> prior  <- c(1,1)                  # Beta(1,1) is uniform prior
> coin_1 <- c(3,0)                  # number of heads and tails observed
> coin_2 <- c(7,3)                  # number of heads and tails observed
> p_1    <- rbeta(n, prior[1]+coin_1[1], prior[2]+coin_1[2])
> p_2    <- rbeta(n, prior[1]+coin_2[1], prior[2]+coin_2[2])
> p_diff <- p_1 - p_2
> length(p_diff[p_diff > 0]) / n    # proportion with p_1 > p_2
[1] 0.758118

-