A friend of mine asked me the following question, and I am not sure how to solve it:

You are given two weighted coins, $C_1$ and $C_2$. Coin $C_1$ has probability $p_1$ of landing heads and $C_2$ has probability $p_2$ of landing heads. The following experiment is preformed:

Coin $C_1$ is flipped 3 times, and lands heads 3 times.

Coin $C_2$ is flipped 10 times, and lands heads 7 times.

Based on this experiment, choose the coin which is more likely to have a higher probability of being heads. In other words, which is more likely: $p_1>p_2$ or $p_2>p_1$.

Intuition tells me coin $C_1$ is the better choice, but this could be wrong, and I am wondering how do you solve this in general. Consider the experiment, $C_1$ is flipped $n_1$ times and lands heads $m_1$ times, $C_2$ is flipped $n_2$ times and lands heads $m_2$ times.

Thanks for the help,

Edit: I think this might answer some questions: Suppose that the probabilities of the coins, $p_1$ and $p_2$ are chosen uniformly from $[0,1]$.

  • 2
    $\begingroup$ You really need a prior distribution for how "weighted" the coins may be, and then use Bayesian techniques. $\endgroup$
    – Henry
    Apr 18, 2011 at 0:18
  • 1
    $\begingroup$ I would be very interested in seeing a "prior-free", possibly frequentist, approach to this problem, if there is such a thing. I feel like Bayesian techniques get all the press these days. $\endgroup$
    – user856
    Apr 18, 2011 at 0:20
  • $\begingroup$ I wonder if you can do this with nonparameteric statistics. The only test I can think of requires that the two sequences have the same length, i.e. $n_1=n_2$ and that's pretty dull here. $\endgroup$ Apr 18, 2011 at 0:49
  • 1
    $\begingroup$ @Rahul: The question "which is more likely: $p_1 > p_2$ or $p_2 > p_2$" cries out for a solution which treats both as uncertain. $\endgroup$
    – Henry
    Apr 18, 2011 at 1:15
  • 1
    $\begingroup$ @Eric: Yes - you get a figure of about 0.758 for the chance $p_1 > p_2$ $\endgroup$
    – Henry
    Apr 18, 2011 at 8:58

4 Answers 4


Given a specific probability $p$ of heads, the probability of getting $h$ heads and $t$ tails is just the binomial distribution: $P(H = h, T = t | n, p) = p^h (1-p)^t {n \choose h}$ (though we consider $p$ as varying, rather than $n$ and $h$.

With a uniform prior $g(p) = 1$, that probability is just the weight. The integral of this is $\frac{1}{1 + h + t}$, giving a probability density of $(1 + h + t) p^h (1-p)^t {n \choose h}$. The mean is the integral of $p$ times this, is $(1 + h + t) \int p^{(h+1)} (1-p)^t {n \choose h} dp$. Reusing the result from last time, we know that this must be $(1+h+t) {n \choose h}/{n+1 \choose h + 1}/(2+h+t) = (1+n)(h+1)/(n+1)(n+2) = (h+1)/(n+2)$. This is slightly "hedged toward the center" from the naïve estimator $h/n$ (which is the peak of the distribution).

Another common prior that a Bayesian might use is the beta distribution. It's handy because it is a conjugate prior for the binomial distribution. After collecting data generated by the binomial distribution, the probability is still in the form of a beta distribution. In fact, the uniform prior is just the beta distribution with $\alpha = \beta = 1$. Heads and tail each just add one to the parameters $\alpha$ and $\beta$ respectively. The integrals were essentially worked out above -- factorials generalize to $\Gamma(x+1)$. It's often considered that this case of $\alpha = \beta = 1$ is too conservative, and that $\alpha = \beta = 1/2$ "assumes less" and "lets the data speak more".

With the Uniform Prior (Beta(1,1)), $\overline{p} = (h+1)/(n+2)$:
$C_1$, 3 heads, 0 tails: $\overline{p} = 4/5$
$C_2$, 7 heads, 3 tails: $\overline{p} = 8/12 = 2/3$

$C_1$ is expected to do better.

With the Beta(1/2, 1/2) prior, $\overline{p} = (h+1/2)/(n+1) = (2h+1)/(2n+2)$:
$C_1$, 3 heads, 0 tails: $\overline{p} = 7/8$
$C_2$, 7 heads, 3 tails: $\overline{p} = 15/22$

$C_1$ is expected to do better.

Actually calculating the chances of $C_1$ being better than $C_2$ involve a rather nasty integral, but the calculated $p$ value is enough to tell you which is the better bet.

  • 1
    $\begingroup$ How do you prove that it's ok to compare estimators for $p_1$ and $p_2$ as you did here, instead of actually testing (or finding the probability of) the hypothesis that $p_1 > p_2$? It's intuitive, but does it not require proof? $\endgroup$ Jun 2, 2011 at 3:26
  • $\begingroup$ Huh, you're right @ShreevatsaRt. This only gets $E[p_1 - p_2] > 0$, not $P(p_1 > p_2) > 1/2$. For most purposes what you want is the first, as payoffs are linear in $p$. $\endgroup$
    – wnoise
    Jun 2, 2011 at 13:45

As Henry mentions, I think one needs some information about the prior distributions of the weights.

Denote by $r$ the weight of a coin. Suppose that the weight has some prior distribution $g(r)$. Let $f(r|H=h, T=t)$ be the posterior probability density function of $r$ having observed $h$ heads and $t$ tails tossed. Bayes' Theorem tells us that: $$ f(r|H=h, T=t) = \frac{Pr(H=h|r, N = h+t)g(r)}{\int_0^1 Pr(H=h|r, N = h+t)g(r)\ dr}. $$

This should allow you to answer your question. Once you have the posterior pdf for each coin, just find their respective expected weights.


You could do the integration which wnoise is talking about approximately using the following R code, and it is easily adapted to other cases:

> n <- 1000000                      # number of cases to simulate for integration
> prior  <- c(1,1)                  # Beta(1,1) is uniform prior
> coin_1 <- c(3,0)                  # number of heads and tails observed
> coin_2 <- c(7,3)                  # number of heads and tails observed
> p_1    <- rbeta(n, prior[1]+coin_1[1], prior[2]+coin_1[2])
> p_2    <- rbeta(n, prior[1]+coin_2[1], prior[2]+coin_2[2])
> p_diff <- p_1 - p_2
> length(p_diff[p_diff > 0]) / n    # proportion with p_1 > p_2
[1] 0.758118

I agree with the beta approach, but given the question, I think it makes more sense to plot out the results and compare visually:

$x$ <- seq($0,1,$length $= 1000$) #set $x$ from $0$ to $1$ since we're looking at probabilities

$y_1$ <- dbeta$(x,4,1)$ #calculate density based on a prior of $[1,1]$ and a posterior of $[3,0]$.

$y_2$ <- dbeta$(x,8,4)$ #calculate density based on a prior of $[1,1]$ and a posterior of $[7,3]$.

plot($x,y_1$,type = "l") #plot density of coin 1 in black

lines($x,y2$,col $= 2$) #plot density of coin 2 in red

enter image description here

This yields the following:

Think of the densities as representing the "probability of probability".

  • Coin 1 (represented by the black line) has a higher probability of having a higher probability of showing heads.

  • Coin 2 (represented by the red line) has a lower probability of having a higher probability of showing heads (or a higher probability of having a lower probability of showing heads...).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .