8
$\begingroup$

Suppose I have a coin with unknown probability of success (let say Heads) $p$ which is uniformly distributed on $[0, 1]$.

I toss a coin $N$ times.

What is the probability that I have got $n \leq N$ Heads?

Ok. I've calculated (using Wolfram) that it is exactly $\frac{1}{N+1}$, i.e. every $n$ is equiprobable.

What is the intuitive explanation for this fact?

I can understand by symmetry argument, why getting $k$ Heads should be exactly the same as getting $k$ Tails. But why getting $0$ Heads is the same as getting $7$ Heads (in case $N \neq 7$) is well beyond me.

Update: Calculations involved: $$P(\text{# of Heads} = k) = \int\limits_0^1 {N \choose k} p^k (1-p)^{N-k} dp, $$

solving it gives: $$P(\text{# of Heads} = k) = {N \choose k} \frac{\Gamma(k+1)\Gamma(N-k+1)}{\Gamma(N+2)},$$

which turns out to be $\frac{1}{N+1}$.

$\endgroup$
10
  • 2
    $\begingroup$ Huh? What is your calculation? What is the prior? Is it the uniform prior? $\endgroup$
    – Brian Tung
    Aug 16 at 19:35
  • $\begingroup$ @BrianTung, since you ask and I do not understand you, we can assume that indeed uniform. $\endgroup$
    – dEmigOd
    Aug 16 at 19:36
  • 1
    $\begingroup$ What I mean is, are you assuming that $p$ is uniformly distributed a priori? I.e., before you start flipping, $p$ is equally likely to be any value between $0$ and $1$? $\endgroup$
    – Brian Tung
    Aug 16 at 19:46
  • 1
    $\begingroup$ @DavidG.Stork There's only symmetry if $N=7$, which I don't think is assumed. $\endgroup$ Aug 17 at 4:59
  • 2
    $\begingroup$ @Youem With an argument such as the one you've given, yes! I'm just saying that an argument like that needs to be made; the $k=0$ case is only obviously symmetric to the $k=N$ case. $\endgroup$ Aug 17 at 5:21

3 Answers 3

13
$\begingroup$

Of course there is an intuition.

To see it let's ask the question differently. For that I will be considering $X_0, X_1,\ldots,X_{N}$ as $N+1$ independent uniform distributed random variables on $(0,1)$.

$X_0$ will be the probability of having heads.

The number of heads in your trials is exactly the number of $X_i$ smaller than $X_0$. So if you want to have $k$ heads you want $X_0$ to be ranked as the $k+1$-th element.

However there is no difference between $X_0$ and any other $X_j$. So ranking $X_0$ as the $k+1$-th element will have $1/(N+1)$ probability.

$\endgroup$
7
  • 4
    $\begingroup$ That is a good argument, I read it once and didn't understand it, but reading it again I think it's a very good way to look at this. $\endgroup$ Aug 17 at 1:49
  • 1
    $\begingroup$ So you say: "Let first toss a probability of success", then "count as success everything that is tossed in range $[0, X_1]$". I like it for simplicity, but why is this a different view on the coin tosses? This is unclear. $\endgroup$
    – dEmigOd
    Aug 17 at 5:22
  • 1
    $\begingroup$ Toss a coin with success probability $p$ is equivalent to generate a random variable uniform on $(0,1)$ and see if it smaller than $p$ $\endgroup$
    – Youem
    Aug 17 at 5:58
  • $\begingroup$ It does need re-reading. My thinking: Let $H=\sum\limits_{i=2}^{N+1} \mathbf{1}_{X_i<X_1}$ be the resulting number of heads. We have $\mathbb{P}(H=h \mid X_1=p) ={N \choose h}p^h(1-p)^{N-h}$ as a conditional binomial distribution, while $f_{X_1}(x)=1$ for $0 \le x \le 1$ and marginally $\mathbb{P}(H=h) =\frac{1}{N+1}$ with integer $0 \le h \le N$ due to the exchangeability. It might have been easier if the random variables had been $X_0,X_1,\ldots,X_N$ $\endgroup$
    – Henry
    Aug 17 at 8:35
  • 1
    $\begingroup$ Youem - as @SuzuHirose commented, it is not instantly obvious to me what you said, so I reread it and then wrote those notes to help me understand what you were saying. For me, the intuitive bit is that you do not need to calculate the marginal $\int_0^1 {N \choose h}p^h(1-p)^{N-h} \, dp$ since your model makes it clear it must be $\frac1{N+1}$. I am happy to leave this in the comments, though I have edited your answer to make the special value $X_0$ $\endgroup$
    – Henry
    Aug 17 at 12:57
4
$\begingroup$

Ok. I've calculated (using Wolfram) that it is exactly $\frac{1}{N+1}$, i.e. every $n$ is equiprobable.

That result is correct, but we don't need no stinkin' Wolfram Alpha to do this.

$$P(\text{# of Heads} = k) = \int\limits_0^1 {N \choose k} p^k (1-p)^{N-k} dp, $$

Well first of all $$ \int\limits_0^1 {N \choose k} p^k (1-p)^{N-k} dp ={N \choose k}\int_0^1 p^k(1-p)^{N-k} dp $$ Now the integral $Q(k,N)\equiv\int_0^1 p^k(1-p)^{N-k}dp$ can be done e.g. by integration by parts. If $k>0$ then $$ \begin{align} \int p^k(1-p)^{N-k}&=p^k\int (1-p)^{N-k}-k\int p^{k-1}\int(1-p)^{N-k+1}\\ &={1\over N-k+1}p^k(1-p)^{N-k+1}\biggr|_0^1\\ &\;\;\;\;\;\;+{k\over N-k+1}\int_0^1p^{k-1}(1-p)^{N-k+1}dp\\ &=0+{k\over N-k+1}Q(k-1,N) \tag{1} \end{align} $$ and if $k=0$ then $$Q(0,N)=\int_0^1 (1-p)^N dp={1\over N+1}\tag{2}$$ so using (1) and (2) $$Q(k,N)={k!(N-k)!\over (N+1)!}={1\over(N+1) {N\choose k}}.$$

What is the intuitive explanation for this fact?

The obvious intuitive explanation for the probabilities all being ${1\over N+1}$ is that when we have no idea whatsoever what the probability of getting a head is on any trial, that is to say "there is a uniform distribution of probability of getting a head", then we have absolutely no idea whatsoever what outcome we will get on each trial, so the probability of getting $k$ heads on $N$ trials comes out to be exactly the same for all the different $k$s.

It's quite neat that the maths comes out like this.

$\endgroup$
4
$\begingroup$

To be more explicit, what you are describing can be viewed in conditional probability with an uninformative uniform prior on $p$.

That is, first draw $p \sim U[0,1]$, then draw $k \mid p \sim \operatorname{Bin} (N,p)$.

As you are aware, conditional probability tells us

$$ P(k) = \int P(k \mid p) P(p) dp $$

The other answers give good intuition: the uninformative prior in this case gives "no information" via $p$ of $k$, and it turns out in this case due to some mathematical symmetries of binomial distribution (the sum of independent Bernoulli trials) that if we know "nothing" about $p$ then we also know "nothing" about $k$ so it comes out as uniform.

And if I encountered this problem in real life and wanted to mathematically verify my intuition, there's actually a shortcut statisticians use to avoid any integrals:

Recall that Binomial likelihood with Beta conjugate prior (where continuous uniform distribution $U(0,1)$ is a special case $\operatorname{Beta}(1,1)$) has a beta-binomial posterior-predictive (marginalizing over $p$), plug-in $\alpha = \beta = 1$, and the discrete uniform distribution $U(0, N)$ again pops out :)

$\endgroup$
1
  • 3
    $\begingroup$ Fantastic answer, motivating the construction of the random variable over and beyond the result, and I think it gives a very decent and general explanation of the situation. Great stuff right here. $\endgroup$ Aug 17 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.