Why would a uniform prior distribution give a different result than a purely frequentist approach?

I would expect a uniform prior to be a good example of an uninformed prior and get the same result as the frequentist approach. However, this is not the case.

As an example, let's look the classical Bernoulli problem - you flip coins and $s$ are heads, $f$ are tails.

You want to predict $p$, the probability of heads on any given coin flip.

Frequentist Approach

The frequentist approach is to say that $p$ is simply $s/(s + f)$. In this approach, we assume nothing about the distribution of $p$ and we're simply finding the maximium likelihood estimate. This makes sense to me.

Bayesian approach

Now, let's say you do the Bayesian approach and you say:

$$P(p|data) = \dfrac{P(data | p) \cdot P(p)} {P(data)}$$

According to wikipedia, if you make your prior the beta distribution, you get:

$$P(p|data) = \dfrac{p^{s+\alpha-1}(1-p)^{f+\beta-1}}{B(s+\alpha, f+\beta)}$$

Now let's say we want to find the expected value of p:

$$E[p|data] = \int_{0}^{1} P(p|data) \cdot p \cdot dp$$ $$E[p|data] = \int_{0}^{1} \dfrac{p^{s+\alpha-1}(1-p)^{f+\beta-1}}{B(s+\alpha, f+\beta)} \cdot p \cdot dp$$ $$E[p|data] = \int_{0}^{1} \dfrac{p^{s+\alpha}(1-p)^{f+\beta-1}}{B(s+\alpha, f+\beta)} \cdot dp$$ $$E[p|data] = \dfrac{B(s+\alpha + 1, f+\beta)}{B(s+\alpha, f+\beta)}$$

Great! We can continuing simplifying this since we know a property of the beta function:

$$B(s + 1, f) = B(s, f) \cdot \dfrac{s} {s + f}$$

Which implies:

$$E[p|data] = \dfrac{B(s+\alpha, f+\beta)}{B(s+\alpha, f+\beta)} \cdot \dfrac{s+\alpha}{f+\beta}$$

$$E[p|data] = \dfrac{s+\alpha}{s+\alpha + f+\beta}$$

Makes sense so far! Now, let's say we want to choose a really "objective" prior to emulate the frequentist approach, like a uniform distribution. This is equivalent to Beta(1,1). In this case, we would get:

$$E[p|data;\alpha=1,\beta=1] = \dfrac{s+1}{s + f + 2}$$

This hurts my brain. Even though we're being as objective as we can ("p is equally likely to be anything"), we're getting a different result than the frequentist approach.

On the flip side, if we do the improper prior Beta(0,0), we get the same result as the frequentist approach:

$$E[p|data;\alpha=0,\beta=0] = \dfrac{s}{s + f}$$

What makes Beta(0,0) more similar to the frequentist approach than Beta(1,1)? What assumption are we making in Beta(1,1)? How is Beta(0,0) "less informed"?

• P.S. Bonus points if you can explain to me how $\int_{0}^{1} p^{s+\alpha}(1-p)^{f+\beta-1} \cdot dp = B(s + \alpha + 1, f + \beta)$. I used WolframAlpha to get that result! – Temuz Jan 21 '15 at 20:58
• That's straight from the definition of the Beta function. $B(x, y) \mathop{=}^\text{def}\int_0^1 t^{x-1} (1-t)^{y-1}\operatorname d t$ – Graham Kemp Jan 21 '15 at 21:15

The difference is in the prior expectation. The pure frequentists doesn't make any.

Suppose I show you a coin. Before I even toss it, what would you expect the bias to be?

The pure frequentist approach says that the probability is completely indeterminate, $0/0$. The approach only uses sample data to and cannot make claims without it. Divide by zero error.

The baysian approach says that we already expect the probability to be $1/2$ because we made assumptions on how coins work.

Next I toss the coin just once and it comes up heads. The pure frequentist now expects that the coin will have a probability of heads of $1/1$, while the baysian adjusts the expectation of the probability to be $2/3$.

Tossing the coin again and it comes up heads, the pure frequentist persists in measuring the probability as $1/1$ while the baysian adjusts it to $3/4$. And so on.

The pure frequentist always assumes that the sample data is exactly representative of the population of coin tosses, because that's all it can do. While the baysian applies a 'correction factor' due to the prior assumptions that can be made about the population.

Although, we can make better prior assumptions, if we have more information about coin manufacture or, say, the likelihood of me tampering with the coin. Assuming a uniform distribution of bias is fairly naïve.

Of course, as we take more and more samples the two approaches should converge towards an agreement that I used a two headed coin, or something.

• I guess I don't understand how a uniform prior distribution can lead to a different result than the frequentist approach. How does the additional information "p is equally probable to be anywhere here" change the estimate we make? – Temuz Jan 22 '15 at 0:06
• Okay, wait, I think I understand now! The original assumption of the coin being 1/1 is changing the current estimate, despite the fact that you have more data than you did before. Say you get 8 heads in a row - the original estimate is still pulling you closer to 50%. – Temuz Jan 22 '15 at 0:18
• Furthermore, if we go back to the case of Beta(0,0), since the pdf of Beta(0,0) is 0 for all values above 0 and below 1, this improper prior is basically not "pulling us" anywhere. That's why it isn't affecting our estimates. Is this correct? – Temuz Jan 22 '15 at 0:20
• And lastly, Beta(1,1) is the equivalent of initially giving the frequentist 1 heads and 1 tails to begin with! – Temuz Jan 22 '15 at 0:22

A frequentist maximum likelihood estimate is akin to the mode of the likelihood function, so in a Bayesian context with uniform prior, the MLE is like a mode of the posterior; whereas the expectation of the posterior is going to be influenced by the overall shape of the distribution.

Indeed, the mode of the posterior is called the "maximum a posteriori" estimate, which you can derive yourself and is $$\hat p_{MAP} = \frac{s+\alpha-1}{s+f+\alpha+\beta-2},$$ and with a uniform prior ($\alpha = \beta = 1$), this clearly reduces to the frequentist MLE.

• Okay, that doesn't make too much sense to me. Why is the frequentist maximum likelihood more like a mode? – Temuz Jan 21 '15 at 21:20
• Never mind, this makes sense! By definition, the maximum likelihood estimate is the value where the likelihood estimate function is at its highest. That's kind of like a mode. – Temuz Jan 21 '15 at 21:46
• In this case, isn't the frequentist expected value equal to their maximum likelihood estimate? Their expected value is 0 times the probability of 0 + 1 times the probability of 1. – Temuz Jan 21 '15 at 21:50
• @Temuz re: your most recent comment: the expected value of what quantity? In the frequentist sense, expectations are calculated for random variables. The MLE is a statistic, thus it is a collection of observations that are drawn from an underlying distribution, and its expectation is related to its bias. But there is no way to talk about the expectation of a parameter in the frequentist sense. The expected value of a single observation is of course $p$, but the point of estimation is that we don't know what $p$ is: we infer it from the sample. – heropup Jan 21 '15 at 22:13
• Okay, so say you're exclusively a frequentist and you want to provide the expected value of a single observation. Would you give the MLE? Probably not, right? You'd probably so something like $\sum\limits_{i} P(i) \cdot i$ for each possible outcome i and you'd guess P(i) based off historical data. In this case, it happens to be the same value as the MLE. – Temuz Jan 21 '15 at 23:48