I'm learning about bayesian inference and I've heard that there's also another inference called frequentist inference. I still can't understand the difference the way frequentist and bayesian count the probability, can someone give me a simple example of a problem, and how frequentist and bayesian solve the problem?


Suppose you have polling data on Proposition A in an upcoming election. In a random sample of $n = 1000$ expected voters $x = 620$ people favor the proposition.

Frequentist analysis. There is some unknown fixed parameter $\theta,$ which is the probability a randomly chosen person favors the proposition. We have data from a binomial distribution with parameters $n = 1000$ and $\theta,$ and wish to estimate $\theta.$

The usual (maximum likelihood) point estimate is $\hat \theta = x/n = 620/1000 = 0.62.$ This estimate is based on maximizing the likelihood function $$p(x| \theta) = {n \choose x}\theta^x(1-\theta)^{n-x} \propto \theta^x(1-\theta)^{n-x}$$ with respect to $\theta.$ Of course, we do not expect $\theta$ to be exactly 0.62, so we seek an interval that expresses our uncertainty about this result.

Without going into details (available in almost all elementary statistics texts), a 95% confidence interval (CI), based on the normal approximation to binomial, is of the form $\hat \theta \pm 1.96\sqrt{\hat \theta(1-\hat \theta)/n}.$ Somewhat more precisely, we might use $\tilde n = n+4,$ and $\tilde \theta = (x+2)/\tilde n,$ to obtain the 'Agresti' confidence interval $\tilde \theta \pm 1.96\sqrt{\tilde \theta(1-\tilde \theta)/\tilde n}.$ The Agresi confidence interval for our poll is $(0.5895, 0.6496).$

Roughly, the idea is that there are $n+1$ possible CIs, one for each of the possible values of $x.$ Some of them 'cover' (include) the 'true' value of $\theta,$ and some don't. For given values of $n$ and $\theta,$ the binomial probability ${n \choose x}\theta^x(1-\theta)^{n-x}$ is associated with each CI. Ideally (and for the Agresi-style interval nearly actually), the intervals that cover the true value of $\theta$ have total probability 95%.

Thus the process by which the CI is constructed implies that it covers the true value of $\theta$ with probability 95%. To a frequentist, this means that the process 'works' 95% of the time, over the long run.

Therefore, once $x$ is observed and a particular CI is constructed, one must not say that the true $\theta$ lies in the that CI with 95% probability. Either it lies in the interval or it does not. But we are allowed to say we have 95% 'confidence' that the CI covers the true $\theta.$

Bayesian analysis. You say you are familiar with Bayesian ideas, so I will be briefer here. The crucial distinction is that we begin by regarding $\theta$ as a random variable (not an unknown fixed constant). If you have no useful prior information about Proposition A or the attitudes of the electorate, you might select an uninformative prior such as $Unif(0,1) \equiv Beta(\alpha_0=1, \beta_0 =1).$

The likelihood function is as in the frequentist discussion above. So the posterior distribution is given by $$ p(\theta|x) = p(\theta)\times p(x|\theta) \propto \theta^{\alpha_0 -1}(1-\theta)^{\beta_0-1} \times \theta^x(1-\theta)^{n-x} \propto \theta^{621-1}(1-\theta)^{381-1},$$

which we recognize as the kernel of $Beta(621, 381)$. We might use the mean, median, or mode of the posterior distribution as a point estimate for $\theta$. A probability symmetric Baysian probability interval for $\theta$ is $(0.5895, 0.6496).$

qbeta(c(.025,.975), 621, 381)
## 0.5894984 0.6495697

However, because an initial probability distribution for the random variable $\theta$ is provided by the prior, we may say there is probability 95% that $\theta$ lies in the stated interval in the particular instance at hand.

Of course, if you have some meaningful prior information, you might choose an informative prior. perhaps the mildly optimistic $Beta(330, 270)$. Then the 'Bayesian probability interval' would be $(0.5696, 0.6177).$ Sometimes such intervals are called 'Bayesian credible intervals'.

Acknowledgment: The Bayesian example is condensed from Suess & Trumbo (2010), Ch 8. Coverage probabilities for frequentist binomial CIs are discussed in Ch 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.