the way of thinking of frequentist vs bayesian? I'm learning about bayesian inference and I've heard that there's also another inference called frequentist inference. I still can't understand the difference the way frequentist and bayesian count the probability, can someone give me a simple example of a problem, and how frequentist and bayesian solve the problem?
 A: Suppose you have polling data on Proposition A in an upcoming election.
In a random sample of $n = 1000$ expected voters $x = 620$ people favor
the proposition. 
Frequentist analysis. There is some unknown fixed parameter $\theta,$
which is the probability a randomly chosen person favors the proposition.
We have data from a binomial distribution with parameters $n = 1000$
and $\theta,$ and wish to estimate $\theta.$ 
The usual (maximum likelihood) point estimate is $\hat \theta = x/n = 620/1000 = 0.62.$
This estimate is based on maximizing the likelihood function
$$p(x| \theta) = {n \choose x}\theta^x(1-\theta)^{n-x} \propto \theta^x(1-\theta)^{n-x}$$
with respect to $\theta.$ Of course, we do not expect $\theta$ to be exactly
0.62, so we seek an interval that expresses our uncertainty about this result.
Without going into details (available in almost all elementary statistics texts),
a 95% confidence interval (CI), based on the normal approximation to binomial, is of the form 
$\hat \theta \pm 1.96\sqrt{\hat \theta(1-\hat \theta)/n}.$
Somewhat more precisely, we might use $\tilde n = n+4,$ and $\tilde \theta = (x+2)/\tilde n,$ to obtain the 'Agresti' confidence interval 
$\tilde \theta \pm 1.96\sqrt{\tilde \theta(1-\tilde \theta)/\tilde n}.$
The Agresi confidence interval for our poll is $(0.5895, 0.6496).$ 
Roughly, the idea is that there are $n+1$ possible CIs, one for each of
the possible values of $x.$ Some of them 'cover' (include) the 'true' value of $\theta,$ and some don't. For given values of $n$ and $\theta,$ the
binomial probability ${n \choose x}\theta^x(1-\theta)^{n-x}$ is associated
with each CI. Ideally (and for the Agresi-style interval nearly actually),
the intervals that cover the true value of $\theta$ have total probability
95%.
Thus the process by which the CI is constructed implies that it covers
the true value of $\theta$ with probability 95%. To a frequentist, this
means that the process 'works' 95% of the time, over the long run. 
Therefore, once $x$ is
observed and a particular CI is constructed, one must not say that 
the true $\theta$ lies in the that CI with 95% probability. Either it
lies in the interval or it does not. But we are allowed to say we have
95% 'confidence' that the CI covers the true $\theta.$
Bayesian analysis. You say you are familiar with Bayesian ideas, so
I will be briefer here. The crucial distinction is that we begin by
regarding $\theta$ as a random variable (not an unknown fixed constant). If you have no useful prior information about
Proposition A or the attitudes of the electorate, you might select
an uninformative prior such as $Unif(0,1) \equiv Beta(\alpha_0=1, \beta_0 =1).$
The likelihood function is as in the frequentist discussion above.
So the posterior distribution is given by
$$ p(\theta|x) = p(\theta)\times p(x|\theta) \propto 
\theta^{\alpha_0 -1}(1-\theta)^{\beta_0-1} \times \theta^x(1-\theta)^{n-x}
\propto \theta^{621-1}(1-\theta)^{381-1},$$
which we recognize as the kernel of $Beta(621, 381)$. We might use the
mean, median, or mode of the posterior distribution as a point estimate for $\theta$.
A probability symmetric Baysian probability interval for $\theta$ is 
$(0.5895, 0.6496).$
qbeta(c(.025,.975), 621, 381)
## 0.5894984 0.6495697

However, because an initial probability distribution for the random
variable $\theta$ is provided by the prior, we may say there is probability 95% 
that $\theta$ lies in the stated interval in the particular instance 
at hand.
Of course, if you have some meaningful prior information, you might
choose an informative prior. perhaps the mildly optimistic $Beta(330, 270)$.
Then the 'Bayesian probability interval' would be $(0.5696, 0.6177).$
Sometimes such intervals are called 'Bayesian credible intervals'.
Acknowledgment: The Bayesian example is condensed from Suess & Trumbo (2010), Ch 8.
Coverage probabilities for frequentist binomial CIs are discussed in Ch 1.
