# Question about the Bayesian Inference of a parameter

In order to understand the difference between the Frequentist and Bayesian inference, I was reading the presentation at: http://www.stat.ufl.edu/archived/casella/Talks/BayesRefresher.pdf . In order to explain the difference between the approaches, the author is using the following example(Page 6): If I understand correctly, there are three different cases of protocols (something related to toxicity, it seems) and there are different studies for each of the three protocols, which has examined the relation of the protocol to a specific cause, which is called something as "AAN".

For example, for one of the three protocols, different studies showing the total number of cases (first) and relation to "AAN" (second) is as following :

1) 66,11

2) 1756,129

3) 272, 48

4) 151, 18

... etc. Each of these number pairs belong to different studies.

Now, the Bayesian model for these studies are given as such:

Given $X_j$ is the AAN frequency of the $j$th study about the protocol $i$, $X_j$ is distributed as $X_j \sim Binomial(n_j,p_i)$. $n_j$ is the total number of incidences for the $j$th study and $p_i$ is the parameter of the distribution we want to infer.

What I did not understand here is, the author says that, in this model $p_i$ can vary from study to study.

In my understanding of the Bayesian approach, here, all of the studies constitute our data, $D=\left( n_1,X_1,n_2,X_2,\ldots,n_K,X_K \right)$ where $K$ is the total number of studies. For $p_i$ we have a prior distribution of $P(p_i)$. So we try to find the posterior distribution $P(p_i\mid D)$. In my understanding $p_i$ cannot vary from study to study: $p_i$ is first generated from the prior distribution as $p_i \sim P(p_i)$ and then this generated value of $p_i$ is used to generate each $X_j \sim Binomial(n_j,p_i)$. Yes, $p_i$ is not fixed as in the frequentist approach, but it varies over different realizations of all $K$ studies, once a $p_i$ is generated from the prior, then all $K$ studies use the same $p_i$. So, it should not change from a single study to another. So, this slide has confused me at this particular point. Am I right with my thoughts here or did I misunderstand something?

In the example, there are three protocols. In the Bayesian viewpoint, the idea is that the number of events of interest is a binomial random variable whose probability of "success" is itself a random variable $p_i$.

Specific realizations of the data may be drawn from realizations of the underlying random variables in the model, but that does not mean that once those data are observed, the model parameters are fixed.

Perhaps an easier example is needed to illustrate this. Suppose I show you a coin and you are interested in some sort of statistical inference about whether the coin is biased in some way. Under the frequentist viewpoint, you would then want to take a sample and conduct a hypothesis test of the form $$H_0 : p = 0.5 \quad \mathrm{vs.} \quad H_a : p \ne 0.5$$ where $p$ is a fixed parameter that represents the probability that an individual toss of the coin will yield heads. You can pick a significance level $\alpha$, say $0.01$, which represents the maximum Type I error probability you are willing to tolerate--i.e., the probability you incorrectly infer that the coin is biased when in fact it is fair, purely due to random chance, must not exceed 1%. You can also set the power; say, you want $1-\beta = 0.9$; that is, you want your test to be able to detect if the coin is truly biased at least 90% of the time. From these criteria, you can calculate the necessary sample size assuming that the coin is fair, and you would conduct your experiment, calculate your test statistic, and arrive at a conclusion about $p$. Throughout, you assume that $p$ is a fixed but unknown parameter, and that sample you take is a specific realization of a random process that reflects the true value of this parameter.

Under a Bayesian viewpoint, you would impose a prior distribution on $p$; say, you could use a Jeffreys prior or a uniform prior, or a beta prior with specific hyperparameters if you have reason to believe that the coin is more likely to be biased in one way than another. You take a sample as before, but this time, you regard the sample as fixed data that represents information that updates your prior belief about how $p$ is distributed; i.e., in light of this data, you calculate a posterior distribution for $p$, conditional on what was observed. This posterior can then be turned into a credible interval, which tells you something about how likely $p$ is in a particular range of values. The more data you observe, the more information you have with which to update your belief about $p$, and the narrower this credible interval (hopefully) becomes. One way to make a "frequentist-style" inference about the biasedness of $p$ is to say that if the credible interval does not contain $0.5$, then you would conclude the coin is biased. But for the Bayesian, it is the entire posterior distribution that captures your belief about $p$, because as we have already discussed, $p$ is a random variable--it is not fixed.

While this notion of the probability of a single coin toss yields heads as a random variable seems counterintuitive on its face, it is no more so than the random observations of the outcome of the coin itself: that is what Casella is saying when he mentions that the frequentist regards the data as random and the parameter as fixed, whereas the Bayesian regards the data as fixed and the parameter as random. To a Bayesian, the outcomes are what they are. You saw them and it's a done deal. Given that information, you are free to adjust your belief about $p$. And why not? It is not necessarily the case that $p$ needs to be an immutable quantity for all time. Indeed, in many random processes, interpreting the underlying parameter(s) as fixed may not be a sensible assumption.

So, back to Casella's example. Say under the first protocol, you observe the data he gives in the table. The data is fixed, and it updates the prior distribution to a posterior distribution about $p_1$. That doesn't mean each $X_j$ for the first protocol shares the same $p_1$. Instead, it means that each $X_j$ for the first protocol shares the same posterior distribution for $p_1$. The more studies you do under the first protocol, the more information you have about this posterior, and you may become more certain about the behavior of this random variable, but it is nevertheless random. You can use the posterior of $p_1$ to generate a posterior predictive distribution on $X_{j(\mathrm{new})}$, that is to say, based on your updated knowledge of $p_1$, you can model the distribution of the number of events of interest observed for a hypothetical new study.

• I get more confused. Let's keep with the simple coin example you have given. We want to infer whether the coin is biased or not. In the Bayesian view, $p$ is a random variable, I am completely OK with that. We have a prior distribution on $p$, let's call it $P(p)$. Suppose that we have observed $N$ coin tosses $x_1,x_2,...,x_N$ and we assume that these tosses are independent, given $p$. So, the posterior probability $P(p|x_{1:N})$ becomes proportional to $P(p)\prod_{i=1}^{N}P(x_i|p)$. – Ufuk Can Bicici Aug 18 '14 at 17:48
• Thinking this as a generative process, $p$ comes from the prior distribution at the start and then the $N$ coin tosses use the same $p$. Then we can build the credible interval on the posterior distribution $P(p|x_{1:N})$. This is a sample of the Bayesian inference, isn't that right? If this is right, I did not get what is different in Casella's example about protocols and studies compared to this coin example. Isn't the posterior distribution for Casella's example $P(p_1|X_1,n_1,...,X_K,n_K)$? If it is so, how come we have different $p_1$s for each $X_j$? – Ufuk Can Bicici Aug 18 '14 at 17:52
• See, the beginning of your second comment is where your misunderstanding lies: each realization of a coin toss does not involve generating a specific realization of a $p$, because the posterior distribution collapses. The posterior distribution is obtained by computing the normalized likelihood over the entire support of $p$. That each $\Pr[X_i \mid p]$ are calculated with the same $p$ simply represents the joint probability of obtaining the sample given the prior probability $p$. This results in a posterior probability of $p$ given the data. (continued) – heropup Aug 18 '14 at 18:06
• Then, if you want to realize a new coin toss, you use the posterior predictive distribution, which is calculated using information from the entire posterior distribution on $p$. – heropup Aug 18 '14 at 18:09
• In other words, $\Pr[p \mid x_{1:N}]$ is a function of $p$, given the fixed data that was observed, that reflects in some sense the probability that the parameter equals $p$ given the data. But under this model, the individual $X_j$ that were observed did not necessarily come from a process with the same $p_1$. They were assumed to come from a process with the same prior distribution on $p_1$, because $p_1$ is a random variable. – heropup Aug 18 '14 at 18:15