# Does the parameter change during data generation in Bayesian Inference?

Let's assume that we have the following graphical model: This graph encodes the joint distribution $P(p,x_1,x_2,x_3,x_4) = P(p)\prod_{i=1}^{4}P(x_i|p)$. In the Bayesian inference, if we know $x_1,x_2,x_3$ then a full Bayesian predictive posterior for $x_4$ is given as:

$$P(x_4|x_3,x_2,x_1) = \int_{p}P(x_4|p)P(p|x_3,x_2,x_1)dp$$

I used to interpret a model like in the above as follows: Some probabilistic system picks a $p$ from $P(p)$. Then this picked $p$ generates data as $x_1\sim P(x_1|p),x_2\sim P(x_2|p),x_3\sim P(x_3|p)$. We do not know the true value of $p$ and therefore we integrate over all possible values of $p$ using its posterior distribution given the data in order to obtain the posterior predictive distribution of $x_4$.

In this interpretation, the value of $p$ stays the same once it is picked from the prior, during the data generation. But our lack of knowledge about it leads us to integrate over its all possible values in order to infer $x_4$. This can be considered as a generative model.

My question is, is my interpretation correct here? I am asking this, because I have seen some sources on web which imply that $p$ changes during generating $x_i$s. If it were that way, shouldn't each of these different values be named as different random variables like $p_1$ for $x_1$, $p_2$ for $x_2$, etc. ?

This has greatly confused me. I appreciate any comments.

In general, yes, your interpretation is correct. Bayesian models support a generative interpretation, unlike classical frequentist models (a grey area is mixed-effects modeling).

• Thanks for the answer. What other possible interpretations could we have as well as with the generative one? And could it be really possible that the parameters vary during the data generation? I think this leads to a completely different model, if we assume that every $x_i$ is generated by a different $p$ in my toy example model. Aug 19, 2014 at 16:13
• @UfukCanBiçici based on your formulation, the value for p should be fixed before generating the data. The other interpretation is as a subjective probability...which leads to the same mathematics.
– user76844
Aug 19, 2014 at 16:15
• Does the subjective probability interpretation mean that we assign our degree of beliefs as probability measures to each outcome $\omega$ in the sample space $\Omega$? I mean in the above model for example, each $(p,x_1,x_2,x_3,x_4)$ tuple gets a probability measure assigned to it? Actually I asked a related question very recently: math.stackexchange.com/questions/902742/… Aug 19, 2014 at 19:10
• @UfukCanBiçici we assign a degree of belief to each value of teh prior, f(p), which then supposedly generates the results g(x|p). So, yes, that is what you are doing with subjective probability, you are essentially carrying through your degree of belief through the normal probability machinery.
– user76844
Aug 19, 2014 at 19:14

I am posting my opinion about this, because I had the same question. I looked into page 83 of "Statistical Rethinking: A Bayesian Course with Examples in R and Stan," by Richard McElreath (1st Edition), and found the following code:

sample_mu <- rnorm(1e4, 178, 20)
sample_sigma <- runif(1e4, 0, 50)
prior_h <- rnorm(1e4, sample_mu, sample_sigma)
dens(prior_h)


Basically this code samples 10,000 heights from a normal distribution with mean mu and standard deviation sigma, where mu follows a normal prior, and sigma also follows a uniform prior.

The way the heights prior_h are sampled, indicates that for each observation of height, a brand new mu and sigma is drawn from the prior.

So to answer your question, I think that yes, each draw from the prior should be labeled with a subscript i=1,2,3,... But I guess this is just not done by convention.