I often seen the posterior predictive distribution mentioned in the context of machine learning and bayesian inference. The definition is as follows:

$ p(D'|D) = \int_\theta p(D'|\theta)p(\theta|D)$

How/why does the integral on the right equal the probability distribution on the left? In other words, which laws of probability can I use to derive $p(D'|D)$ given the integral?

Edit - After further consideration, I think I am able to see much of the derivation. That is,

$p(D'|D) = \int_\theta p(D', \theta | D)$ via the law of total probability
$p(D'|D) = \int_\theta p(D' | D, \theta) * p(\theta | D)$ via the chain rule

But I don't understand why $D$ may be dropped from the list of conditioned variables belonging to the integral's first term.

  • $\begingroup$ This definitely isn't a bad place to ask this. But also you may find stats.stackexchange.com to be a good place. They deal with a lot of machine learning as well as general statistics and probability theory there. $\endgroup$ – frogeyedpeas Jan 10 '16 at 5:12
  • $\begingroup$ Thanks for the suggestion. Is it frowned upon to cross-post to both boards? $\endgroup$ – bronxbomber92 Jan 10 '16 at 5:13
  • $\begingroup$ Generally yes, you want to have one question on one site, but in the event that it doesn't get enough activity here say in a week, it wouldn't be a bad idea to post there. In the meantime any researching, digging at articles, trying to make sense of some guide, that you do wouldn't also be bad to post here. You might get very close and only need the smallest "hint" $\endgroup$ – frogeyedpeas Jan 10 '16 at 5:29
  • $\begingroup$ I'd say cross-posting should happen only infrequently (although I agree with @frogeyedpeas that this question is good for both forums). However, in any event you should definitely mention in each that you have cross-posted and update both when you get an answer. $\endgroup$ – JimB Jan 10 '16 at 5:30
  • $\begingroup$ As a follow up, I found later that this is clearly explained in a lecture on Bayesian methods at the University of Edinburgh. The video can be found here and the explanation takes place about 8 minutes through the "Bayesian methods" video. $\endgroup$ – bronxbomber92 Jan 22 '16 at 3:32

$p(D',\theta | D) = p(D' | \theta,D)p(\theta | D)$ is from Bayes rules, provided we have densities:

$p(D',\theta | D) = \frac{P(D', \theta, D)}{P(D)} = \frac{P(D'|\theta, D) P(\theta, D)}{P(D)} = P(D'|\theta, D) P(\theta | D)$.

Now integrate out the nuisance variable $\theta$ on both sides. Your formula also appears to have a Markov-type assumption $p(D'|\theta,D)=p(D'|\theta)$.

  • $\begingroup$ Can you explain what is meant by "Markov-type assumption?" $\endgroup$ – bronxbomber92 Jan 10 '16 at 19:21
  • $\begingroup$ That's the assumption that $D'$ is independent of $D$ conditional on $\theta$, equivalent to the formula that I wrote above. This is characteristic of Markov processes but my referring to it as a Markov assumption rather than conditional independence may be completely misleading in your context. $\endgroup$ – snarfblaat Jan 10 '16 at 22:45

To show this one can follow a somewhat standard argument. In what follows, for notational convenience, I have replaced your "$D$"s with "$S$"s. By the law of total expectation (in terms of conditional expectation) and Fubini's theorem, applied to any bounded measurable function $f$ defined on the relevant sample space $\Omega$, we observe that $$ \eqalign{ \int_{\Omega}f(s^{'})p(s^{'}\mid s)\mathrm ds^{'}&=\mathbb E[f(S^{'})\mid S=s]=\mathbb E[E[f(S^{'})\mid \Theta\,,s]\mid S=s]\\&=\int_{\theta}\left(\int_{\Omega}f(s^{'})p(s^{'}\mid \theta\,,s) \mathrm ds^{'}\right)p(\theta\mid s)\mathrm d\theta \\&= \int_{\Omega}f(s^{'})\left(\int_{\theta}p(s^{'}\mid \theta\,,s)p(\theta\mid s)\mathrm d\theta\right) \mathrm ds^{'}} $$

Since the far l.h.s. is equal to the far r.h.s. for all bounded measurable functions, we conclude that $$ p(s^{'}\mid s)=\int_{\theta}p(s^{'}\mid \theta\,,s)p(\theta\mid s)\mathrm d\theta $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.