# Confusing paragraph about Bayesian priors and marginal likelihood

I'm somewhat confused by the following paragraph, taken from Koller's book , on how $P(D)$ is the "probability of seeing this particular data set given our prior beliefs". I would have thought that was the definition of $P(D|\theta)$, not $P(D)$. Furthermore, if we integrate over all $\theta$, then how would it matter what prior beliefs we have for $\theta$? That is, how would $P(D)$ change if we change beliefs about $\theta$?

• Can you give the page number for this? – Bitwise Jul 29 '13 at 0:06
• @Bitwise pg. 738 – T. Webster Jul 29 '13 at 3:19

Consider the following rather curious question:

Suppose that you witness a series of ten coin tosses. All of the tosses come up heads. What is the probability that this is a two-headed coin?

Your data $\mathcal D$ is "the coin came up heads ten times in a row". The parameter $\theta$ is either "the coin is fair" or "the coin has two heads".

We're dealing with discrete rather than continuous probability here, so we'll replace the integral sign with a summation sign and the equation from the text becomes,

$$P(\mathcal D) = \sum_\theta P(\mathcal D | \theta) P(\theta)$$

which in this case expands to

\begin{align} P(\mathcal D) = & P(\mathcal D \mid \textrm{"coin is fair"}) \cdot P(\textrm{"coin is fair"})\\ + & P(\mathcal D \mid \textrm{"coin has two heads"}) \cdot P(\textrm{"coin has two heads"}) \end{align}

It's obvious that

$$P(\mathcal D \mid \textrm{"coin is fair"}) = \frac{1}{2^{10}}$$

and

$$P(\mathcal D \mid \textrm{"coin has two heads"}) = 1$$

But how on Earth are you supposed to evaluate $P(\textrm{"coin is fair"})$ or $P(\textrm{"coin has two heads"})$? Mathematics alone doesn't have an answer to that question. You need to supply one based on common sense. The answer you choose to supply is called your Bayesian prior.

If you believe, for example, that you have a one in five hundred chance of encountering a two-headed coin given whatever particular set of circumstances you find yourself in, then the calculation becomes:

$$P(\mathcal D) = \frac{1}{2^{10}} \cdot \frac{499}{500} + 1 \cdot \frac{1}{500} \approx 2.975 \cdot 10^{-3}$$

In other words, seeing the ten heads in a row is a pretty surprising result.

However, given this result, we can now use Bayes' law to calculate $P(\textrm{"coin has two heads"} \mid \mathcal D)$, which is your posterior belief in the likelihood of this being a two-headed coin.

\begin{align} P(\textrm{"coin has two heads"} \mid \mathcal D) = & \frac{P(\mathcal D \mid \textrm{"coin has two heads"}) \cdot P(\textrm{"coin has two heads"})}{P(\mathcal D)} \\ \approx & \frac{1 \cdot \frac{1}{500}}{2.975 \cdot 10^{-3}} \\ \approx & \mathbf{0.6723} \end{align}

Which is the answer to the question I posed to you.

You asked, "how would $P(\mathcal D)$ change if we change beliefs about $\theta$?".

Well, let's say that instead of this experiment taking place at a reputable casino in Vegas, it took place at a seedy carnival booth. Your prior estimation of the likelihood of seeing a two-headed coin would then be considerably higher: let's say one in ten. Now you calcuate,

$$P(\mathcal D) = \frac{1}{2^{10}} \cdot \frac{9}{10} + 1 \cdot \frac{1}{10} \approx 0.10088$$

meaning you're a lot less suprised than before at seeing the ten heads in a row. This is why $P(\mathcal D)$ depends on your beliefs about $P(\theta)$.