# Why does $p(\textbf{x} | X) = E_{\theta}[p(\textbf{x}|\theta)]$

I'm reading about Bayesian inference and there is one derivation I don't understand or see (from my book):

$\textbf{x} = (x_1, ..., x_n)$ is $n$-dimensional vector, $X = (\textbf{x}_1, ..., \textbf{x}_m)$ is a set of training examples and $\theta = (\theta_1, ..., \theta_k)$ is a set of parameters. $$p(\textbf{x}\:|\:X) = \int p(\textbf{x}, \theta\:|\:X)\;\text{d}\theta$$

From the definition of conditional probability densities we can then write

$$p(\textbf{x}, \theta \:|\: X) = p(\textbf{x}\:|\:\theta, X)\:p(\theta\:|\:X)$$

The first factor however is independent of $X$ since it is just our assumed form for the parametrized density, and is completely specified once the values of the parameters $\theta$ have been set. We therefore have

$$p(\textbf{x}\:|\:X) = \int p(\textbf{x}\:|\:\theta)\:p(\theta\:|\:X)\;\text{d}\theta$$

Looking carefully at the formula above and assuming that $p(\theta\:|\:X)$ is known , then $p(\textbf{x}\:|\:X)$ is nothing but the average of $p(\textbf{x}\:|\:\theta)$ with respect to $\theta$, that is,

$$p(\textbf{x}\:|\:X) = E_{\theta}[p(\textbf{x}\:|\:\theta)]$$

Somehow this confuses me :( Can someone clarify me in more detail why is the last part true? Is $Y = p(\textbf{x}\:|\:\theta)$ a random variable here now? What does the average with respect to $\theta$ mean ($E_{\theta}$)?

This confuses me, because if I would use the definition of expected value:

$$E(Y) = \int y\:p(y)\;\text{d}y$$

I would deduce in my case that $y = p(\textbf{x}\:|\:\theta)$ and $p(y) = p(p(\textbf{x}\:|\:\theta))$...and now I start to scratch my head x) I would really also want to know what does $E_{\theta}(Y)$ mean? How is expected value of random variable $\alpha$ with respect to variable $\beta$, $E(\alpha)_{\beta}$ defined?

Thank your for any help! =)

• Which text is this from? – Did Jul 1 '13 at 8:05
• Pattern recognition by Sergios Theodoridis 4th edition (pages 39-40) and Neural networks for pattern recognition, Bishop (pages 42-43) – jjepsuomi Jul 1 '13 at 8:08
• If you interpret $\theta$ as a random variable, then $Y = p(\mathbb x \mid \theta)$ is also a random variable. ($\mathbb x$ is fixed here.) $E_\theta$ here should mean the expectation when $\theta$ is considered a random variable. – Tunococ Jul 1 '13 at 8:19
• What Tunococ said is correct. Thinking of it that way, the following equation makes sense: $E_\theta[p(\mathbf{x}|\theta)] = \int p(\mathbf{x}|\theta)p(\theta)d\theta$, but the equation in your question has $p(\theta |X)$ in the integrand instead of just $p(\theta )$. Indeed, the concluding equation has a left-hand side which depends on $X$ and a right-hand side which doesn't, so something seems a bit off to me. – Amit Kumar Gupta Jul 3 '13 at 6:32
• @AmitKumarGupta Thank you for your help, I took these equations from the books I mentioned. Yes $p(\theta|X)$ seems a bit off to me too. Otherwise it would be more clearer :) – jjepsuomi Jul 3 '13 at 6:57

Perhaps it would help if you instead considered the likelihood. That is, let $p(\mathbf{x}|\theta)=L(\theta|\mathbf{x})$, as $\mathbf{x}$ is considered fixed making it more sensible to consider it as a function of $\theta$. Then what you have is $$p(\mathbf{x}|X)=\int p(\mathbf{x}|\theta)p(\theta|X)d\theta=\int L(\theta|\mathbf{x})p(\theta|X)d\theta.$$ If we then proceed by momentarily being a little sloppy and dropping the conditioning, the expression is essentially $$\int L(\theta|\mathbf{x})p(\theta|X)d\theta\equiv\int g(\theta)p(\theta)d\theta=E_\theta[g(\theta)].$$