1
$\begingroup$

I'm reading about Bayesian inference and there is one derivation I don't understand or see (from my book):

$\textbf{x} = (x_1, ..., x_n)$ is $n$-dimensional vector, $X = (\textbf{x}_1, ..., \textbf{x}_m)$ is a set of training examples and $\theta = (\theta_1, ..., \theta_k)$ is a set of parameters. $$p(\textbf{x}\:|\:X) = \int p(\textbf{x}, \theta\:|\:X)\;\text{d}\theta$$

From the definition of conditional probability densities we can then write

$$p(\textbf{x}, \theta \:|\: X) = p(\textbf{x}\:|\:\theta, X)\:p(\theta\:|\:X)$$

The first factor however is independent of $X$ since it is just our assumed form for the parametrized density, and is completely specified once the values of the parameters $\theta$ have been set. We therefore have

$$p(\textbf{x}\:|\:X) = \int p(\textbf{x}\:|\:\theta)\:p(\theta\:|\:X)\;\text{d}\theta$$

Looking carefully at the formula above and assuming that $p(\theta\:|\:X)$ is known , then $p(\textbf{x}\:|\:X)$ is nothing but the average of $p(\textbf{x}\:|\:\theta)$ with respect to $\theta$, that is,

$$p(\textbf{x}\:|\:X) = E_{\theta}[p(\textbf{x}\:|\:\theta)]$$

Somehow this confuses me :( Can someone clarify me in more detail why is the last part true? Is $Y = p(\textbf{x}\:|\:\theta)$ a random variable here now? What does the average with respect to $\theta$ mean ($E_{\theta}$)?

This confuses me, because if I would use the definition of expected value:

$$E(Y) = \int y\:p(y)\;\text{d}y$$

I would deduce in my case that $y = p(\textbf{x}\:|\:\theta)$ and $p(y) = p(p(\textbf{x}\:|\:\theta))$...and now I start to scratch my head x) I would really also want to know what does $E_{\theta}(Y)$ mean? How is expected value of random variable $\alpha$ with respect to variable $\beta$, $E(\alpha)_{\beta}$ defined?

Thank your for any help! =)

$\endgroup$
5
  • $\begingroup$ Which text is this from? $\endgroup$ – Did Jul 1 '13 at 8:05
  • $\begingroup$ Pattern recognition by Sergios Theodoridis 4th edition (pages 39-40) and Neural networks for pattern recognition, Bishop (pages 42-43) $\endgroup$ – jjepsuomi Jul 1 '13 at 8:08
  • 2
    $\begingroup$ If you interpret $\theta$ as a random variable, then $Y = p(\mathbb x \mid \theta)$ is also a random variable. ($\mathbb x$ is fixed here.) $E_\theta$ here should mean the expectation when $\theta$ is considered a random variable. $\endgroup$ – Tunococ Jul 1 '13 at 8:19
  • 1
    $\begingroup$ What Tunococ said is correct. Thinking of it that way, the following equation makes sense: $E_\theta[p(\mathbf{x}|\theta)] = \int p(\mathbf{x}|\theta)p(\theta)d\theta$, but the equation in your question has $p(\theta |X)$ in the integrand instead of just $p(\theta )$. Indeed, the concluding equation has a left-hand side which depends on $X$ and a right-hand side which doesn't, so something seems a bit off to me. $\endgroup$ – Amit Kumar Gupta Jul 3 '13 at 6:32
  • $\begingroup$ @AmitKumarGupta Thank you for your help, I took these equations from the books I mentioned. Yes $p(\theta|X)$ seems a bit off to me too. Otherwise it would be more clearer :) $\endgroup$ – jjepsuomi Jul 3 '13 at 6:57
2
$\begingroup$

Perhaps it would help if you instead considered the likelihood. That is, let $p(\mathbf{x}|\theta)=L(\theta|\mathbf{x})$, as $\mathbf{x}$ is considered fixed making it more sensible to consider it as a function of $\theta$. Then what you have is $$ p(\mathbf{x}|X)=\int p(\mathbf{x}|\theta)p(\theta|X)d\theta=\int L(\theta|\mathbf{x})p(\theta|X)d\theta. $$ If we then proceed by momentarily being a little sloppy and dropping the conditioning, the expression is essentially $$ \int L(\theta|\mathbf{x})p(\theta|X)d\theta\equiv\int g(\theta)p(\theta)d\theta=E_\theta[g(\theta)]. $$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.