2
$\begingroup$

From Wikipedia

  1. The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. $p(\theta \mid \alpha )$. ...

    The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. $p(\mathbf {X}\mid \theta )$ . This is also termed the likelihood,...

    The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. $$p(\mathbf {X}\mid \alpha )=\int_{\theta }p(\mathbf {X}\mid \theta )p(\theta \mid \alpha )\operatorname {d}\!\theta .$$

    The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference: $$ p(\theta \mid \mathbf {X},\alpha )={\frac {p(\mathbf {X}\mid\theta )p(\theta \mid \alpha )}{p(\mathbf {X} \mid \alpha )}}\propto p(\mathbf {X} \mid \theta )p(\theta \mid \alpha ) $$

    In the calculation of the marginal likelihod and posterior distribution, I wonder what is the reason that $p(\mathbf {X }\mid \theta )$ is not $p(\mathbf {X} \mid \theta, \alpha )$ instead?

  2. The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: $$ p(\tilde {x} \mid \mathbf {X},\alpha )=\int_{\theta}p(\tilde {x} \mid \theta )p(\theta \mid \mathbf {X},\alpha )\operatorname {d}\!\theta $$

    Why is $p(\tilde{x} \mid \theta )$ not $p(\tilde {x} \mid \theta, X, \alpha )$ instead?

Thanks!

$\endgroup$
  • $\begingroup$ The likelihood is not at all the same thing as the conditional distribution of the data given the parameters. The likelihood is a function of the parameters with the data fixed. $\endgroup$ – Michael Hardy Feb 10 '14 at 16:42
  • $\begingroup$ @MichaelHardy: Thanks! What would you call $P(X|\theta)$ when $\theta$ is fixed? $\endgroup$ – Tim Feb 10 '14 at 17:48
  • 1
    $\begingroup$ @Tim likelihoods and densities have the same form...its just what you consider to be the variable. Likelihoods consider the data fixex, and the parameter $\theta$ as the varaible, densities do the opposite. This is an imporatnt distinction since likelihoods do not need to integrate to 1 but densities do. I.e., if you integrate the likelihood over all values of $\theta$ it will not necessarily be 1 (it can be infinite). $\endgroup$ – user76844 Feb 10 '14 at 18:06
  • 1
    $\begingroup$ This is just a personal opinion, but I wouldn't start on wikipedia to learn math (I find it useful for reference, but not for learning). There are obviously a lot of resources out there, but Cross Validated has a very good selection of posts on the Bayesian approach, such as this one. $\endgroup$ – TooTone Feb 10 '14 at 22:54
  • $\begingroup$ @Tim : I would probably never write anything like "$P(X\mid\theta)$". I'm wondering if by that you mean the same thing you appear to mean by $p(x\mid\theta)$? One CAREFULLY distinguishes between $X$ and $x$, and thus becomes able to understand expressions like $P(X\le x)$, and between $P$ and $p$. And I find this practice of using the same letter, $p$, to refer to various different functions, horribly obnoxious. If $p(x)$ (with a lower-case $x$) means the density function of a random variable (capital) $X$, then $p(3)$ means that density evaluated at $3$. But then if..... $\endgroup$ – Michael Hardy Feb 11 '14 at 19:09
1
$\begingroup$

The $\alpha$ are not random variables, but parameters of the assumed prior. Hence, they aren't events, and do not contribute to the conditional probability. This is why the likelihood of $\mathbf{X}$ does not include $\alpha$.

The same holds for $\mathbf{X}$ -- those are given values in the context of $\theta$. Therefore, when forming the predictive distiribution, $p(\theta |{\mathbf {X}},\alpha )$ already incorporates the information from the data on the probable values of $\theta$, hence the data are treated like parameters in a predictive setting.

$\endgroup$
1
$\begingroup$

For your first question

In the calculation of the marginal likelihod and posterior distribution, I wonder what is the reason that $p({\mathbf {X}}|\theta )$ is not $p({\mathbf {X}}|\theta, \alpha )$ instead?

$\alpha$ is a parameter of the probability density for $\theta$ i.e. it's a parameter of the prior. The likelihood $p({\mathbf{X}}|\theta )$ takes $\theta$ as a parameter not $\alpha$.

A simple example should help. Consider a beta prior with parameters $(a,b)$ for a binomial probability $\rho$. In this case for a single observation, $p({\mathbf{X}}|\theta )$ is of the form $\binom{\cdot}{\cdot}\rho^\cdot(1-\rho)^\cdot$ and the prior is proportional to $(1-\rho)^a\rho^b$. Here $\alpha = (a,b)$, and $\theta = \rho$.

On the second question

Why is $p({\tilde {x}}|\theta )$ not $p({\tilde {x}}|\theta, X, \alpha )$ instead?

The details are in Eupraxis1981's answer. I would simply say that the data and the hyper parameters don't appear in $p({\tilde {x}}|\theta )$, so conditioning on them is redundant. A similar example could be constructed in this case also.

$\endgroup$
1
$\begingroup$

It is assumed that $X$ is independant of $\alpha\,$ given $\theta$, and also that your new point is independant of $X$ given $\theta$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.