2
$\begingroup$

From Wikipedia

  1. The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. $p(\theta \mid \alpha )$. ...

    The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. $p(\mathbf {X}\mid \theta )$ . This is also termed the likelihood,...

    The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. $$p(\mathbf {X}\mid \alpha )=\int_{\theta }p(\mathbf {X}\mid \theta )p(\theta \mid \alpha )\operatorname {d}\!\theta .$$

    The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference: $$ p(\theta \mid \mathbf {X},\alpha )={\frac {p(\mathbf {X}\mid\theta )p(\theta \mid \alpha )}{p(\mathbf {X} \mid \alpha )}}\propto p(\mathbf {X} \mid \theta )p(\theta \mid \alpha ) $$

    In the calculation of the marginal likelihod and posterior distribution, I wonder what is the reason that $p(\mathbf {X }\mid \theta )$ is not $p(\mathbf {X} \mid \theta, \alpha )$ instead?

  2. The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: $$ p(\tilde {x} \mid \mathbf {X},\alpha )=\int_{\theta}p(\tilde {x} \mid \theta )p(\theta \mid \mathbf {X},\alpha )\operatorname {d}\!\theta $$

    Why is $p(\tilde{x} \mid \theta )$ not $p(\tilde {x} \mid \theta, X, \alpha )$ instead?

Thanks!

$\endgroup$
6
  • $\begingroup$ The likelihood is not at all the same thing as the conditional distribution of the data given the parameters. The likelihood is a function of the parameters with the data fixed. $\endgroup$ Feb 10, 2014 at 16:42
  • $\begingroup$ @MichaelHardy: Thanks! What would you call $P(X|\theta)$ when $\theta$ is fixed? $\endgroup$
    – Tim
    Feb 10, 2014 at 17:48
  • 1
    $\begingroup$ @Tim likelihoods and densities have the same form...its just what you consider to be the variable. Likelihoods consider the data fixex, and the parameter $\theta$ as the varaible, densities do the opposite. This is an imporatnt distinction since likelihoods do not need to integrate to 1 but densities do. I.e., if you integrate the likelihood over all values of $\theta$ it will not necessarily be 1 (it can be infinite). $\endgroup$
    – user76844
    Feb 10, 2014 at 18:06
  • 1
    $\begingroup$ This is just a personal opinion, but I wouldn't start on wikipedia to learn math (I find it useful for reference, but not for learning). There are obviously a lot of resources out there, but Cross Validated has a very good selection of posts on the Bayesian approach, such as this one. $\endgroup$
    – TooTone
    Feb 10, 2014 at 22:54
  • $\begingroup$ @Tim : I would probably never write anything like "$P(X\mid\theta)$". I'm wondering if by that you mean the same thing you appear to mean by $p(x\mid\theta)$? One CAREFULLY distinguishes between $X$ and $x$, and thus becomes able to understand expressions like $P(X\le x)$, and between $P$ and $p$. And I find this practice of using the same letter, $p$, to refer to various different functions, horribly obnoxious. If $p(x)$ (with a lower-case $x$) means the density function of a random variable (capital) $X$, then $p(3)$ means that density evaluated at $3$. But then if..... $\endgroup$ Feb 11, 2014 at 19:09

3 Answers 3

1
$\begingroup$

The $\alpha$ are not random variables, but parameters of the assumed prior. Hence, they aren't events, and do not contribute to the conditional probability. This is why the likelihood of $\mathbf{X}$ does not include $\alpha$.

The same holds for $\mathbf{X}$ -- those are given values in the context of $\theta$. Therefore, when forming the predictive distiribution, $p(\theta |{\mathbf {X}},\alpha )$ already incorporates the information from the data on the probable values of $\theta$, hence the data are treated like parameters in a predictive setting.

$\endgroup$
1
$\begingroup$

For your first question

In the calculation of the marginal likelihod and posterior distribution, I wonder what is the reason that $p({\mathbf {X}}|\theta )$ is not $p({\mathbf {X}}|\theta, \alpha )$ instead?

$\alpha$ is a parameter of the probability density for $\theta$ i.e. it's a parameter of the prior. The likelihood $p({\mathbf{X}}|\theta )$ takes $\theta$ as a parameter not $\alpha$.

A simple example should help. Consider a beta prior with parameters $(a,b)$ for a binomial probability $\rho$. In this case for a single observation, $p({\mathbf{X}}|\theta )$ is of the form $\binom{\cdot}{\cdot}\rho^\cdot(1-\rho)^\cdot$ and the prior is proportional to $(1-\rho)^a\rho^b$. Here $\alpha = (a,b)$, and $\theta = \rho$.

On the second question

Why is $p({\tilde {x}}|\theta )$ not $p({\tilde {x}}|\theta, X, \alpha )$ instead?

The details are in Eupraxis1981's answer. I would simply say that the data and the hyper parameters don't appear in $p({\tilde {x}}|\theta )$, so conditioning on them is redundant. A similar example could be constructed in this case also.

$\endgroup$
1
$\begingroup$

It is assumed that $X$ is independant of $\alpha\,$ given $\theta$, and also that your new point is independant of $X$ given $\theta$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .