11
$\begingroup$

Consider the following characterization of the Bayes' theorem:

Bayes' Theorem

Given some observed data $x$, the posterior probability that the paramater $\Theta$ has the value $\theta$ is $p(\theta \mid x) = p(x \mid \theta) p (\theta) / p(x)$, where $p(x \mid \theta)$ is the likelihood, $p(\theta)$ is the prior probability of the value $\theta$, and $p(x)$ is the marginal probability of the value $x$.

Is there any special reason why we call $p(x)$ the "marginal probability"? What is "marginal" about it?

$\endgroup$
2
  • 2
    $\begingroup$ "Marginal" does not mean "barely making the grade" but that the probability has been derived from a joint probability. The numerator is this instance is $p(x\mid \theta)p(\theta) = p(x,\theta)$ which is the joint probability and $p(x)$ (as well as $p(\theta)$) are marginal probabilities of they are derived from $p(x,\theta)$. $\endgroup$
    – Dilip Sarwate
    Jun 26, 2015 at 2:40
  • $\begingroup$ "Marginal probability" means the same thing here that it means in other contexts, i.e. "unconditional". (The meaning is context-dependent, since all probabilities are conditional.) $\qquad$ $\endgroup$
    – Michael Hardy
    Sep 27, 2016 at 7:20

3 Answers 3

Reset to default
6
$\begingroup$

If you consider a joint distribution to be a table of values in columns and rows with there probabilities entered in the cells, then the "marginal distribution" is found by summing the values in the table along rows (or columns) and writing the total in the margins of the table.

$$\begin{array}{c c} & X \\ \Theta & \boxed{\begin{array}{c|cc|c} ~ & 0 & 1 & X\mid \Theta \\ \hline 0 & 0.15 & 0.35 & 0.5 \\ 1 & 0.20 & 0.30 & 0.5 \\\hline \Theta\mid X & 0.35 & 0.65 & ~\end{array}}\end{array}$$

$\endgroup$
2
  • 1
    $\begingroup$ Yes, though I am not sure I understand your $\Theta \mid X$ or $X \mid \Theta$. I would have thought $0.65$ was $\mathbb{P}(X=1)$ $\endgroup$
    – Henry
    Sep 27, 2016 at 7:23
  • 2
    $\begingroup$ I hope you are aware of the fact that your answer is confusing. $\Theta \mid X$ is often used to represent "$\Theta$ given $X$" (a conditional) and not to represent a marginal. A marginal is just X or $\Theta$. Furthermore, the OP asked why p(x), the denominator is called a "marginal" probability. I think that the doubt lies in the fact that p(x) is called marginal, whereas $p(\theta)$ is called prior, but both can be calculated as marginals. $\endgroup$
    – user168764
    Jan 18, 2019 at 23:49
6
$\begingroup$

To me, bayes theorem is all about inverting likelihood functions, and in that context calling it marginal probabity makes sense.

  • Lets say I have a observation $c$,
  • and a collection of states $\mathbf{s}=\{s_1,\ldots,s_n\}$, that could be causing that observation.
  • And each of those states also defines a likelihood: $P(c\mid s_i)$
  • as well we have a prior $P(s_i)$ (I'm assuming you have already motivated the prior, if not ask another question on this site)
  • So I want to know the state, based on the variable
  • If I just wanted to know the most likely state, and how they compair to each other, I could define a scoring function -- combining the likelihood of our observation given we are in the state, with the base chance of being in the state: $$\operatorname{score}_c(s_i)= P(c\mid s_i)P(s_i)$$
  • Then to find the most likely state $s^\star$, i would just find the argmax $$s^\star = \operatorname{argmax}_{\forall s_i \in \mathbf{s}} \operatorname{score}_c(s_i) = \operatorname{argmax}_{\forall s_i \in \mathbf{s}} P(c\mid s_i)P(s_i) $$
  • That score function is quite nice. We can think of a score vector, which has all the scores and we can see which is the most likely, and which is the least. But it does not sum to one. We'd like to make it sum to one -- we would normalise it and call it a probability (even if it isn't -- but it will turn out it is). Our normalised score depends only on $c$ so it will be $P(s_i\mid c)$, proper justification of this is beyond scope of this answer. The normalised score is given by $$P(s_i\mid c)=\dfrac{\operatorname{score}_c(s_i)}{\sum_{\forall s_j\in \mathbf{s}} \operatorname{score}_c(s_j) } = \dfrac{P(c\mid s_i)P(s_i)}{\sum_{\forall s_j\in \mathbf{s}} P(c\mid s_j)P(s_j) }$$
  • the above is a very useful form of Bayes' Theorem.
  • let's take a closer look at the bottom line: $$\sum_{\forall s_j\in \mathbf{s}} P(c\mid s_j)P(s_j) = \sum_{\forall s_j\in \mathbf{s}} P(c,s_j)$$
  • So we are summing the Joint probability, over all possible values that one of its fields can take. That is the very definition of the marginal probability of the other field. $$P(c) = \sum_{\forall s_j\in \mathbf{s}} P(c,s_j)$$
  • Our bottom line -- the normalising factor to make it sum to one -- that is just the marginal probability of $c$. Substituting that back in: $$P(s_i\mid c) = \dfrac{P(c\mid s_i)P(s_i)}{P(c)}$$

So the bottom line $P(c)$ was just a marginally probability, that we find by summing over all possible values for the other field ($s_i$) in the top line.

$\endgroup$
3
  • 1
    $\begingroup$ Superb explanation. If you (or anyone else) could provide a motivation for the prior, i'd be grateful. $\endgroup$
    – blz
    Nov 2, 2018 at 14:52
  • $\begingroup$ Please ask a separate question and link back to this QA $\endgroup$
    – Frames Catherine White
    Nov 3, 2018 at 15:23
  • $\begingroup$ Thank you for giving this answer. Amazing explanation! $\endgroup$
    – Rachid Atmai
    Apr 3, 2020 at 8:15
3
$\begingroup$

The explanation I was given when I was taught conditional probabilities is that if you draw up a table of the probabilities $p(x,y)$, then the row/column sums $$ p(x) = \sum_{y} p(x,y) $$ (by the law of total probability) are written in the margins of the table.

$\endgroup$
6
  • $\begingroup$ By $p(x,y)$ do you just mean $p(x \wedge y)$ (i.e., the probability of $x$ and $y$ co-occurring)? $\endgroup$
    – PP121
    Jun 26, 2015 at 2:42
  • 1
    $\begingroup$ @PP121 Yes. It's an abbreviation for the joint probability. More specifically $p_{X,Y}(x,y) = \mathsf P(X=x \cap Y=y)$. $\endgroup$
    – Graham Kemp
    Jun 26, 2015 at 2:44
  • $\begingroup$ So then what is analogous to $X$ and $Y$ in the original example I wrote out? Is it $X = \{\ldots x \ldots\}$ and $\Theta = \{\ldots \theta \ldots \}$? $\endgroup$
    – PP121
    Jun 26, 2015 at 2:49
  • 1
    $\begingroup$ Yes. @PP121 That would be so. $p(x\mid \theta) = P(X=x\mid \Theta=\theta)$ $\endgroup$
    – Graham Kemp
    Jun 26, 2015 at 2:51
  • 1
    $\begingroup$ @PP121 No; it is literal, at least for discrete random variables. $X$ being a discrete random variable means that, on inspection, it will be found to have one of the values within the sample space with a certain probability. For continuous random variables the appropriate measure is a probability density and things are somewhat more involved, but mostly the same principles apply. $\endgroup$
    – Graham Kemp
    Jun 26, 2015 at 3:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .