A Measure Theoretic formulation of Bayes' Theorem

Question

I am trying to find a measure theoretic formulation of Bayes' theorem, when used in statistical inference, Bayes' theorem is usually defined as:

$$p\left(\theta|x\right) = \frac{p\left(x|\theta\right) \cdot p\left(\theta\right)}{p\left(x\right)}$$

where:

• $p\left(\theta|x\right)$: the posterior density of the parameter.
  
• $p\left(x|\theta\right)$: the statistical model (or likelihood).
  
• $p\left(\theta\right)$: the prior density of the parameter.
  
• $p\left(x\right)$: the evidence.

Now how would we define Bayes' theorem in a measure theoretic way?
So, I started by defining a probability space:

$$\left(\Theta, \mathcal{F}_\Theta, \mathbb{P}_\Theta\right)$$

such that $\theta \in \Theta$.
I then defined another probability space:

$$\left(X, \mathcal{F}_X, \mathbb{P}_X\right)$$

such that $x \in X$.
From here now on I don't know what to do, the joint probability space would be:

$$\left(\Theta \times X, \mathcal{F}_\Theta \otimes \mathcal{F}_X, ?\right)$$

but I don't know what the measure should be.
Bayes' theorem should be written as follow:

$$? = \frac{? \cdot \mathbb{P}_\Theta}{\mathbb{P}_X}$$

where:

$$\mathbb{P}_X = \int_{\theta \in \Theta} ? \space \mathrm{d}\mathbb{P}_\Theta$$

but as you can see I don't know the other measures and in which probability space they reside.
I stumbled upon this thread but it was of little help and I don't know how was the following measure-theoretic generalization of Bayes' rule reached:

$${P_{\Theta |y}}(A) = \int\limits_{x \in A} {\frac{{\mathrm d{P_{\Omega |x}}}}{{\mathrm d{P_\Omega }}}(y)\mathrm d{P_\Theta }}$$

I'm self-learning measure theoretic probability and lack guidance so excuse my ignorance.

Answer 1

I'm not 100% convinced by the expression in the linked thread. The notion of conditional probability itself is not itself so easy to express in a measure-theoretic way. I will try to restrict my answer to the more basic formulations.

The first thing to note is that everything is defined on the same probability space $(\Omega, \mathcal{F}, \mathbf{P})$. The quantities $\theta$ and $X$ are then random variables taking values in some spaces $\Theta, \mathcal{X}$ respectively. In particular, the random variable $(\theta, X)$ has a joint distribution, which is the object of Bayesian statistical inquiry.

Now, the statistical set-up is that $\theta$ has a marginal distribution $\pi$ (which is called the prior), and the statistical model is a family of conditional distributions, which specify the distribution of $X$ based on the parameter values. In particular, we have what's called a transition kernel $\nu \colon \Theta \times \mathcal{F}_{\mathcal{X}} \rightarrow [0, 1]$, which encodes the conditional likelihood through $$ \mathbf{P}(X \in A|\theta) = \nu(\theta, A). $$

For measurability reasons, we demand that each $\nu(\theta, \cdot)$ is a probability measure and that each $\nu(\cdot, A)$ is a measurable function of $\theta$.

To make sense of Bayes rule, we also demand that each $\nu(\theta, \cdot)$ is absolutely continuous with respect to a common $\sigma$-finite carrying measure $\mu$ (this is not much of a condition, since in 99.9% of cases this is the Lebesgue measure on $\mathbf{R}^n$, the counting measure on $\mathbf{Z}$ or some combination of these). This condition allows us to define the conditional likelihood of our model: $$ f(x|\theta) := \frac{\mathrm{d} \nu(\theta, \cdot)}{\mathrm{d} \mu}(x), $$ which is measurable as a function of $x$ from the Radon-Nikodym theorem and as a function of $\theta$ from our regularity condition on $\nu$.

With all this set-up, the measure-theoretic formulation of Bayes theorem is that for each $x \in \mathcal{X}$, the conditional distribution of $\theta|X = x$ is defined by the probability measure $$ \pi(\mathrm{d} \theta | x) \propto f(x | \theta) \pi(\mathrm{d} \theta). $$

More fully (and including the constant of proportionality), this is the statement: $$ \mathbf{P}(\theta \in \Gamma | X = x) = \frac{\int_{\Gamma} f(x | \theta) \pi(\mathrm{d} \theta)}{\int_{\Theta} f(x | \theta) \pi(\mathrm{d} \theta)}. $$

Answer 2

This is the content of the DISINTEGRATION THEOREM, which we below reword in the context of the original question.
First of all, let's start with some definitions:
$\Theta$ is the parameter space, $X$ is the sample space, and typically in Bayesian inference, we kick off with a probability measure $\Bbb{P}_{\Theta}(\theta)$ on $\Theta$, and a conditional probability measure $\Bbb{P}_{X | \Theta}(\cdot | \theta)$ on $X$ for $\Bbb{P}_{\Theta}$-almost every $\theta \in \Theta$.
Together, these give us a joint probability measure $$\Bbb{P}_{X \times \Theta}(x, \theta) := \Bbb{P}_{\Theta}(\theta) \times \Bbb{P}_{X | \Theta}(x | \theta)$$ on $X \times \Theta.$
Now, if we assume that for $\Bbb{P}_{X \times \Theta}$-almost every $(x, \theta) \in X \times \Theta$, the marginal probability $\Bbb{P}_X(x)$ and the conditional probability $\Bbb{P}_{\Theta | X}(\theta | x)$ exist and are well-defined, then by definition, $$\Bbb{P}_X(x) \times \Bbb{P}_{\Theta | X}(\theta | x) = \Bbb{P}_{X \times \Theta}(x, \theta) = \Bbb{P}_{\Theta}(\theta) \times \Bbb{P}_{X | \Theta}(x | \theta),$$ so as long as the marginal probability distribution $\Bbb{P}_X$ and conditional probability distribution $\Bbb{P}_{\Theta | X}$ exist, we can recover Bayes' theorem by dividing through by $\Bbb{P}_X(x)$: $$\Bbb{P}_{\Theta | X}(\theta | x) = \frac{\Bbb{P}_{X \times \Theta}(x, \theta)}{\Bbb{P}_X(x)} = \frac{\Bbb{P}_{\Theta}(\theta) \times \Bbb{P}_{X | \Theta}(x | \theta)}{\Bbb{P}_X(x)}.$$ Therefore, the question reduces to showing that the marginal probability distribution $\Bbb{P}_X$ and conditional probability distribution $\Bbb{P}_{\Theta | X}$ exist. This is what we need the disintegration theorem for:

Statement of the disintegration theorem.

IF:

• $X \times \Theta$ and $X$ are both Radon spaces;
• $\Bbb{P}_{X \times \Theta} := \Bbb{P}_{\Theta} \times \Bbb{P}_{X | \Theta}$ is a Borel probability measure on $X \times \Theta$;
• the projection map $$\begin{align*} \pi: X \times \Theta &\to X \\ (x, \theta) &\mapsto x \end{align*}$$ is Borel measurable;
• $\Bbb{P}_X := \pi_*(\Bbb{P}_{X \times \Theta})$ is the marginal probability distribution on $X$, given by the pushforward of the joint probability distribution $\Bbb{P}_{X \times \Theta}$ under the projection $\pi$;

THEN: there exists a $\Bbb{P}_X$-almost everywhere uniquely determined family of probability measures $\{\Bbb{P}_{\Theta | X}(\cdot | x)\}_{x \in X}$ on $\Theta$, such that:

• for each Borel measurable set $B \subseteq \Theta$, the function $$\begin{align*} f_B: X &\to [0, 1] \\ x &\mapsto \Bbb{P}_{\Theta | X}(B | x)\end{align*}$$ is Borel measurable;
• for $\Bbb{P}_X$-almost all $x \in X$, the measure $\Bbb{P}_{\Theta | X}(\cdot | x)$ is supported on the preimage $\pi^{-1}(x)$, in the sense that $$\Bbb{P}_{\Theta | X}(E | x) = \Bbb{P}_{\Theta | X}(E_x | x)$$ for all Borel measurable $E \subseteq X \times \Theta$, where $$E_x := E \cap \pi^{-1}(x);$$
• and, for every Borel measurable function $\phi: X \times \Theta \to [0, \infty]$, $$\int_{X \times \Theta} \phi(x, \theta) \hspace{0.1 cm} d\Bbb{P}_{X \times \Theta} = \int_X \left( \int_{\pi^{-1}(x)} \phi(x, \theta) \hspace{0.1 cm} d\Bbb{P}_{\Theta | x} \right) d\Bbb{P}_X. \blacksquare$$

Since the conditions of the disintegration theorem hold in a wide variety of cases of interest, it justifies our use of Bayes' theorem in the measure-theoretic context.

Answer 3

This is an old thread, but I just spent a month clarifying exactly this issue for myself (this is the current version of my notes), so I thought I'd share. It's not exactly the question that you asked, but I would like to show that your "usual" expression of Bayes Theorem, is completely well-defined from the measure-theoretic perspective.

For the measure-theoretic description of probability, I found Kolmogorov's original 1933 text (well, the English translation from 1956) to be an excellent reference. It is compact, precise, and everything remains true (although some things can be said more simply by direct appeal to measure theory; measure theory was brand new at the time). Frederic Schuller's lectures on Measure Theory and Lebesgue Integration were invaluable background lessons for me. Billingsley's modern probability text is also a great resource.

To answer the question, it is useful to state the problem a bit more precisely, and to use better notation than, e.g., "$f(x|\theta)$". We are interested in calculating the probability density of a random variable, $X$, with respect to the event that a second random variable, $Y$, takes some value, $f_X\left(x|Y = y\right)$. So, $X$ and $Y$ are real-valued random variables on the probability space $\left(\Omega, \mathfrak{F}, P\right)$, i.e., measurable maps from $\Omega$ to $\mathbb{R}$. By the event "$Y = y$" we mean $\left\{Y = y\right\}$, or, most precisely, $\left\{\omega \in \mathfrak{F} | Y(\omega) = y \right\} $.

For a discrete random variable $Y$, this conditional distribution is perfectly well defined, assuming that $P\left( Y = y\right)>0$: \begin{align} f_X\left(x | Y = y \right) &= \frac{{\rm d}F_X(x|Y=y)}{{\rm d}x} \\ &= \frac{{\rm d}}{{\rm d}x} \left[P^{(X)}\left((-\infty, x) | Y = y \right) \right] \\ &= \frac{{\rm d}}{{\rm d}x} \left[ P\left({\rm preim}_X (-\infty, x) | Y = y \right) \right] \\ &= \frac{{\rm d}}{{\rm d}x} \left[ \frac{P\left({\rm preim}_X(-\infty, x) \, \cap \, \{Y = y\} \right)}{P(Y = y)} \right] \end{align} where $F_X(x)$ is the cumulative distribution function associated to $X$, and $P^{(X)}$ is the probability measure on $\mathbb{R}$ induced by $P$ (i.e., the pushforward of $P$ under the map $X$).

In standard Bayesian inference, however, one is usually interested in the distributions of continuous random variables, which take uncountably-many values, and for which the probability of a single value is usually zero, i.e., $P(Y=y) = 0$. Therefore, the expression above cannot be used. Instead, we use almost the same notation to define it (compare to the third line above): $$ f_X\left(x | Y = y \right) = \frac{{\rm d}}{{\rm d}x} \left\{ \left[P\left({\rm preim}_X (-\infty, x) | Y = y \right)\right] \right\} $$ where the "conditional probability of the event $A$ with respect to the random variable $X$," $\left[P(A|X)\right]$, is assumed to itself be a random variable on $\Omega$ whose expectation takes the value of the associated conditional probability when evaluated over any measurable set $B = {\rm preim}_X(u)$ (with $u$ measureable on $\mathbb{R}$). Kolmogorov shows (Section 1, Chapter 5) that the defining equation for this quantity: $$ P\left(A \,|\,{\rm preim}_X(u)\right) = \mathbb{E}_{{\rm preim}_X(u)} \left\{ \, \left[P\left(A | X \right)\right] \, \right\} $$ implies the existence of a unique random variable, which (assuming some continuity conditions) can be written: $$ \left[P\left(A | X=x \right)\right] = P(A) \,\frac{f_X(x | A)}{f_X(x)}\,. $$ Using this expression to write the above conditional density, one can easily show that: $$ f_X\left(x | Y = y \right) = \frac{f_{XY}\left(x,y\right)}{f_Y(y)}\,. $$ Writing down the analogous expression for $f_Y\left(y | X = x \right)$ and equating the two versions of the joint probability density, $f_{XY}\left(x,y\right)$, leads to: $$ f_X\left(x|Y=y\right) \, = \, \frac{f_Y\left(y | X = x\right) \, f_X(x)}{f_Y(y)}\,, $$ which is the desired version of Bayes' Theorem.