34
$\begingroup$

If we have a probability space $(\Omega,\mathcal{F},P)$ and $\Omega$ is partitioned into pairwise disjoint subsets $A_{i}$, with $i\in\mathbb{N}$, then the law of total probability says that $P(B)=\sum_{i=1}^{n}P(B|A_{i})P(A_i{})$. This law can be proved using the following two facts: \begin{align*} P(B|A_{i})&=\frac{P(B\cap A_{i})}{P(A_{i})}\\ P\left(\bigcup_{i\in \mathbb{N}} S_{i}\right)&=\sum_{i\in\mathbb{N}}P(S_{i}) \end{align*} Where the $S_{i}$'s are a pairwise disjoint and a $\textit{countable}$ family of events in $\mathcal{F}$.

However, if we want to apply the law of total probability on a continuous random variable $X$ with density $f$, we have (like here): $$P(A)=\int_{-\infty}^{\infty}P(A|X=x)f(x)dx$$ which is the law of total probabillity but with the summation replaced with an integral, and $P(A_{i})$ replaced with $f(x)dx$. The problem is that we are conditioning on an $\textit{uncountable}$ family. Is there any proof of this statement (if true)?

$\endgroup$
2
  • 7
    $\begingroup$ Fortunately it is true; otherwise we would also have trouble with conditional expectation for continuous variables. I was not able to find the proof in my measure theory book this morning, skimming through it, but going the long way, you may be convinced by the proof of the law of total expectation together with the relation that $P(A\mid B) = E[1_A\mid B]$. $\endgroup$
    – Therkel
    Nov 29, 2016 at 7:54
  • $\begingroup$ @Therkel this relation won't help: conditional expectation wrt a sigma-field is a random variable. $\endgroup$
    – user159517
    Dec 3, 2020 at 9:19

3 Answers 3

15
$\begingroup$

Think of it like this: Suppose you have a continuous random variable $X$ with pdf $f(x)$. Then $P(A)=E(1_{A})=E[E(1_{A}|X)]=\int E(1_{A}|X=x)f(x)dx=\int P(A|X=x)f(x)dx$.

$\endgroup$
2
  • 2
    $\begingroup$ The event $\{X = x \}$ has 0 probability for a continuous random variable $X$, so your $P(A | X = x) = P(A, X=x)/P(X = x)$ is not well defined. You can condition on any event $B \in \sigma(X)$ i.e. on any event from the sigma algebra generated by $X$. $\endgroup$
    – baibo
    Feb 11, 2020 at 21:33
  • 4
    $\begingroup$ The problem with this answer is that the subtlety here is what $\mathbb{E}[1_A | X=x]$ actually means. The connection between $\mathbb{E}[1_A |X=x]$ and $\mathbb{E}[1_A | X]$ is nontrivial with any definition I am familiar with, while proving the law of total probability is typically quite easy (see my answer for more details). $\endgroup$
    – user159517
    May 12, 2021 at 15:10
15
$\begingroup$

Excellent question. The issue here is that you first have to define what $\mathbb{P}(A|X=x)$ means, as you're conditioning on the event $[X=x]$, which has probability zero if $X$ is a continuous random variable. Can we still give $\mathbb{P}(A|X=x)$ a meaning? In the words of Kolmogorov,

"The concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible."

The problem with conditioning on a single event of probability zero is that it can lead to paradoxes, such as the Borel-Kolmogorov paradox. However, if we don't just have an isolated hypothesis such as $[X=x]$, but a whole partition of hypotheses $\{[X=x] ~|~ x \in \mathbb{R}\}$ with respect to which our notion of conditional probability is supposed to make sense, we can give a meaning to $\mathbb{P}(A|X=x)$ for almost every $x$. Let's look at an important special case.


Continuous random variables in Euclidean space

In many instances where we might want to apply the law of total probability for continuous random variables, we are actually interested in events of the form $A = [(X,Y) \in B]$ where $B$ is a Borel set and $X,Y$ are random variables taking values in $\mathbb{R}^d$ which are absolutely continuous with respect to Lebesgue measure. For simplicity, I will assume here that $X,Y$ take values in $\mathbb{R}$, although the multivariate case is completely analogous. Choose a representative of $f_{X,Y}$, the density of $(X,Y)$, and a representative of $f_X$, the density of $X$, then the conditional density of $Y$ given $X$ is defined as $$ f_{Y|X}(x,y) = \frac{f_{X,Y}(x,y)}{f_{X}(x)}$$ at all points $(x,y)$ where $f(x) > 0$. We may then define for $A = [(X,Y) \in B]$ and $B_x := \{ y \in \mathbb{R} : (x,y) \in B\}$

$$\mathbb{P}(A | X = x) := \int_{B_x}^{} f_{Y|X}(x,y)~\mathrm{d}y, $$ at least at all points $x$ where $f(x) > 0$. Note that this definition depends on the choice of representatives we made for the densities $f_{X,Y}$ and $f_{X}$, and we should keep this in mind when trying to interpret $P(A|X=x)$ pointwise. Whichever choice we made, the law of total probability holds as can be seen as follows:

\begin{align*} \mathbb{P}(A) &= \mathbb{E}[1_{B}(X,Y)] = \int_{B} f_{X,Y}(x,y)~\mathrm{d}y~\mathrm{d}x = \int_{-\infty}^{\infty}\int_{B_x} f_{X,Y}(x,y)~\mathrm{d}y~\mathrm{d}x \\ &= \int_{-\infty}^{\infty}f_{X}(x)\int_{B_x} f_{Y|X}(x,y)~\mathrm{d}y~\mathrm{d}x = \int_{-\infty}^{\infty}\mathbb{P}(A|X=x)~ f_X(x)~\mathrm{d}x. \end{align*}

One can convince themselves that this construction gives us the properties we would expect if, for example, $X$ and $Y$ are independent, which should give us some confidence that this notion of conditional probability makes sense.


Disintegrations

The more general name for the concept we dealt with in the previous paragraph is disintegration. In complete generality, disintegrations need not exist, however if the probability space $\Omega$ is a Radon space equipped with its Borel $\sigma$-field, they do. It might seem off-putting that the topology of the probability space now comes into play, but I believe for most purposes it will not be a severe restriction to assume that the probability space is a (possibly infinite) product of the space $([0,1],\mathcal{B},\lambda)$, that is, $[0,1]$ equipped with the Euclidean topology, Borel $\sigma$-field and Lebesgue measure. A one-dimensional variable $X$ can then be understood as $X(\omega) = F^{-1}(\omega)$, where $F^{-1}$ is the generalized inverse of the cumulative distribution function of $X$. The disintegration theorem then gives us the existence of a family of measures $(\mu_x)_{x \in \mathbb{R}}$, where $\mu_x$ is supported on the event $[X=x]$, and the family $(\mu_x)_{x\in \mathbb{R}}$ is unique up to $\text{law}(X)$-almost everywhere equivalence. Writing $\mu_x$ as $\mathbb{P}(\cdot|X=x)$, in particular, for any Borel set $A \in \mathcal{B}$ we then again have

$$\mathbb{P}(A) = \int_{-\infty}^{\infty} \mathbb{P}(A|X=x)~f_X(x)~\mathrm{d}x.$$


Reference for Kolmogorov quote:

Kolmogoroff, A., Grundbegriffe der Wahrscheinlichkeitsrechnung., Ergebnisse der Mathematik und ihrer Grenzgebiete 2, Nr. 3. Berlin: Julius Springer. IV + 62 S. (1933). ZBL59.1152.03.>

$\endgroup$
2
$\begingroup$

May I propose the following approach?

Factorization Lemma: Let $X$ and $Y$ be real-valued random variables on the same underlying space. If $Y$ is measurable with respect to $\sigma(X)$, then there is a Borel function $g : \mathbb{R} \to \mathbb{R}$ such that $Y = g \circ X$.

This Lemma is assigned as Exercise 1.3.8 in Durret's PTE 5e, and the proof is basically to first treat the case when $Y$ is an indicator function (observe that $\chi_A$ is $\sigma(X)$-measurable iff $A = X^{-1}(\theta)$ for some $\theta$ Borel, but $\chi_{X^{-1}(\theta)} = \chi_\theta \circ X$). In fact, the lemma holds more generally, but the stated case is enough, for now.

Definition: Given real-valued random $Y$ and $X$ on the same underlying space, let us define $E(Y \mid X=x) := g(x)$ where $g$ is any Borel function $\mathbb{R} \to \mathbb{R}$ such that $E(Y \mid X) = g \circ X$. For an event $A$, we define $P(A \mid X=x) := E(\chi_A \mid X=x)$. [edit: definition learned from these lecture notes]

The Factorization Lemma addresses existence. Unfortunately, I'm not aware of any reason why this function $g$ would be unique. The equation $Y = g \circ X$ determines $g$ only on the image of $X$ (i.e., on the set $X(\Omega)$ where $\Omega$ is the previously un-named underlying space). Notably, attempting to impose a canonical choice that $g=0$ outside of $X(\Omega)$ isn't necessarily an option, since $X(\Omega)$ might not be a Borel subset of $\mathbb{R}$, in which case this imposition could fail to result in something measurable.

[edit: In fact, there are often some choices $g$ which are much better than others. See user159517's answer, as well as the comments to this answer. Thus, more results would be able to be proven with a slightly more restrictive definition than the above, as the above allows both "good" and "bad" choices of $g$. However, the above (relatively simple) definition is sufficient for this problem, and the strategy of the solution given below works even if one refines the definition by allowing only "good" $g$ (as should be done).]

Solution: If one is willing to accept a definition of $P(A \mid X=x)$ that's only well-defined modulo the choice of $g$, this definition usually makes it quite easy to prove results of the form "for any choice of $P(A \mid X=x)$, a property holds." For instance, take $Y := E(\chi_A \mid X)$ as in John_Wick's answer, for which $E(Y) = P(A)$. For any Borel $g : \mathbb{R} \to \mathbb{R}$ such that $Y = g \circ X$, we obtain $$ P(A) = E(Y) = E(g \circ X) = \int_{\mathbb{R}} g(x)f(x) \mathrm{d} x $$ by the law of the unconscious statistician (which is given as Theorem 1.6.9 in Durrett PTE 5e). Finally, our definition means that, literally, $g(x) = P(A \mid X=x)$. [edit: in the process of writing this, I forgot why I originally googled the question! This solution has the virtue of working even if $f$ isn't given by a density. Just replace "$f(x) \mathrm{d}x$" with "$\mathrm{d}\alpha(x)$" where $\alpha$ is the distribution of $X$]

$\endgroup$
4
  • 1
    $\begingroup$ This is a natural idea, one problem with it is that this way, you run into problems when considering more than one event at a time, because with this approach $A \mapsto \mathbb{P}(A|X=x)$ doesn't need to be probability measure for every $x$. Take, e.g, $X$ to be standard normal, and $A = \{X < 0\}$, then $\mathbb{P}(A|X) = 1_{(-\infty,0]}(X)$ so we may take $g(x) := 1_{(-\infty,0]}(x)$ and similarly $\mathbb{P}(A^c|X=x) = 1_{[0,\infty)}(x)$. But then $\mathbb{P}(A|X=0) + \mathbb{P}(A^c|X=0) = 2$. That's why I consider disintegrations / regular conditional distributions in my post. $\endgroup$
    – user159517
    Jun 7, 2022 at 7:05
  • $\begingroup$ Oh, that's a great point! Please, let me see if I understand correctly. Of course, the $g$ that I chose works only for one particular $A$. Say, $g_A$ is chosen to work for the set $A$. As you pointed out (and, indeed, I hadn't considered), choosing $A \mapsto g_A$ with the axiom of choice leads to inadequate results. The content of the disintegration theorem is, essentially, that we can choose $g_A$ in a manner "consistent" for all $A$, namely $g_A(x) = \mu_x(A)$, in which case $A \mapsto g_A(x)$ is a measure for each $x$. This improves on the AoC approach. Have I understood correctly? $\endgroup$ Jun 9, 2022 at 18:32
  • $\begingroup$ yes, that is correct. $\endgroup$
    – user159517
    Jun 10, 2022 at 7:20
  • 1
    $\begingroup$ Great! So, in particular, it sounds like my strategy of proof is fine, since it applies to any choice of $g(x) = P( A \mid X=x)$, not only the "correct" choice. But you raise an excellent point: we should be aware that most choices of $g$ are unsatisfactory. In that sense, it seems my definition is incomplete, but the proof is correct. In fact, your comment has single-handedly helped me, finally, understand where the idea of an RCD "comes from." You've really made the concept "click" for me. I appreciate it! $\endgroup$ Jun 10, 2022 at 21:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .