What is the intuition behind conditional expectation in a measure-theoretic treatment of probability?

Question

What is the intuition behind conditional expectation in a measure-theoretic sense, as opposed to a non-measure-theoretic treatment?

You may assume I know:

• what a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ refers to
• probability without measure theory really well (i.e., discrete and continuous random variables)
• what the measure-theoretic definition of a random variable is
• that a Lebesgue integral has something to do with linear combinations of step functions, and intuitively, it involves partitioning the $y$-axis (as opposed to the Riemann integral, which partitions the $x$-axis).

I haven't had time to learn measure-theoretic probability lately due to graduate school creeping in as well as other commitments, and conditional expectation is often covered as one of the last topics in every measure-theoretic probability text I've seen.

I have seen notations such as $\mathbb{E}[X \mid \mathcal{F}]$, where I assume $\mathcal{F}$ is some sort of $\sigma$-algebra - but of course, this looks very different from, say, $\mathbb{E}[X \mid Y]$ from what I saw in my non-measure-theoretic treatment of probability, where $X$ and $Y$ are random variables.

I was also surprised to see that one book I have (Essentials of Probability Theory for Statisticians by Proschan and Shaw (2016)), if I recall correctly, explicity states that conditional expectation is defined as a conditional expectation, rather than the conditional expectation, which implies to me that there's more than one possible conditional expectation when given two pairs of random variables. (Unfortunately, I don't have the book on me right now, but I can update this post later).

The Wikipedia article is quite dense, and I see words such as "Radon-Nikodym" which I haven't learned yet, but I would at least like to get an idea of what the intuition of conditional expectation is in a measure-theoretic sense.

Answer 1

So, intuitively you can think of $E(X|\mathcal{F})$ as the "best guess" of the value of $X$ given information about the events in $\mathcal{F}$. However, more formally speaking, $E(X|\mathcal{F})$ is a random variable that satisfies the following:

1. $E(X|\mathcal{F})$ is an $\mathcal{F}-$measurable function
2. $\int_A E(X|\mathcal{F}) dP = \int_A X dP$ for every $A \in \mathcal{F}$

Now, the theorem that guarantees the existence of $E(X|\mathcal{F})$ is precisely the Radon-Nikodym Theorem (that's why these words appear in this context). And, in order to fully understand this definition of conditional expectation it's really recommended to cement some Measure Theory knowledge. Having said that, with the intuition above we can already "guess" the conditional expectation for some cases:

i) If $X$ itself is a $\mathcal{F}-$measurable function, then $E(X|\mathcal{F}) = X$. That is, by having all the information about $X$, then our best guess is $X$ itself.

ii) If $X$ is independent of $\mathcal{F}$, i.e. for all $A \in \mathcal{F}$ and $B \in \mathcal{B}_\mathbb{R}$ we have $P(\{X \in B\} \cap A) = P(X \in B)P(A)$, then $E(X|\mathcal{F}) = E(X)$. In other words, if we don't have any information about $X$, then our best guess of the value of $X$ is $E(X)$.

Finally, there are other useful intuitions of this conditional expectation. For instance, if you are familiar with Hilbert spaces and $X \in L^2(\mathcal{G})$ (consider the probability space $(\Omega, \mathcal{G},P)$), then $E(X|\mathcal{F})$ with $\mathcal{F} \subset \mathcal{G}$ is a projection in $L^2(\mathcal{F})$. This a commonly used geometric intuition.