# Conditional Expectation and Entropy

Let $\mathscr{F,G}$ be $\sigma$-algebras with $\mathscr{F} \subset \mathscr{G}$. Then given a random variable $X$, consider the conditional expectations of $X$ with respect to these two $\sigma-$algebras, i.e. $\mathbb{E}[X|\mathscr{F}]$ and $\mathbb{E}[X|\mathscr{G}]$.

Question: Given that $\mathscr{F} \subset \mathscr{G}$, is there a straightforward relationship between $$H(\mathbb{E}[X|\mathscr{F}]) \quad \text{and} \quad H(\mathbb{E}[X|\mathscr{G}])$$ where $H(Y)$ denotes the entropy of the random variable $Y$?

My suspicion is that one should have $H(\mathbb{E}[X|\mathscr{F}]) \le H(\mathbb{E}[X|\mathscr{G}])$, since a larger $\sigma-$algebra is supposed to allow us to characterize events with more precision and thus acquiring the value of $\mathbb{E}[X|\mathscr{G}]$ should give us more information about $X$ than acquiring the value of $\mathbb{E}[X|\mathscr{F}]$ would.

I think Proposition 5.8 and 5.9 of this document says this, although I am not sure, since I still have trouble understanding conditional probability distributions. Can anyone tell me what it says?

This is a follow-up question to Information in Filtrations

The answer to your question is basically no. Here is a counterexample: suppose we draw a random integer between $1$ and $6$ inclusive with probabilities $\frac{1}{9}, \frac{2}{9}, \frac{1}{6}, \frac{1}{6}, \frac{2}{9}, \frac{1}{9}$ (in that order). The random variable $X$ will take the value of $1$ if the drawn number is even and $0$ otherwise. Now, take $\mathscr{G}$ to be all subsets of $\{1, 2, \ldots , 6\}$, while $\mathscr{F}$ is generated just by $\{1, 2\}$, $\{3, 4\}$, and $\{5, 6\}$. Then $\mathbb{E}[X|\mathscr{F}]$ is uniformly distributed among $\frac{1}{3}, \frac{1}{2}, \frac{2}{3}$, while $X = \mathbb{E}[X|\mathscr{G}]$ is uniform among $0$ and $1$. The former has higher entropy. You can also construct examples where the inequality goes the other way.

The link you gave says something different. The confusion comes from interpreting expressions of the form $H(\cdot)$. Normally, one starts out by first defining entropies of probability distributions. Then, when $X$ is a random variable, writing something like $H(X)$ is an abuse of notation which (usually) means the entropy of the distribution of $X$.

In the linked document, $X$ does not refer to a random variable but rather the set underlying the probability space (in the above example, it would be $\{1, 2, 3, 4, 5, 6\}$). So the correct statement would be something like $H(\mathscr{F}) \le H(\mathscr{G})$ (where implicitly there is also a probability measure $\mu$ on both $\sigma$-algebras). Again using the above example, letting $p_i$ be the probability the randomly drawn number is $i$, the correct inequality would read

$$(p_1 + p_2) \log \frac{1}{p_1 + p_2} + (p_3 + p_4) \log \frac{1}{p_3 + p_4} + (p_5 + p_6) \log \frac{1}{p_5 + p_6} \le \sum_{i = 1}^6 p_i \log \frac{1}{p_i}.$$

If it's still confusing, I would recommend studying entropies of discrete probability distributions first and then basically thinking about the general case by analogy.

• This is a great answer -- I never would have thought of such a counterexample -- it demonstrates a lot of ingenuity and effort and deserves even more upvotes. Just to confirm, then $H(\mathbb{E}[X|\mathscr{F}]) = \log 27$ while $H(\mathbb{E}[X|\mathscr{G}])= \log 4$? (since $3^3=27$ and $2^2=4$) – Chill2Macht Aug 19 '16 at 20:02
• Wait, so is the statement $H(\mathscr{F}) \le H(\mathscr{G})$ true? (i.e. is $H(\mathscr{F}) \le H(\mathscr{G})$ what Proposition 5.8 says)? That would actually be even more interesting to me -- I didn't realize/know/believe that there was a way to measure directly the entropy of a $\sigma-$algebra, I thought it was only possible to measure the entropy of probability distributions (and random variables by associating them with their pushforward distributions). I was just using the conditional expectation random variables as a proxy to describe the $\sigma-$algebras since I thought I needed to. – Chill2Macht Aug 19 '16 at 20:02
• The notation $H(\mathscr{F})$ hides the fact that there should be an underlying measure $\mu$ as well. It just so happens that regardless of what $\mu$ is, it will always be the case that $H(\mathscr{F}) \le H(\mathscr{G})$. However, the actual values of the entropy will differ depending on $\mu$. Perhaps a more "correct" notation should be something like $H(\mathscr{F}, \mu)$. – arghbleargh Aug 19 '16 at 20:08
• Regarding the examples above, we actually have $H(\mathbb{E}[X|\mathscr{F}]) = \log 3$ and $H(\mathbb{E}[X|\mathscr{G}]) = \log 2$. Again, thinking about the discrete case is helpful here. For a probability distribution that takes $n$ possible values with probabilities $p_1, \ldots , p_n$, the entropy is $\sum p_i \log(1/p_i)$. – arghbleargh Aug 19 '16 at 20:12
• I'm not sure that's exactly how I would say it, but the important thing is just to understand what is true and what isn't. I added a little bit to the answer which should hopefully clear up the remaining confusion if there is any. – arghbleargh Aug 19 '16 at 20:30