# Multiplying two conditional expectations

After some calculations I ended up with (there each $V_k : \mathbb{R} \to \mathbb{R}$)

$$\sum_d \widetilde{P}_D(d) \left( \sum_{f' \in \mathcal{F}} P_{F|\;D}(f'|\;d) V_j(f',d) \right) \left( \sum_{f \in \mathcal{F}} P_{F|\;D}(f|\;d) V_i(f,d) \right)$$

Since

$$E_{P_{F|\;D}}[V_i|\;d] = \sum_{f \in \mathcal{F} } P_{F|\;D}(f|\;d) V_i(f,d)$$

I get

$$\sum_d \widetilde{P}_D(d) E_{P_{F|\;D}}[V_j|\;d] E_{P_{F|\;D}}[V_i|\;d]$$

Which I believe equals

$$E_{\widetilde{P}_d P_{F|\;D}}[V_i V_j]$$

and just rewriting the first expression it is equal to:

$$\sum_d \widetilde{P}_D(d) \sum_{f \in \mathcal{F}} \sum_{f' \in \mathcal{F}} P_{F|\;D}(f|\;d) P_{F|\;D}(f'|\;d) V_j(f',d) V_i(f,d)$$

Which seems strange to me, that the expectation value of $V_i V_j$ would take into consideration the value when they are fed the same $d$ but different $f$.

I thought it would be written as

$$\sum_d \widetilde{P}_D(d) \sum_{f \in \mathcal{F}} P_{F|\;D}(f|\;d) V_j(f,d) V_i(f,d)$$

Is it correct that the expression at the top equals $E_{\widetilde{P}_d P_{F|\;D}}[V_i V_j]$?

-

The trouble lies in the step where you write Which I believe equals, and comes from the fact that for some random variables $X$, $Y$ and $Z$, in general, $$\mathrm E(XY)\ne \mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z)).$$ For a counterexample, assume that $X$ and $Y$ are i.i.d. and square integrable random variables and that $Z=\frac12(X+Y)$.
The computations below hold in this general setting but one can assume for simplicity that the random variables $X$ and $Y$ are two independent centered $\pm1$ Bernoulli random variables or, equivalently, that $(X,Y)$ is uniform on the set $\{-1,1\}^2$. Then $\mathrm E(X)=\mathrm E(Y)=\mathrm E(Z)=0$ and $\mathrm E(Z^2)=\frac12$.
Then, on the one hand, $\mathrm E(X\mid Z)=\mathrm E(Y\mid Z)=\frac12\mathrm E(X+Y\mid Z)=Z$ by symmetry, hence $$\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))=\mathrm E(Z^2).$$ On the other hand, $\mathrm E(XY)=\mathrm E(X)\mathrm E(Y)$ by independence and $\mathrm E(X)=\mathrm E(Y)=\mathrm E(Z)$ by symmetry again, hence $\mathrm E(XY)=\mathrm E(Z)^2$. One sees that for every nondegenerate distribution of $X$ and $Y$, $$\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))-\mathrm E(XY)=\text{Var}(Z)\ne0.$$ Edit For a case where $\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))-\mathrm E(XY)\lt0$, consider $Z=\frac12(X-Y)$.