Multiplying two conditional expectations After some calculations I ended up with (there each $V_k : \mathbb{R} \to \mathbb{R}$)
$$
\sum_d \widetilde{P}_D(d) \left( \sum_{f' \in \mathcal{F}} P_{F|\;D}(f'|\;d) V_j(f',d) \right) \left( \sum_{f \in \mathcal{F}} P_{F|\;D}(f|\;d) V_i(f,d) \right)
$$
Since
$$
 E_{P_{F|\;D}}[V_i|\;d] = \sum_{f \in \mathcal{F} } P_{F|\;D}(f|\;d) V_i(f,d) 
$$
I get
$$
\sum_d \widetilde{P}_D(d) E_{P_{F|\;D}}[V_j|\;d] E_{P_{F|\;D}}[V_i|\;d]
$$
Which I believe equals
$$
E_{\widetilde{P}_d P_{F|\;D}}[V_i V_j]
$$
and just rewriting the first expression it is equal to:
$$
\sum_d \widetilde{P}_D(d) \sum_{f \in \mathcal{F}} \sum_{f' \in \mathcal{F}}  P_{F|\;D}(f|\;d) P_{F|\;D}(f'|\;d) V_j(f',d) V_i(f,d) 
$$
Which seems strange to me, that the expectation value of $V_i V_j$ would take into consideration the value when they are fed the same $d$ but different $f$.
I thought it would be written as
$$
\sum_d \widetilde{P}_D(d) \sum_{f \in \mathcal{F}} P_{F|\;D}(f|\;d) V_j(f,d) V_i(f,d) 
$$
Is it correct that the expression at the top equals $E_{\widetilde{P}_d P_{F|\;D}}[V_i V_j]$? 
 A: The trouble lies in the step where you write Which I believe equals, and comes from the fact that for some random variables $X$, $Y$ and $Z$, in general,
$$
\mathrm E(XY)\ne \mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z)).
$$
For a counterexample, assume that  $X$ and $Y$ are i.i.d. and square integrable random variables and that $Z=\frac12(X+Y)$. 
The computations below hold in this general setting but one can assume for simplicity that the random variables $X$ and $Y$ are two independent centered $\pm1$ Bernoulli random variables or, equivalently, that $(X,Y)$ is uniform on the set $\{-1,1\}^2$. Then $\mathrm E(X)=\mathrm E(Y)=\mathrm E(Z)=0$ and $\mathrm E(Z^2)=\frac12$.
Then, on the one hand, $\mathrm E(X\mid Z)=\mathrm E(Y\mid Z)=\frac12\mathrm E(X+Y\mid Z)=Z$ by symmetry, hence 
$$
\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))=\mathrm E(Z^2).
$$
On the other hand, $\mathrm E(XY)=\mathrm E(X)\mathrm E(Y)$ by independence and $\mathrm E(X)=\mathrm E(Y)=\mathrm E(Z)$ by symmetry again, hence $\mathrm E(XY)=\mathrm E(Z)^2$. One sees that for every nondegenerate distribution of $X$ and $Y$, 
$$
\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))-\mathrm E(XY)=\text{Var}(Z)\ne0.
$$
Edit For a case where $\mathrm E(\mathrm E(X\mid Z)\mathrm E(Y\mid Z))-\mathrm E(XY)\lt0$, consider $Z=\frac12(X-Y)$.
