If $X$ is a random variable and $\rho$ is a parameter, and $L$ is a concave function of $(\rho,X)$, under what conditions is the following statement true? $$\mathbb{E}\max_{\rho} L(\rho,X) =\max_{\rho}\mathbb{E}( L(\rho,X)).$$
I can show that $$\mathbb{E}\max_{\rho} L(\rho,X) \geq \max_{\rho}\mathbb{E}( L(\rho,X)),$$ for all $L$ and $X$ (using a proof very similar to that of the min-max theorem). However, I am not able to derive conditions under which the other inequality holds.
I'm going to use sup rather than max, because in some situations the maximum is not actually attained, and I don't want to bother worrying about that.
It's always true (no concavity necessary) that for any $\rho$ and any $x$, $\sup_\rho L(\rho,x) \ge L(\rho,x)$ so since expected value preserves $\ge$, ${\mathbb E} \sup_\rho L(\rho,X) \ge {\mathbb E} L(\rho, X)$. Taking the supremum of the right side, ${\mathbb E} \sup_\rho L(\rho,X) \ge \sup_\rho {\mathbb E} L(\rho, X)$.
Now, let's turn the question around and ask how could we ensure strict inequality? Suppose there are sets $A$, $B$ with $\mathbb P(X \in A) > 0$ and $\mathbb P(X \in B) > 0$, disjoint sets $C$ and $D$ of $\rho$ values, and $\epsilon > 0$ such that
For any $x$ and $\rho$, let $Q(\rho,x) = \sup_\rho L(\rho,x) - L(\rho,x) \ge 0$. If $\rho \notin C$, $Q(\rho,x) \ge \epsilon$ when $x \in A$. Thus $\mathbb E \sup_\rho L(\rho, X) - L(\rho,X) \ge \mathbb P(X \in A) \epsilon > 0$. Similarly, if $\rho \notin D$, $Q(\rho,x) \ge \epsilon$ when $x \in B$, and $\mathbb E \sup_\rho L(\rho, X) - L(\rho,X) \ge \mathbb P(X \in B) \epsilon > 0$. But since $C$ and $D$ are disjoint, that covers every $\rho$.
Thus the only way to have equality is that such $A,B,C,D,\epsilon$ do not exist. Essentially, what this means is that those $\rho$ that make $L(\rho,x)$ large for one $x$ must also make it large for all other $x$. For example, if $L(\cdot,x)$ has a unique maximum at some $\rho_x$, then $\rho_X$ must be almost surely constant.
