Does a Lipshitz loss induces a Lipshitz property on expected loss with respect to some distance metric over distributions? Consider some loss function: $L:\mathbb{R} \rightarrow \mathbb{R}$, that is $K$-Lipshitz:
$$\left \lvert L(x) - L(y)\right \rvert \leq K \left \lvert x- y\right \rvert, \;\; \forall x,y \in \mathbb{R}$$
Now let $\bar{L}(X) = \mathbb{E}[L(X)]$ for some random variable $X$, (it maps a random variable to a real number, slightly overloading notation here). Does $\bar{L}$ satisfy some analogous Lipshitz condition with respect to some distance measure in the space of distributions? For example, something like:
$$ \left \lvert \bar{L}(X) - \bar{L}(Y)\right \rvert \leq K' \cdot d( \mathbb{P}_X, \mathbb{P}_Y).$$
I have been trying to make something work in terms of TV distance, or KL-divergence, but have had no success. It seems like such a result would have come up somewhere before, but I have not been able to find anything thus far. Any suggestions would be appreciated.
 A: I'll prove that $\overline{L}$ is $K-$Lipschitz with respect to the first Wasserstein Metric, i.e $$ W_1(\mu,\nu) = \inf\{\mathbb E|X-Y| : X \sim \mu, Y \sim \nu\}.$$
Indeed, note that $\overline{L}(X)$ depends only on the distribution of $X$, that is if $X\sim X'$ then $\overline{L}(X) = \mathbb E[L(X)] = \mathbb E[L(X')] = \overline{L}(X')$ because $\mathbb E[\cdot]$ itself depends only on the distribution and $L(X) \sim L(X')$. Hence, for any pairs $X' \sim X, Y' \sim Y$, we get $$ |\overline{L}(X) - \overline{L}(Y)| = |\overline{L}(X') - \overline{L}(Y')| \le \mathbb E|L(X')-L(Y')| \le K \mathbb E|X'-Y'|.$$ Taking infimum on the RHS over all such pairs $X' \sim X, Y' \sim Y$ we arrive at $$ |\overline{L}(X) - \overline{L}(Y)| \le K \inf\{\mathbb E|X'-Y'| : X' \sim X, Y' \sim Y\} = K W_1(\mathbb P_X,\mathbb P_Y).$$
As a side note, by Holder's inequality, we have $W_1(\mu,\nu) \le W_p(\mu,\nu)$ for any $p \ge 1$, where $W_p$ is the $p'$th Wasserstein Metric, i.e $$ W_p(\mu,\nu) = \inf\{ \left( \mathbb E|X-Y|^p \right)^{\frac{1}{p}} : X \sim \mu, Y \sim \nu \},$$ so $\overline{L}$ is $K-$Lipschitz with respect to $W_p$, too.
