1
$\begingroup$

Let $X$, $Y$, and $Z$ be three random variables on a common measurable space. I am interested in the total variation distance between the two joint distributions $(X,Z)$ and $(Y,Z)$, i.e. \begin{equation} \label{eq:tv-distance} d_{TV} \left( (X, Z) , (Y, Z) \right) \quad (\dagger) \end{equation}

I am looking for a "law of total probability" for the total variation distance which may be used to decompose the total variation distance above into the total variation distance between the conditional distributions $X \mid Z = z$ and $Y \mid Z = z$. In particular, define the function $h: \mathbb{R} \to \mathbb{R}$ as $$ h(z) = d_{TV} \left( X \mid Z = z , Y \mid Z = z \right) \enspace. $$

Is there a "law of total probability" way to decompose the total variation $(\dagger)$ to the conditional total variation distance function $h(z)$ and the distribution of $Z$? Say, for example, $$ d_{TV} \left( (X, Z) , (Y, Z) \right) = \mathbb{E} \left[ h(Z) \right] $$

$\endgroup$

1 Answer 1

1
$\begingroup$

Yes, there is. For simplicity, I am focusing on the discrete case, but this will/should generalize to continuous distributions (given suitable measurability assumptions, etc.).

We have $$\begin{align*} \mathrm{d}_{\rm TV}((X,Z),(Y,Z)) &= \frac{1}{2}\sum_{v,z} \left|\Pr[ X=v, Z=z ]-\Pr[ Y=v, Z=z ]\right| \\ &= \frac{1}{2}\sum_{v,z} \left|\Pr[ X=v \mid Z=z ]-\Pr[ Y=v \mid Z=z ]\right|\cdot \Pr[Z=z] \\ &=\sum_z\Pr[Z=z]\sum_{v} \frac{1}{2}\left|\Pr[ X=v \mid Z=z ]-\Pr[ Y=v \mid Z=z ]\right| \\ &=\sum_z\Pr[Z=z]\mathrm{d}_{\rm TV}(X\mid Z=z,Y\mid Z=z) \\ &= \mathbb{E}_Z[h(Z)] \end{align*}$$ where $h\colon \mathbb{R}\to\mathbb{R}$ is defined as in your post, $h(z) = \mathrm{d}_{\rm TV}(X\mid Z=z,Y\mid Z=z)$.

$\endgroup$
4
  • $\begingroup$ Thanks for the great answer, Clement! I see how to generalize this to the case where X, Y, and Z have density functions. Can you provide any insights / comments on the general case where the random variables are neither discrete nor have a density functions? No need for a full proof or anything -- I'm trying to slog through the details but my recollection of measure theoretic probability is somewhat lacking. $\endgroup$ Jan 5, 2023 at 1:39
  • $\begingroup$ @ChrisHarshaw Dang. I have to admit the exact steps for the fully general statement are a bit hazy in my mind too, I'd need to spend a bit of time in my measure theory textbook to formalize it. $\endgroup$
    – Clement C.
    Jan 5, 2023 at 23:28
  • $\begingroup$ that's fine - thanks for the answer in this case! $\endgroup$ Jan 7, 2023 at 0:54
  • $\begingroup$ You're welcome! $\endgroup$
    – Clement C.
    Jan 7, 2023 at 3:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .