I know about the Kullback Leibler divergence and that it can be used to measure the difference between two probability distributions.

But it is not very symmetric. For example watching P from Q, if $q$ ever becomes zero:

$$p(x)\log\left(\frac{p(x)}{q(x)}\right), q(x)=0, p(x)\neq 0$$

Will of course be infinite. This is reasonable in the sense that if an event is impossible in $q$ but not in $p$ then it is impossible to "repair" in some sense. But this never happens from the other "view":

$$q(x)\log\left(\frac{q(x)}{p(x)}\right), q(x)=0,p(x)\neq 0$$

We can convince ourselves (probably by investigating some limit) this should not be considered infinite.

So how can we build a less unsymmetric distance measure?


There are a lot of these, you can see some examples here.

The ones I feel like come up the most often are:

Though, there are plenty of them.

There's a lot of ways one could approach the idea of a "distance on probability distributions". For instance, the Total Variation Distance defined on $\mathcal{P}\times \mathcal{P}$, the product space of probability distributions defined on the same measurable space $(\Omega,\mathcal{B})$, is denoted as: $TV(p_1,p_2) = \sup_{B \in \mathcal{B}}|p_1[B] - p_2[B]|$ the largest gap in probabilities assigned to sets in the shared $\sigma-$algebra.

That is kind of an abstracted notion of distance, however, if $p_1$ and $p_2$ have densities $f_1$ and $f_2$, then: \begin{equation} TV(p_1,p_2) = ||f_1 - f_2||_{L_1} = \int_{x\in\mathcal{X}}|f_1(x) - f_2(x)|dx \end{equation}

Which is usually more convenient to work with.

Similarly, the Wasserstein distance turns out to be useful for different settings as well. You can read more about it in that link, but generally Wasserstein distance on continuous spaces is kind of abstracted and unwieldy. However, computing Wasserstein distance on discrete spaces reduces to solving an integer/linear program. A lot of research (especially in computer imaging) goes into framing these problems into programs or other optimization problems.

See here, here. It's also used to show consistency in convergence of probability measures in statistical applications, for example here.

I guess a last point would be that just because some Divergences are not symmetric does not mean that they can't sometimes be stronger than symmetric distance metrics. For example if $KL(P_n||Q)\overset{n\rightarrow\infty}{\rightarrow}0$ then $P_n\overset{T.V.}{\rightarrow}Q$

  • $\begingroup$ Nice one! I know Wasserstein as Earth Mover Distance in information theory and Total Variation was very popular all over image analysis & computer vision 5-10 years ago. But I'm pretty sure I haven't seen Hellinger yet. $\endgroup$ – mathreadler Dec 31 '17 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.