Distance between two distributions I would like your opinion with a two metrics.
It is known that the Kantorovich–Rubinstein metric is a distance function defined between probability distributions on a given metric space M.
On the other hand, considering two continuous probability distributions $q(x)$ and $p(x)$,  The Kullback–Leiber divergence is defined as the measure of the information lost when $q(x)$ is used to approximate $p(x)$.
\begin{equation*}
    \mathrm{KL} (p(x)|| q(x)) = \int_{\mathbb{R}} p(x) \ln \frac{p(x)}{q(x)} dx
\end{equation*}
Thank you very much for your help/hints/suggestions.
 A: General comments: Convergence of the Kantorovich-Rubinstein metric (Wasserstein distance) is equivalent to weak convergence, under some regularity conditions, which makes it a strong candidate.
The same is not true for the KL-divergence.
Moreover, the KL-divergence is in general not a distance, since it is not symmetric (although it can be, see below).
The answer depends on your specific problem.
Here are some examples.
A case where it doesn't matter:
If you compare normal distributions with different mean $\mathcal{N}(\theta_1,1)$, $\mathcal{N}(\theta_2,1)$, both the Kantorovich-Rubinstein metric (2-Wasserstein distance) and the KL-divergence just give you the same answer, namely $|\theta_1-\theta_2|$.
So, in this case, it doesn't matter.
A case where the Kantorovic-Rubinstein metric is superior.
If you compare distributions whose support differs.
E.g. if you compare uniform distributions supported on $(\theta,\theta+1)$ for $\theta\in\mathbb{R}$, then convergence in the Kantorovich-Rubinstein metric is still equivalent to $\theta_n\rightarrow\theta$ (since convergence of this metric is equivalent to weak-convergence, under some regularity conditions).
On the other hand, the KL-divergence is infinite, unless $\theta=\theta'$
A case where you'd want to use the KL-divergence. In some cases, the two distributions you are comparing have different roles.
For example, in hypothesis testing.
Here, the KL-divergence can be used to calculate the expected value of the likelihood ratio between the null and the alternative hypothesis, under the alternative hypothesis see here. In this case, the asymmetry of the KL-divergence is not a bug, but a feature.
