I understand that the KL divergence between two discrete probability distributions $p$ and $q$ is defined as

$$D(p||q) = \sum_i p_i\log\frac{p_i}{q_i}$$

This quantity is not symmetric and doesn't satisfy the triangle inequality and is therefore not a metric. However, the Wikipedia article has a section on the connection between the KL divergence and the Fisher information. I am not familiar with the Fisher information and do not fully follow what is said on the Wikipedia article but it seems to imply that if $p$ and $q$ can be parameterized by some $\theta$ and $\theta$ is sufficiently small, then the KL divergence does behave like a metric?

Can someone elucidate this idea? In general, can one say that for $p\approx q$ (the role of $\theta$ is not clear), the KL divergence is a "distance" and if yes, is there an intuitive way to see this?

  • 1
    $\begingroup$ Sanjay Gupta more or less says this in his answer, but: the point is that if you do a Taylor expansion of the KL divergence, then the Fisher information turns out to be the second order approximation. See for instance Section 4.1 here: sas.upenn.edu/~vbalasub/public-html/Inference_files/… $\endgroup$ – pseudocydonia Sep 30 '19 at 5:44
  • 1
    $\begingroup$ I answered a similar question and I saw this one in the related section and I think both are much the same if you think these are not tell me so to see if I could help you. $\endgroup$ – Daniel D. Dec 1 '19 at 20:28

Learning from Wikipedia is a terrible idea (not peer reviewed and anyone can edit), why don't you read a book on this stuff, for example, "Information Geometry" by Shun-ichi Amari.

I don't know why you call $p, q$ probability distributions. Distributions are functions, but what you have are clearly vectors in the simplex. Those are the images of the probability distribution, i.e. $\Pr[X = i]= p_i \in [0,1], p = (p_i)_{i = 1}^n$, where $X$ is some underlying random variable, which is said to have categorical distribution. The domain of KL divergence are not functional spaces, they are the simplex.

The Hessian of the KL divergence is so-called Fisher's information matrix. That's the connection.

KL divergence is never a metric. Metric has a specific and rigorous definition in mathematics. Some people call it a distance, but they are using it in a colloquial way. It is an example in a class of divergences called Bregman divergence. The only Bregman divergence that is a metric is the Euclidean distance.

  • $\begingroup$ I think Wikipedia is an excellent source of information, and parts of it are probably subject to more peer review than some scientific publications. $\endgroup$ – Roberto Rastapopoulos Oct 2 '19 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.