# KL divergence, Fisher information and “distance” in information theory

I understand that the KL divergence between two discrete probability distributions $$p$$ and $$q$$ is defined as

$$D(p||q) = \sum_i p_i\log\frac{p_i}{q_i}$$

This quantity is not symmetric and doesn't satisfy the triangle inequality and is therefore not a metric. However, the Wikipedia article has a section on the connection between the KL divergence and the Fisher information. I am not familiar with the Fisher information and do not fully follow what is said on the Wikipedia article but it seems to imply that if $$p$$ and $$q$$ can be parameterized by some $$\theta$$ and $$\theta$$ is sufficiently small, then the KL divergence does behave like a metric?

Can someone elucidate this idea? In general, can one say that for $$p\approx q$$ (the role of $$\theta$$ is not clear), the KL divergence is a "distance" and if yes, is there an intuitive way to see this?

• Sanjay Gupta more or less says this in his answer, but: the point is that if you do a Taylor expansion of the KL divergence, then the Fisher information turns out to be the second order approximation. See for instance Section 4.1 here: sas.upenn.edu/~vbalasub/public-html/Inference_files/… – pseudocydonia Sep 30 '19 at 5:44
• I answered a similar question and I saw this one in the related section and I think both are much the same if you think these are not tell me so to see if I could help you. – Daniel D. Dec 1 '19 at 20:28

I don't know why you call $$p, q$$ probability distributions. Distributions are functions, but what you have are clearly vectors in the simplex. Those are the images of the probability distribution, i.e. $$\Pr[X = i]= p_i \in [0,1], p = (p_i)_{i = 1}^n$$, where $$X$$ is some underlying random variable, which is said to have categorical distribution. The domain of KL divergence are not functional spaces, they are the simplex.