I am reading 'Deep Learning' by Goodfellow et al and in its review of Information Theory states that the Kullback-Leibler divergence can measure how different two distributions are, without any further qualification.

However, to me it seems that KL divergence measures the difference of two distributions from the point of view of the Information Value and not in general.

If I understand well, two symmetric distributions around their mean should have the same Information Value -- the amount of 'certainty' and 'surprise' from messages relating to Events generated from these two distributions should be the same given their symmetry. In the same time these two symmetric distributions would be very different, one representing a high mass of probability for low values and the other for high values.

Is my understanding correct?

And finally how Kullback_leibler divergence compares to the Kolmogorov-Smirnov Test?

Your advice will be appreciated.

  • $\begingroup$ stats.stackexchange.com/q/9311/129122 $\endgroup$ – Lorenzo Najt May 25 '18 at 20:13
  • $\begingroup$ There's a bunch of differences - for instance, the K-S statistic is only defined for distributions on $\mathbb{R},$ whilst the K-L is defined for arbitrary alphabets. Distributional discrepancy (which is morally K-S but without to $\mathbb{R}$) can be upper bounded by total variation distance, at which point you can invoke Pinsker's inequality to relate it to K-L. As always, which divergence is appropriate/useful for a particular problem will vary (for instance, discrepancy induces a rather strong topology). This might be a useful read. $\endgroup$ – stochasticboy321 Jun 3 '18 at 7:41
  • $\begingroup$ Some insight to be found here: stats.stackexchange.com/questions/188903/… $\endgroup$ – kjetil b halvorsen Aug 22 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.