Strange statistical concepts like spaces of probability distributions, "metrics" like Fisher information or relative entropy, and convergence with respect to these quantities are necessitated in my research problems. I need to come up to speed on these tools quickly, but can't find any standard, rigorous treatment of them.

Can anyone share good references on this and the needed prerequisite knowledge?

It looks like to even start with this topic you need quite a lot of knowledge in stochastic processes (I am using the two volumes on diffusion by Rodgers as reference), functional analysis (using Folland and Conway), and maybe some differential geometry (using Lee's 3 books).

Edit: This conference paper linked by this question gives a strong argument that studying these things from a geometric perspective is misguided, which rules out information geometry references. But there is still plenty of motivation to study relative entropy and Fisher information.

  • $\begingroup$ How long does it take you to read all these books you mentioned? Seems a pretty big goal, if you just need something that "pops up in your research". It is always better to ask experts and collaborate, in such cases. $\endgroup$ – Peter Franek Feb 5 '16 at 21:31
  • $\begingroup$ If I went through them all properly and hadn't seen any of the material before it would take me at least a year and change of very concentrated effort. Although, I've already seen a lot of the material before and I won't have to become a pro in any of those fields. Those books are just the groundwork I imagine this theory builds off of. $\endgroup$ – Christian Chapman Feb 5 '16 at 21:41

Information geometry is introduced in Amari's book.

A concise treatment of the relation between information theory and statistics can be found in "Information Theory and Statistics : A Tutorial by Csiszar and Shields. Particularly section 3 and 4 are dedicated to information geometry. They are credible researchers on this topic.

There is this chapter of Cover-Thomas book on information theory, targeting mostly engineers.

However, spaces of probability distributions are discussed in many classic books of Probability, particularly where the notion of convergence in distribution is discussed. Billingsley's "Convergence of Probability Measures" is a classic here.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.