A good textbook to learn about entropy and information theory I’m wondering if someone could recommend me a good graduate level textbook or resource to better learn about entropy and information theory, divergence criteria and so on. I have had two graduate courses on probability, one on measure theory and one on stochastic control. I’ve only seen very little about this in probability books, and in large deviations books, with a focus on rate functions. 
Thanks!
 A: I recommend Cover and Thomas, 
https://www.goodreads.com/book/show/433439.Elements_of_Information_Theory
as well as Gallager
https://www.amazon.com/Information-Theory-Reliable-Robert-Gallager/dp/0471290483
A: I recommend Patrick Billingsley’s beautiful book “Ergodic theory and information”. Rigorous but very ‘readerfriendly’. The treatment of the concept of entropy is measure theoretical, not topological. It is published in 1965, so you should probably consult a university library. 
Btw, for both treatments of entropy and their interrelationships, you should consult Walters ‘Ergodic Theory’ (GTM). Also excellent but a bit more demanding. 
A: Try Khinchin's book.
For a more intuitive, thorough, and literary artful discussion of what entropy is, see if you can't get a copy of Renyi's Diary on Information Theory.
A: Late to the party, but for people who will find this answer in the future, I would wholeheartedly recommend the first part of McKay's Information theory, inference and learning algorithms. It is freely available online, and there are accompanying lectures on YouTube.
http://www.inference.org.uk/mackay/itila/
