# From combinatorial entropy to Shannon entropy

This blog, Shannon entropy, by Yurii Lahodiuk shows the link (derivation) of Shannon entropy from basic combinatorics. I would like to know the first person that made this combinatorial interpretation of Shannon entropy.

Who is the first person who gave this interpretation? What is the first appearance of this combinatorial interpretation in the literature?

• Yurii Lahodiuk is a member here known as @stemm – Mohammad Al-Turkistany Jul 12 '18 at 14:50
• FWIW there's a beta History of Science and Mathematics stack where this question would also be on topic. To be clear, I am not advising that you cross-post. However, if you don't get an answer here you could request migration and take your chances there. – Peter Taylor Jul 13 '18 at 9:27

## 2 Answers

I have written the blog post: http://lagodiuk.github.io/computer_science/2016/10/31/entropy.html.

Frankly speaking, I have learnt about this interpretation from the examination-preparation notes, which I have found here: http://nsu.videosoft.org/static/content/pdf/entropy.pdf and http://nsu.videosoft.org/2009/exam/ (in Russian language).

Which seems to be a part of the course taught by Kovalev Dmitry Sergeyevich "Data presentation and compression" (in Novosibirsk State University): http://nsu.videosoft.org/2009/.

• Thanks Yurii, I have enjoyed reading your nice blog. – Mohammad Al-Turkistany Oct 28 '18 at 13:04
• Thank you for your feedback, Mohammad! – stemm Oct 28 '18 at 13:23

According to Wikipeida, this interpretation (known as the maximum entropy principle) was given by Jaynes (based on a suggestion by Wallis). Here is an excerpt:

The principle was first expounded by E. T. Jaynes in two papers in 1957 where he emphasized a natural correspondence between statistical mechanics and information theory. In particular, Jaynes offered a new and very general rationale why the Gibbsian method of statistical mechanics works. He argued that the entropy of statistical mechanics and the information entropy of information theory are basically the same thing.