# What are some of the first examples of Entropy in a course on Introduction to Probability?

Question is as in the title:

What are some of the first examples of Entropy in a course on Introduction to Probability?

I am TA for a course on Introduction to Probability. I was asked to write down some notes on Entropy of a random variable.

I could not find a reference for a book on Probability that introduces Entropy of a random variable but this is what it is :

Let $$(\Omega,\rho)$$ be a probability space. Let $$X:\Omega\rightarrow \mathbb{R}$$ be a discrete random variable with $$X(\Omega)=\{a_1,\cdots,a_n\}$$ and $$P(X=a_i)=p_i$$ for $$1\leq i\leq k$$. Then, \textit{entropy of the random variable $$X$$} is defined to be the sum $$H(X)=-\sum_{i=1}^k p_i \log (p_i).$$

I can then recall examples of distributions and compute their Entropy to get some idea of what is going on something like

$$H(X)=p\log (1/p)+(1-p)\log (1/(1-p))$$ for Bernouli distribution $$Ber(p)$$.

$$H(X)= -\log p -q^{2}\log q$$ for Geometric distribution $$Geom(p)$$.

Further, I want to give some other examples that gives a better idea of what role Entropy of a random variable plays. Thus, the question:

What are some of the first examples of Entropy in a course on Introduction to Probability?

If it helps, this course is for second year undergrad (physics/chemistry/biology) students.

The second edition of Achim Klenke, Probability Theory, Springer has a section on "Entropy and Source Coding Theorem" and it contains also the definition. (The book is free at the moment!) Coding theory might be interesting as a non-trivial example.

In my lectures, the first thing we did was to show that the uniform distribution (i.e. $$p_i = \frac{1}{k}$$) maximises the (mathematical) entropy. This might not be the easiest calculation to start with (it requires constrained minimisation), but it links entropy with a measure of disorder.

Coming from physics, I think it is important to point out the different sign between the disciplines.

• It is a part of section 5.3 Strong Law of Large Numbers, in page number 116.. Thanks for the book.. Apr 12, 2020 at 8:25