0
$\begingroup$

Question is as in the title:

What are some of the first examples of Entropy in a course on Introduction to Probability?

I am TA for a course on Introduction to Probability. I was asked to write down some notes on Entropy of a random variable.

I could not find a reference for a book on Probability that introduces Entropy of a random variable but this is what it is :

Let $(\Omega,\rho)$ be a probability space. Let $X:\Omega\rightarrow \mathbb{R}$ be a discrete random variable with $X(\Omega)=\{a_1,\cdots,a_n\}$ and $P(X=a_i)=p_i$ for $1\leq i\leq k$. Then, \textit{entropy of the random variable $X$} is defined to be the sum $$H(X)=-\sum_{i=1}^k p_i \log (p_i).$$

I can then recall examples of distributions and compute their Entropy to get some idea of what is going on something like

$H(X)=p\log (1/p)+(1-p)\log (1/(1-p))$ for Bernouli distribution $Ber(p)$.

$H(X)= -\log p -q^{2}\log q$ for Geometric distribution $Geom(p)$.

Further, I want to give some other examples that gives a better idea of what role Entropy of a random variable plays. Thus, the question:

What are some of the first examples of Entropy in a course on Introduction to Probability?

If it helps, this course is for second year undergrad (physics/chemistry/biology) students.

$\endgroup$

1 Answer 1

1
$\begingroup$

The second edition of Achim Klenke, Probability Theory, Springer has a section on "Entropy and Source Coding Theorem" and it contains also the definition. (The book is free at the moment!) Coding theory might be interesting as a non-trivial example.

In my lectures, the first thing we did was to show that the uniform distribution (i.e. $p_i = \frac{1}{k}$) maximises the (mathematical) entropy. This might not be the easiest calculation to start with (it requires constrained minimisation), but it links entropy with a measure of disorder.

Coming from physics, I think it is important to point out the different sign between the disciplines.

$\endgroup$
1
  • $\begingroup$ It is a part of section 5.3 Strong Law of Large Numbers, in page number 116.. Thanks for the book.. $\endgroup$ Apr 12, 2020 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.