The Shannon entropy of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ is defined: $${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{2}\mathrm {P} (x_{i})}}.$$ The measure should be maximal if all the outcomes are equally likely (uncertainty is highest when all possible events are equiprobable); in this case $H\leq\log _{2}(n)$.

Let us consider two discrete random variables ${\displaystyle X}$ and ${\displaystyle Y}$. Mutual information can be equivalently expressed as $$\operatorname {I} (X;Y)=\mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y),$$ where ${\displaystyle \mathrm {H} (X)}$ and ${\displaystyle \mathrm {H} (Y)}$ are the marginal entropies, and ${\displaystyle \mathrm {H} (X,Y)}$ is the joint entropy of ${\displaystyle X}$ and ${\displaystyle Y}$.

I wonder if there exists an upper bound for $\operatorname {I} $ like the $H\leq\log _{2}(n)$ and if it involves the equiprobability of events.


First :

in this case $H\leq\log _{2}(n)$.

should be

in this case $H = \log _{2}(n)$; hence in general $H\leq\log _{2}(n)$

Regarding upper bounds for $I(X;Y)$, you should write

$$I(X;Y)=H(X) -H(X|Y) \implies I(X;Y) \le H(X) \le \log(|\mathcal X|)$$

where $|\mathcal X|$ is the cardinality of the alphabet for $X$ ($n$ above). Because you can also write $I(X;Y)=H(Y) -H(Y|X)$ you get

$$I(X;Y) \le \log( \min(|\mathcal X|,|\mathcal Y|))$$

  • $\begingroup$ Thank you @leonbloy, it's clear! $\endgroup$ – Mark Feb 10 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.