2
$\begingroup$

The Shannon entropy of a discrete random variable ${\textstyle X}$ with possible values ${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$ and probability mass function ${\textstyle \mathrm {P} (X)}$ is defined: $${\displaystyle \mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{2}\mathrm {P} (x_{i})}}.$$ The measure should be maximal if all the outcomes are equally likely (uncertainty is highest when all possible events are equiprobable); in this case $H\leq\log _{2}(n)$.

Let us consider two discrete random variables ${\displaystyle X}$ and ${\displaystyle Y}$. Mutual information can be equivalently expressed as $$\operatorname {I} (X;Y)=\mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y),$$ where ${\displaystyle \mathrm {H} (X)}$ and ${\displaystyle \mathrm {H} (Y)}$ are the marginal entropies, and ${\displaystyle \mathrm {H} (X,Y)}$ is the joint entropy of ${\displaystyle X}$ and ${\displaystyle Y}$.

I wonder if there exists an upper bound for $\operatorname {I} $ like the $H\leq\log _{2}(n)$ and if it involves the equiprobability of events.

$\endgroup$

1 Answer 1

6
$\begingroup$

First :

in this case $H\leq\log _{2}(n)$.

should be

in this case $H = \log _{2}(n)$; hence in general $H\leq\log _{2}(n)$

Regarding upper bounds for $I(X;Y)$, you should write

$$I(X;Y)=H(X) -H(X|Y) \implies I(X;Y) \le H(X) \le \log(|\mathcal X|)$$

where $|\mathcal X|$ is the cardinality of the alphabet for $X$ ($n$ above). Because you can also write $I(X;Y)=H(Y) -H(Y|X)$ you get

$$I(X;Y) \le \log( \min(|\mathcal X|,|\mathcal Y|))$$

$\endgroup$
1
  • $\begingroup$ Thank you @leonbloy, it's clear! $\endgroup$
    – Mark
    Commented Feb 10, 2019 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .