# Upper bound of Mutual Information

The Shannon entropy of a discrete random variable $${\textstyle X}$$ with possible values $${\textstyle \left\{x_{1},\ldots ,x_{n}\right\}}$$ and probability mass function $${\textstyle \mathrm {P} (X)}$$ is defined: $$\mathrm {H} (X)=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{2}\mathrm {P} (x_{i})}}.$$ The measure should be maximal if all the outcomes are equally likely (uncertainty is highest when all possible events are equiprobable); in this case $$H\leq\log _{2}(n)$$.

Let us consider two discrete random variables $$X}$$ and $$Y}$$. Mutual information can be equivalently expressed as $$\operatorname {I} (X;Y)=\mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y),$$ where $$\mathrm {H} (X)}$$ and $$\mathrm {H} (Y)}$$ are the marginal entropies, and $$\mathrm {H} (X,Y)}$$ is the joint entropy of $$X}$$ and $$Y}$$.

I wonder if there exists an upper bound for $$\operatorname {I}$$ like the $$H\leq\log _{2}(n)$$ and if it involves the equiprobability of events.

## 1 Answer

First :

in this case $$H\leq\log _{2}(n)$$.

should be

in this case $$H = \log _{2}(n)$$; hence in general $$H\leq\log _{2}(n)$$

Regarding upper bounds for $$I(X;Y)$$, you should write

$$I(X;Y)=H(X) -H(X|Y) \implies I(X;Y) \le H(X) \le \log(|\mathcal X|)$$

where $$|\mathcal X|$$ is the cardinality of the alphabet for $$X$$ ($$n$$ above). Because you can also write $$I(X;Y)=H(Y) -H(Y|X)$$ you get

$$I(X;Y) \le \log( \min(|\mathcal X|,|\mathcal Y|))$$

• Thank you @leonbloy, it's clear! – Mark Feb 10 at 11:08