1
$\begingroup$

Consider two discrete random variables $X$ $\{x_1,x_2,\dots,x_n\}$ and $Y$ $\{y_1,y_2,\dots, y_n\}$. Lets say that entropy $H(X)=0$ i.e. $X$ has a probability distribution s.t. $P(X=x_j) = 1$ for only one $j \in \{1,2,\dots,n\}$ and $P(X=x_j) = 0$, otherwise. In this case the mutual information between $X$ and $Y$ is

$I(X;Y) = H(X) + H(Y) - H(X,Y) = 0 + H(Y) - H(Y)$ [Since, $H(X,Y) = H(Y)$ as $X$ has zero entropy]. Therefore, $I(X;Y) = 0$ when $H(X) = 0$

My question is, am I wrong anywhere ? If not, what is the intuition behind this, i.e. why does the mutual information becomes zero when $H(X)$ becomes zero ?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

Intuitively, $I(X; Y)$ reflects the amount of information you gain about $Y$ after observing $X$ (or vice versa). If $H(X) = 0$, there is no information gained through observing $X$; there is nothing unpredictable about it. Thus observing $X$ tells you nothing that you didn't already know about $Y$, so indeed, $I(X; Y) = 0$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .