1
$\begingroup$

Consider two discrete random variables $X$ $\{x_1,x_2,\dots,x_n\}$ and $Y$ $\{y_1,y_2,\dots, y_n\}$. Lets say that entropy $H(X)=0$ i.e. $X$ has a probability distribution s.t. $P(X=x_j) = 1$ for only one $j \in \{1,2,\dots,n\}$ and $P(X=x_j) = 0$, otherwise. In this case the mutual information between $X$ and $Y$ is

$I(X;Y) = H(X) + H(Y) - H(X,Y) = 0 + H(Y) - H(Y)$ [Since, $H(X,Y) = H(Y)$ as $X$ has zero entropy]. Therefore, $I(X;Y) = 0$ when $H(X) = 0$

My question is, am I wrong anywhere ? If not, what is the intuition behind this, i.e. why does the mutual information becomes zero when $H(X)$ becomes zero ?

$\endgroup$
2
$\begingroup$

Intuitively, $I(X; Y)$ reflects the amount of information you gain about $Y$ after observing $X$ (or vice versa). If $H(X) = 0$, there is no information gained through observing $X$; there is nothing unpredictable about it. Thus observing $X$ tells you nothing that you didn't already know about $Y$, so indeed, $I(X; Y) = 0$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.