Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

The mutual information is defined on random variables. That is, $I(X;Y)$ denotes the mutual information between random variables $X$ and $Y$.

On the other hand, the the Kullback-Leibler divergence is defined on probability distributions; i.e., $D_{\mathrm{KL}}(P\|Q)$ denotes the K-L divergence of the probability distribution $P$ with respect to the probability distribution $Q$.

Why aren't $I$ and $D_{\mathrm{KL}}$ both defined on random variables (or both defined on probability distributions)?

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

I think it is actually natural to use different notation so in this case. Mutual information is defined on the joint of $X$ and $Y$, and if written as a function of the joint as $I(p(X,Y))$, there could be confusion in higher dimensions. Suppose $X = [x_1, \ldots, x_n]$ and $Y = [x_{n+1}, \ldots, x_{n+m}]$. Then writing $I(p(x_1, \ldots, x_{n+m}))$ would be ambiguous without specifying which variables form $X$ and which form $Y$.

On the other hand, Kullback-Leibler divergence is defined on two marginal distributions of $X$ and $Y$, and the joint must be ignored (or assumed independent). Hence writing $D_{KL}(X,Y)$ would be confusing if they are indeed dependent. In this case writing the marginal distributions directly makes more sense.

And of course, you know that mutual information is just the Kullback-Leibler divergence between the product of marginals and the joint distribution.

share|improve this answer
add comment

Both Mutual Information and Kullback-Leibler Divergence involve the probability mass or density functions of two random variables, depending on whether they are discrete or continuous r.v.'s. Namely, they are both defined for both kinds of r.v.'s. Both are expected values (of a different quantity of course). Both measures need random variables and their pmf or pdf to exist. In other words, the answer to your question is: Because even scientists cannot be as exact and internally consistent (at least in terminology) as science demands.

share|improve this answer
+1 for the last sentence: "Because even scientists cannot be as exact and internally consistent (at least in terminology) as science demands." :D –  Sadeq Dousti Aug 19 '13 at 15:21
@ Sadeq Dousti Hey, thanks. I visited your profile. Could you send me some references related to "Zero-Knowledge" models? Sounds intriguing. My e-mail is $papadopalex(at)aueb.gr$. Thanks again. –  Alecos Papadopoulos Aug 19 '13 at 22:55
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.