16
$\begingroup$

Let two continuous random variables, where the one is a function of the other: $X\, $ and $\, Y=g\left(X\right)$. Their mutual information is defined as $$I\left(X,Y\right)\,=\,h\left(X\right)\,-\,h\left(X|Y\right)=\,h\left(Y\right)\,-\,h\left(Y|X\right)$$ where lowercase h denotes differential entropy, the entropy concept for continuous rv's. It is a proven fact that the mutual information between two variables, for discrete as well as for continuous rv's, is non-negative, and becomes zero only when the two rv's are independent (clearly not our case). Using the fact that Y is a function of X we have $$I\left(X,g\left(X\right)\right)\,=\,h\left(g\left(X\right)\right)\,-\,h\left(g\left(X\right)|X\right) \gt\,0\,\Rightarrow \,h\left(g\left(X\right)\right)\,\gt \,h\left(g\left(X\right)|X\right)$$ Now, differential entropy (unlike entropy for discrete rv's) can take negative values. Assume that it so happens that $h\left(g\left(X\right)\right)\lt\,0$. Then from the positivity of mutual information we obtain $$0\,\gt \,h\left(g\left(X\right)\right)\,\gt \,h\left(g\left(X\right)|X\right) \Rightarrow\; h\left(g\left(X\right)|X\right)\neq\,0$$ And this is the counter-intuitive puzzle: for any discrete random variable Z we always have $h\left(g\left(Z\right)|Z\right)\,=\,0$. This is intuitive: if Z is known, then any function of Z is completely determined -no entropy, no uncertainty remains, and so the conditional entropy measure is zero. But we just saw that, when dealing with continuous rv's where the one is a function of the other, their conditional differential entropy may be non-zero (it doesn't matter whether it is positive or negative), which is not intuitive at all. Because, even in this strange world of continuous rv's, knowing X, completely determines Y=g(X). I have searched high and low to find any discussion, comment or exposition of the matter, but I found nothing. Cover & Thomas book does not mention it, other books do not mention it, a myriad of scientific papers or web sites do not mention it.

My motives: a) Scientific curiosity. b) I want to use the concept of mutual information for continuous rv's in an econometrics paper I am writing, and I feel very uncomfortable to just mention the "non-zero conditional differential entropy" case without being able to discuss it a bit. So any intuition, reference, suggestion, idea, or full answer of course, would be greatly appreciated. Thanks.

$\endgroup$
3
  • 1
    $\begingroup$ Although conditional differential entropy changes, MI does not change for invertible transformations, and is well defined. If you want more interpretable entropies for continuous variables, perhaps you should look into relative entropy. $\endgroup$
    – Memming
    Jul 28, 2013 at 13:34
  • $\begingroup$ Thanks for the suggestion, I will look into it... although the symmetric nature of MI is much more appealing. Still, it is strange that I couldn't find any mention of the case I presented in my question. $\endgroup$ Jul 28, 2013 at 14:15
  • $\begingroup$ @AlecosPapadopoulos I think it returns to the question that why differential entropy is called differential entropy! I remember our professor told us the answer of your question, but unfortunately I don't remember the answer! $\endgroup$ Jul 28, 2013 at 17:09

2 Answers 2

10
$\begingroup$

The paradox can be stated in a simpler form:

We know that $I(X;Y)=h(X)-h(X|Y)\ge 0$ holds, also for continuous variables. Consider the particular case $Y=X$. Then the second term vanishes: $h(X|X)=0$, and we get

$$ I(X;X)= h(X)-h(X|X)=h(X) \ge 0$$

But cannot be right. The differential entropy can be negative. So what?

when dealing with continuous rv's where the one is a function of the other, their conditional differential entropy may be non-zero (it doesn't matter whether it is positive or negative), which is not intuitive at all.

Your problem (and the problem with the above paradox) is to implicitly assume that the concept "zero entropy means no uncertainty" applies also to differential entropy. That's false. The differential entropy is not some type of entropy. It's false that $h(g(X)|X)=0$, it's false that $h(X|X)=0$; and it's false that $h(X)=0$ implies zero uncertainty. The fact that (by a mere change of scale) a differential entropy can be made negative, suggests by itself that here zero differential entropy (conditional or not) has no special meaning. In particular, a uniform variable in $[0,1]$ has $h(X)=0$.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much for reaching back two years. I will read your answer carefully (I am very much still interested in the matter), and maybe I will have a question or two in order to understand it fully. $\endgroup$ May 27, 2015 at 23:24
4
$\begingroup$

Differential entropy is only defined for absolutely continuous random variables with respect to Lebesgue measure. The joint distribution of

$$(X,g(X))$$

is not absolutely continuous because its support is

$$\{(x,g(x)):x\in\mathbb{R}\},$$

a measure 0 subset of $\mathbb{R}^2$.

The definition of conditional entropy requires an absolutely continuous joint distribution, $f(x,y)$: \begin{align} h(X|Y) &= -\int f(x,y) \log f(x|y)\ dxdy \\ &= h(X,Y)-h(Y). \end{align} If $(X,Y)$ is not absolutely continuous against Lebesgue measure in $\mathbb{R}^2$, then there is no function $f(x,y)$ such that the probability measure of $(X,Y)$ can be expressed as an integral against $f$.

Therefore neither the quantity $h(g(X)|X)$, nor the quantity $I(X;g(X))$ are well defined as differential entropies.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. If I understand¨correctly, in naive terms, because $g(X)$ is a function, it follows that the joint support will be a line and not an area in $\mathbb R^2$, which makes it of measure 0, as discussed here, math.stackexchange.com/q/944241/87400 and here, math.stackexchange.com/q/145820/87400 ? $\endgroup$ Feb 3, 2018 at 20:56
  • $\begingroup$ @AlecosPapadopoulos Yes. And the measure 0 support means that the joint variable does not have a pdf. $\endgroup$
    – cantorhead
    Feb 3, 2018 at 22:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .