Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = 0$$ because $Y$ is a function of $X$.

Furthermore, we have $H(Y|X^n) = \infty$ for all $n$ because the distribution of $Y|X^n = x^n$ is continuous.

Now my question is, is it valid to say that $\lim_{n \to \infty} H(Y|X^n) = H(Y|X)$? Intuitively, I would say it is, but it seems impossible to construct an ($\epsilon$-$\delta$) proof for this. Any help is appreciated.

  • $\begingroup$ The differential entropy is not the entropy. Your first assertion (if $H$ is a differential entropy) is false. See eg math.stackexchange.com/a/1398471/312 or math.stackexchange.com/questions/454078/… $\endgroup$ – leonbloy Mar 16 '16 at 0:01
  • $\begingroup$ I don't consider differential entropy here. Since $Y$ is known for a given $X = x$, $H(Y|X = x) = 0$ and then also $H(Y|X) = 0$, right? $\endgroup$ – user303913 Mar 16 '16 at 6:38
  • $\begingroup$ I don't understand how you define the entropy of a continuous variable ($Y$) if it's not the differential entropy. $\endgroup$ – leonbloy Mar 16 '16 at 10:52
  • $\begingroup$ Ok, so I agree that for continuous variables there is differential entropy (denote it with $h(Y)$). In most cases $h(Y) < \infty$ and $H(Y) = \infty$ (where $H(Y)$ is normal entropy). But now, since $Y$ is a function of $X$, the distribution of $Y|X=x$ is no longer continuous but discrete, or am I totally wrong here? Then $H(Y|X) = \int_{\mathcal X} H(Y|X=x) \text{d}P_X(x) = 0$. $\endgroup$ – user303913 Mar 16 '16 at 11:03
  • $\begingroup$ Then you must use different definitions for the two entropies. The "infinite" entropy $H(Y|X^n) = \infty$ can only make sense if defined as a limit. Then, you have two iterated limits, and your assertion is ambiguous. Take as an example $Y=X$ and $X$ uniform in $[0,a]$ $\endgroup$ – leonbloy Mar 16 '16 at 11:45

The probability distribution of $Y|X^n=x$ converges to the probability distribution $Y|X=x$ for increasing $n$. Nevertheless, and that is the problem in this case, entropy is not continuous in a converging sequence of probability measures (see, e.g., this paper).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy