# Conditional entropy under quantization

Let $X$ be a continuous random variable and $X^n$ its quantization that becomes finer with larger $n$. Let $Y$ be a deterministic function of $X$. Then we have that the conditional entropy $$H(Y|X) = 0$$ because $Y$ is a function of $X$.

Furthermore, we have $H(Y|X^n) = \infty$ for all $n$ because the distribution of $Y|X^n = x^n$ is continuous.

Now my question is, is it valid to say that $\lim_{n \to \infty} H(Y|X^n) = H(Y|X)$? Intuitively, I would say it is, but it seems impossible to construct an ($\epsilon$-$\delta$) proof for this. Any help is appreciated.

• The differential entropy is not the entropy. Your first assertion (if $H$ is a differential entropy) is false. See eg math.stackexchange.com/a/1398471/312 or math.stackexchange.com/questions/454078/… – leonbloy Mar 16 '16 at 0:01
• I don't consider differential entropy here. Since $Y$ is known for a given $X = x$, $H(Y|X = x) = 0$ and then also $H(Y|X) = 0$, right? – user303913 Mar 16 '16 at 6:38
• I don't understand how you define the entropy of a continuous variable ($Y$) if it's not the differential entropy. – leonbloy Mar 16 '16 at 10:52
• Ok, so I agree that for continuous variables there is differential entropy (denote it with $h(Y)$). In most cases $h(Y) < \infty$ and $H(Y) = \infty$ (where $H(Y)$ is normal entropy). But now, since $Y$ is a function of $X$, the distribution of $Y|X=x$ is no longer continuous but discrete, or am I totally wrong here? Then $H(Y|X) = \int_{\mathcal X} H(Y|X=x) \text{d}P_X(x) = 0$. – user303913 Mar 16 '16 at 11:03
• Then you must use different definitions for the two entropies. The "infinite" entropy $H(Y|X^n) = \infty$ can only make sense if defined as a limit. Then, you have two iterated limits, and your assertion is ambiguous. Take as an example $Y=X$ and $X$ uniform in $[0,a]$ – leonbloy Mar 16 '16 at 11:45

The probability distribution of $Y|X^n=x$ converges to the probability distribution $Y|X=x$ for increasing $n$. Nevertheless, and that is the problem in this case, entropy is not continuous in a converging sequence of probability measures (see, e.g., this paper).