I am reading Elements of Information Theory (2nd Ed.) by Cover. I have something that I couldn't figure out.

On page 248, theorem 8.3.1

If the density f(x) of the random variable X is Riemann integrable, then $H(X^Δ)+log(⁡Δ)→h(f)=h(X)$, as $Δ→0$.
Thus, the entropy of an n-bit quantization of a continuous random variable X is approximately h(X) + n.

The continuous random variable X is divided into bins of length $\Delta$.
$X^Δ=x_i\quad if\quad i\Delta\le X<(i+1)\Delta$. Where $\Delta f(x_i)=\int_{i\Delta}^{(i+1)\Delta}f(x)dx$.
The book gives a proof of the theorem, but what I didn't understand is how the last conclusion is reached. Clearly, if the range of X is [0, 1), the length of the bin $\Delta$ is related to n. $-log_2(\Delta)=n$ if the range is [0, 1). But this condition is not required in the theorem. Of course, if the range is finite, when $\Delta$ approaches 0, $-log_2(\Delta) \rightarrow n$. Also, the equation only holds when $\Delta$ is close to 0, which is also not stated in the conclusion. ("Thus, the entropy of an n-bit quantization of a continuous random variable X is approximately h(X) + n.")

Did I miss something, or is there some other proofs of that statement?

Thanks a lot.


In the book, by a "$n-$bit quantization" they mean how many binary (fractional) digits of precision we have.

When we speak in decimal base, we say we have "5 decimal digits" of precision (or that 5 decimal fractional digits are meaningful) when we quantize the real line in intervals of length $1/10000=1/10^5$. (Granted, the word precision is also used with different meanings; often it means the amount of significant digits). Similarly, to have 5 bits of precision (or a $5-$bit quantization) we must quantize the real range in intervals of length $\Delta=1/2^5$. Hence, indeed, $-\log_2 \Delta = n$.

See also the second item in example 8.3.1.

I'm not sure if this answers your doubt.

  • $\begingroup$ Thank you. I had thought it was the total number of digits or bits required to describe the value, something like significant digits. $\endgroup$ – LLS Apr 12 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.