Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From Wikipedia:

Let $X$ be a random variable with a probability density function $f$ whose support is a set $\mathbb{X}$. The differential entropy $h(f)$ is defined as $$ h(f) = -\int_\mathbb{X} f(x)\log f(x)\,dx \quad \text{i.e.} \quad \mathrm{E}_f(-\log f(X)). $$

The discrete entropy of a discrete probability measure is also defined as $\mathrm{E}_p (-\log p(X))$, where $p$ is the mass probability function, which can be viewed as the density function with respect to the counting measure on the discrete sample space.

Am I right that the concept of entropy depends on the underlying measure of the sample space, since the integrand is the density function wrt some underlying measure?

If yes, what is the underlying measure on the sample space for differential entropy that Wiki refers to in the above quote?

What is the/some general structure(s) defined on the sample space for it to be meaningful to discuss entropy, more than $\mathbb{R}^n$ with Lebesgue measure being the underlying measure?

Thanks and regards!

share|cite|improve this question
This is a relevant quote from Shannon's paper: "We will not attempt, in the continuous case, to obtain our results with the greatest generality, or with the extreme rigor of pure mathematics, since this would involve a great deal of abstract measure theory and would obscure the main thread of the analysis". – Jacopo Notarstefano Jan 5 '12 at 19:42
If I agreed with that, I wouldn't have asked. – Tim Jan 6 '12 at 3:38
up vote 2 down vote accepted

In general, one can define the relative entropy between two probability measures $\mu$ and $\nu$ (given that $\nu\ll\mu$) as $$H(\nu\mid\mu)=-\int\frac{d\nu}{d\mu}(x)\log\left(\frac{d\nu}{d\mu}(x)\right)\mu(dx)$$ where $\frac{d\nu}{d\mu}$ is the Radon-Nikodym derivative of $\nu$ with respect to $\mu$. In fact, one can put any convex function $U:\mathbb{R}\to\mathbb{R}$ instead of $U(x)=-x\log(x)$.

When one speaks of a probability density, one is speaking of a real-valued r.v. $X$ whose law $\mu_X(A)=\mathbb{P}(X^{-1}(A))$ is absolutely continuous with respect to the Lebesgue measure $\lambda$ on $\mathbb{R}$ and the density function $f$ is nothing more than the Radon-Nikodym derivative $\frac{d\mu_X}{d\lambda}$. In the case of a discrete r.v. one replaces the Lebesge measure with the counting measure on $\mathbb{N}$.

With this in mind, is easy to see that the definition you gave above is nothing more than the entropy $H(\mu_X\mid\lambda)$, where $\mu_X$ is the law of $X$ and $\lambda$ is the Lebesgue measure on $\mathbb{R}$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.