Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

When a random variable $X$ has only one possible outcome $x_0$, the probability density function at $X=x_0$ is infinite, while the probability density at other locations is zero. Then the p.d.f is exactly a delta function $\Pr(X=x) = \delta(x=x_0)$.

However, when I tried to calculate the entropy of the random variable, the problem arises. How can I calculate the integral $\int_{-\infty}^{+\infty}{\delta(x-x_0)\log\delta(x-x_0) \, dx}$?

share|improve this question
2  
Mathematicians prefer to use measures rather than "densities" for probability distributions that are not absolutely continuous. $Pr(X=x) = 1 \ne \infty = \delta(x-x_0)$ –  GEdgar Sep 4 '12 at 15:26
    
@Strin : I'd write "probability density function" in the title. "Distribution function" is usually construed as "cumulative distribution function". –  Michael Hardy Sep 4 '12 at 17:48
add comment

2 Answers

Consider an absolutely continuous distribution with location parameter $x_0$ and scale parameter $\sigma$. We consider two such distributions: normal $\mathcal{D}_1 = \mathcal{N}\left(x_0, \sigma\right)$, and continuous uniform $\mathcal{D}_2 = \mathcal{U}\left(x_0-\sqrt{3} \sigma, x_0+\sqrt{3} \sigma\right)$. Distributions $\mathcal{D}_1$ and $\mathcal{D}_2$ have equal means and variances.

Carrying out the computation for the Shannon entropy: $H(\mathcal{D}) = \mathbb{E}\left(-\ln(f_\mathcal{D})\right)$ yields the following results: $$ \begin{eqnarray} H\left(\mathcal{D}_1\right) &=& \ln(\sigma)+\frac{\ln(2 \pi)}{2} + \frac{1}{2 \sigma^2} \mathbb{Var}(\mathcal{D}_1) = \ln(\sigma) + \frac{1}{2}\left(1 + \ln(2\pi)\right) \\ H\left(\mathcal{D}_2\right) &=& \mathbb{E}\left(\ln(2 \sqrt{3} \sigma)\right) = \ln(\sigma) + \ln\left(2 \sqrt{3}\right) \end{eqnarray} $$ Additionally, consider a Cauchy distribution $\mathcal{D}_3 = \mathrm{Cauchy}\left(x_0, \sigma\right)$: $$ H\left(\mathcal{D}_3\right) = \ln(\sigma) + \ln(\pi) + \frac{1}{\pi} \int_{-\infty}^\infty \frac{\ln\left(1+x^2\right)}{1+x^2}\mathrm{d}x = \ln(\sigma) + \ln(4 \pi) $$ Notice that in the limit $\sigma \to 0^+$, each distribution converges to a degenerate distribution, localized at $x_0$. For each distribution the entropy diverges to $-\infty$ as $\ln(\sigma)$.

share|improve this answer
    
You showed this fact by computing several distributions. Is there any general theorem saying the entrophy of the delta function is $-\infty$? –  Strin Sep 20 '12 at 15:15
add comment

Try to make a limit of random variables equidistributed in $[x_0-\varepsilon, x_0+\varepsilon]$. What is the entropy of these? What happens with $\varepsilon\to0$?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.