Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We know that standard deviation (SD) represents the level of dispersion of a distribution. Thus a distribution with only one value (e.g., 1,1,1,1) has SD equals to zero. Similarly, such a distribution requires little information to be defined. On the other hand, a distribution with high SD requires many bits of information to be defined, therefore we can say its entropy level is high.

So my question: is SD the same as entropy?

If not, which relationship exist between these two measurements?

share|cite|improve this question
up vote 8 down vote accepted

They are not the same. If you have a bimodal distribution with two peaks and allow the spacing between them to vary, the standard deviation would increase as the distance between the peaks increases. However, the entropy $$H(f) = \int f(x) \log f(x) dx$$ doesn't care about where the peaks are, so the entropy would be the same.

share|cite|improve this answer

Entropy and Standard Deviation are certainly not the same, but Entropy in most cases (if not all) depends on the Standard Deviation of the distribution. Two examples:

For the Exponential distribution with density function $$\lambda e^{-\lambda x},\;\; x\ge 0,\, SD=1/\lambda$$we have

$$H(X) = 1-\ln\lambda = 1+\ln SD$$

So as SD increases, so is (here differential) Entropy.

For the Normal distribution, with density function $$\frac{1}{\sigma\sqrt{2\pi}}\, e^{-\frac{(x - \mu)^2}{2 \sigma^2}}, \;\; SD = \sigma$$ we have

$$H(X) = \frac12 \ln(2 \pi e \, \sigma^2) = \frac12 \ln(2 \pi e) +\ln SD $$ so again differential Entropy increases with SD.

(Note that differential Entropy can be negative).

share|cite|improve this answer

More counterexamples:

  1. Let X take be a discrete random variable taking two values $(-a,a)$ with equal probability. Then the variance $\sigma_X^2=a^2$ increases with $a$, but the entropy is constant $H(X)=1$ bit.

  2. Let $X$ be a discrete rv taking values on $1 \cdots N$ with some arbitrary non-uniform distribution $p(X)$. If we permute the values of $p(X)$, the variance will change (decrease if we move the larger values towards the center), but the entropy is constant.

  3. Let $X$ be a continuous rv with uniform distribution on the interval $[-1,1]$ $p(X)=1/2$. Let modify it so that its probability (on the same support) is bigger towards the extremes : say, $p(Y)=|Y|$. Then $\sigma^2_Y > \sigma_X^2$ but $H(Y)< H(X)$ (the uniform distribution maximes the entropy for a fixed compact support).

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.