11
$\begingroup$

Wikipedia says the entropy of the normal distribution is $\frac{1}2 \ln(2\pi e\sigma^2)$

I could not find any proof for that, though. I found some proofs that show that the maximum entropy resembles to $\frac{1}2+\ln(\sqrt{2\pi}\sigma)$ and while I see that this can be rewritten as $\frac{1}2\ln(e\sigma\sqrt{2\pi})$, I do not get how the square root can be get rid of and how the extra $\sigma$ can be put into $\ln$. It is clear that an additional summand $\frac{1}2\ln(\sigma\sqrt{2\pi})$ would help, but where do we get it from? Probably just thinking in the wrong way here...

So, what is the proof for the maximum likelihood entropy of the normal distribution?

$\endgroup$
2
  • 1
    $\begingroup$ Use $\ln(\sqrt{x}) = \frac{1}{2}\ln(x)$ for all $x > 0$. $\endgroup$ May 29, 2016 at 18:23
  • $\begingroup$ $\sigma$ is dimensional so be careful using this as it is not a proper entropy. Probably you want something with correlation? Entropy should probably be scale invariant if using informational form. $\endgroup$
    – mathtick
    Feb 13, 2021 at 12:15

3 Answers 3

13
$\begingroup$

Notice that $\ln(\color{blue}{\sqrt{\color{black}{x}}}) = \ln(x^{\color{blue}{\frac{1}{2}}}) = \color{blue}{\frac{1}{2}}\ln(x)$ and that $\ln(y) \color{red}{+} \ln(z) = \ln(y \color{red}{\cdot} z)$ for all $x,y,z > 0$. Using these identities, let us re-write the maximum entropy, $\frac{1}{2} + \ln(\sqrt{2\pi}\sigma)$, as follows: $$ \begin{align} \frac{1}{2} + \ln(\sqrt{2\pi}\sigma) &= \frac{1}{2} + \ln(\color{blue}{\sqrt{\color{black}{2\pi\sigma^2}}}) \\ &= \frac{1}{2} + \color{blue}{\frac{1}{2}}\ln(2\pi\sigma^2) \\ &= \frac{1}{2}(1 + \ln(2\pi\sigma^2)) \\ &= \frac{1}{2}(\ln(\mathrm{e}) \color{red}{+} \ln(2\pi\sigma^2)) = \frac{1}{2}\ln(\mathrm{e}\color{red}{\cdot}2\pi\sigma^2) \end{align} $$ So, the entropy reported in Wikipedia is correct.

$\endgroup$
5
$\begingroup$

For continuous distribution like Normal/Gaussian we compute the differential entropy.

You can find the derivation here http://www.biopsychology.org/norwich/isp/chap8.pdf

For more info on differential entropy I recommend the book "Elements of Information Theory" by Cover and Thomas.

$\endgroup$
1
  • $\begingroup$ And we should note here that DE in original form is probably not what people want (dimensionally incorrect) etc. Probably need to normalize by trace of cov or something like that to yield something invariant to scaling. $\endgroup$
    – mathtick
    Feb 13, 2021 at 12:08
4
$\begingroup$

You have already gotten some good answers, I thought I could add something more of use which is not really an answer, but maybe good if you find differential entropy to be a strange concept.

Since we can not store a real or continuous number exactly, entropy for continuous distributions conceptually mean something different than entropy for discrete distributions.

It means the information required except for the resolution of representation. Take for example the uniform distribution on $[0,2^a-1]$ for an integer $a$. At integer resolution it will have $2^a$ equiprobable states and that would give $a$ bits of entropy. Also, the differential entropy is $\log(2^a-0)$, which happens to be the same. But if we want another resolution, lesser or more bits are of course required. Double resolution ($\pm 0.5$) would require 1 more bit (on average).

$\endgroup$
1
  • $\begingroup$ thanks for pointing that out, any extra information are welcome to me $\endgroup$
    – IceFire
    May 29, 2016 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.