Assume that the continuous random variable $X$ has a distribution (in a closed form expression) with differential entropy $h(X)$.

Q) Then, is it true for any continuous distribution that the log-standard deviation is additively separable from the entropy expression s.t. $$h(X) = \log \sigma + C,$$ where $\sigma$ is the standard deviation of distribution and $C$ may be a function of parameters of distribution? If my claim is false for some continuous distributions, then for what family of distributions exactly does the claim hold?

I emphasize that when $\sigma$ is dependent on a shape parameter of the distribution, the shape parameter must fixed so that varying $\sigma$ only varies the "spread'' of distribution. Let me illustrate with a few examples.

For a Gaussian random variable $X$, we see \begin{align} h(X)=\log(\sigma \sqrt{2 \pi e}) = \log \sigma + C, \end{align} where $C = \log \sqrt{2 \pi e}$.

For an Erlang random variable $X$, we see \begin{align} h(X) &= \frac{1}{\psi(k)}(1-k) + \log \frac{\Gamma(k)}{\lambda} + k, \\ &= \log \frac{\Gamma(k) \sigma}{\sqrt{k}}+ \frac{1}{\psi(k)}(1-k) + k \\ &= \log { \sigma}+ C, \end{align} where $k$ is the shape parameter and $C = \log \frac{\Gamma(k)}{\sqrt{k}}+ \frac{1}{\psi(k)}(1-k) + k $. Notice how I used the fact that $k$, a shape parameter, is fixed but $\lambda$ may be not when $\sigma$ is varied.

  • $\begingroup$ I do not understand completely. Do you mean to say that $h(X) = \log \sigma + C$,where $C$ shouldn't depend on $\sigma$? What about this en.wikipedia.org/wiki/Cauchy_distribution? $\endgroup$ – Kumara Feb 22 '13 at 5:24
  • $\begingroup$ I am sorry that I didn't include the condition but I am assuming that the variance of the distribution is finite. $\endgroup$ – tatterdemalion Feb 22 '13 at 13:22

If $\sigma$ only affects the "spread" of $X$, as you require, then you can write $X = \mu + \sigma Y$, where the distribution of $Y$ does not depend on $\mu$ or $\sigma$. In that case you have indeed $h(X) = h(\sigma Y) = h(Y) + \log \sigma$.

  • $\begingroup$ What do you mean by spread? $\endgroup$ – Kumara Feb 25 '13 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.