The differential entropy is translation invariant but not scaling invariant: $h(X+c) = h(x)$ for some constant $c$,and $h(aX) = h(X) + \ln (|a|)$ .

I am interested in an extension of the scaling case, where $Y = m(X)$ for a general function $m$. From Wikipedia, it seems there is an inequality $h(Y)\leq h(x) + \int f_X(x)\log |\frac{\partial m}{\partial x}| dx$ assuming that $X$ and $Y$ have the same dimension. The equality holds when $m$ is a bijection.

I googled around and couldn't find any reference to the material for the inequality above. If directly going from the change of variable for a continuous r.v., it should be $h(Y) = h(X) + \int f_X \log |\frac{\partial m}{\partial x}| dx$ for a general function $m$.

Can someone suggest me any reference to the material in Wikipedia, or point out if I miss something when applying change of variables? Thanks very much!

  • $\begingroup$ Note that the formula you have used is valid only for for monotone $m$ such that $m' \neq 0$. Such functions are bijections, and the equality is met. $\endgroup$ – stochasticboy321 Apr 16 '16 at 2:27
  • $\begingroup$ Thank you for pointing out. Any chance that you are aware of any reference for the inequality on wikipedia? I think the inequality was derived partly from the change of variable method, but have no idea how to proceed. $\endgroup$ – user68187 Apr 16 '16 at 5:58
  • $\begingroup$ Yeah, I'm a little stumped on this. Tried a few things which didn't quite work, and citation chasing on wikipedia lead nowhere. I'll try to play with this as I can, but perhaps raising a bounty on and/or asking a prof about this would be a good idea. $\endgroup$ – stochasticboy321 Apr 16 '16 at 16:33
  • $\begingroup$ Here's the attack I tried, maybe you can make them work: $f_{XY}(x,y) = f_X(x)\delta(m(x) - y)$, and we can compute $f_Y$ using these properties for "nice enough" $m$. Then a direct estimation of $h(Y)$ may be possible, although I seem to be screwing up somewhere... $\endgroup$ – stochasticboy321 Apr 16 '16 at 16:34
  • $\begingroup$ ...while doing this. Alternately, note that $h(Y|X) = -\infty$, which suggests the inequality $h(Y) \le h(X) - h(X|Y)$, by expanding $I(X;Y)$ in two different ways, and an estimate of $h(X|Y)$ might be within reach. Of course, I couldn't make either work, and this might just be digging a hole. $\endgroup$ – stochasticboy321 Apr 16 '16 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.