I asked a similar question yesterday, but I organized my question here a little and further asked my second question.

Suppose $X$ is a continuous random variable with the pdf $f_x$, and $Y=g(X)$. If $g$ is a bijection, then via the change of variable method, the pdf of $Y$ is $f_y = f_x/|J|$, where $|J|$ is the Jacobian of $g$. Therefore,

$$h(Y)=h(X)+\int f_x \log |J| dx$$.

From Wikipedia, if $dim(X)=dim(Y)$, there exists an upper bound on $h(Y)$ for a general map $g$:

$$h(Y)\leq h(X)+\int f_x \log |J| dx \tag{1}$$ My first question is how to derive the inequality in Eq.(1). Second, is there a general approach to derive $h(Y)$ or an upper bound on it if $dim(X)\neq dim(Y)$.

I'd appreciate any help and suggestions!

If $g$ maps between two sets that have the same dimension, the inequality may be strict. The inequality appears in Papoulis' book on Probability, Random Variables, and Stochastic Processes - an exact value, albeit depending on the function $g$ in a complicated way, can be found in Corollary 1 in this paper:

$$h(Y) = h(X)+\int f_x\log |J|dx - H(X|Y). $$

If $g$ maps between sets of different dimension, then the Jacobian matrix is not square and computing its determinant does not make sense. I'm not sure if there is an easy way to derive the differential entropy of $Y$ from the differential entropy of $X$ in this case...

  • Thank you Bernhard. I found the reference while reading your dissertation as well. – user68187 May 16 '16 at 18:38

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.