2
$\begingroup$

I see the following simplification used frequently in the literature, but I have not been able to verify it.

Let $X$ and $Y$ be absolutely continuous (i.e. they have pdfs) $\mathbb{R}^d$-valued random variables. Assume the joint variable $(X,X+Y)$ is absolutely continuous on $\mathbb{R}^{2d}$. Then $$h(X,X+Y)=h(X,Y).$$

Here $h$ signifies differential entropy, defined by $$h(W)=-\int_{\mathbb{R}^{d_W}}f_W(w)\log(f_W(w))\ dw$$ whenever $W$ is an $\mathbb{R}^{d_W}$-valued random variable with pdf $f_W$.

Note1: $X$ and $Y$ are not assumed to be independent.

Note2: Examples where the lhs is finite but the rhs is not defined would be accepted as a counterexample.

I am also wondering, if the statement can be proved, then is it more generally true that $$h(X,g(X,Y))=h(X,Y)$$ where $g$ is a deterministic function of its arguments?

This question is similar, but seems to concern Shannon entropy (i.e. discrete variables). Shannon Entropy and Differential Entropy have different sets of properties as discussed in these links answer1, answer2, question1,and question2.

$\endgroup$
4
$\begingroup$

Use the fact that if $W$ has a pdf and $A$ is a linear transformation then \begin{align*} h(AW)&=h(W)+\log|\mathrm{det}A|. \end{align*}

In this case, let $W=[X \ \ Y]^T$, a vector valued variable in $\mathbb{R}^{2d}$. Let \begin{align} A=\left[\begin{array}{c c} I_d & 0 \\ I_d & I_d \end{array}\right], \end{align} where $I_d$ is the $d\times d$ identity block. Then $AW=[X \ \ X+Y]^T$. Therefore, \begin{align} h(X,X+Y)&=h(AW) \\ &= h(W)+\log|\mathrm{det}A| \\ &= h(X,Y) + \log(1) \\ &= h(X,Y) \end{align}

$\endgroup$
2
$\begingroup$

Another proof to that of @HaarD is the following.

Using the chain rule,

$$ \begin{align} h(X, X+Y) &= h(X) + h(X+Y\mid X)\\ &=h(X)+h(Y|X)\\ &=h(X,Y), \end{align} $$ where the first equality is application of the chain rule, the second equality holds because adding a constant to a random variable does not change its entropy, and the third equality is again by application of the chain rule.

This result does not generalize for arbitrary $g(X,Y)$.

$\endgroup$
  • $\begingroup$ I'm a little unsure about the reasoning behind the second equality (adding a constant to a random variable does not change its entropy). Would $h(X^2+Y|X)=h(Y|X)$ for the same reason? $\endgroup$ – cantorhead Feb 5 '18 at 22:33
  • 1
    $\begingroup$ @cantorhead yes $\endgroup$ – Stelios Feb 5 '18 at 22:38
  • $\begingroup$ Then by doing the chain rule in the reverse order we get $$h(X,X^2+Y)=h(X,Y).$$ If this is true then it seems more general than the other answer based on linear change of variables. $\endgroup$ – cantorhead Feb 5 '18 at 22:42
  • $\begingroup$ @cantorhead indeed $\endgroup$ – Stelios Feb 5 '18 at 22:52
  • $\begingroup$ Can you give more detail in your answer about why the second equality holds? My original motivation for asking the question was actually to understand why $h(X+Y|X)=h(Y|X)$. You've said that its related to translation invariance. Can you show step by step how translation invariance implies $h(X+Y|X)=h(Y|X)$? $\endgroup$ – cantorhead Feb 6 '18 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.