1
$\begingroup$

Let $X$ be a random variable with density $f$. The differential entropy is defined by $$h(X):=-\int_{\mathbb R^d} f(x)log(f(x))dx.$$

The conditional entropy is defined by replacing the density with conditional density.

Let $X$ and $Y$ be two independent random variables with densities. I want to show $$h(X+Y)> h(X).$$

I feel like we probably need the formula for the chain rule of conditional entropy somewhere, like $h(X+Y)=h(X,X+Y)-h(X|X+Y)$

$\endgroup$

1 Answer 1

2
$\begingroup$

You can do it directly like this \begin{align*} h(X)&=h(X|Y)\\ &=h(X+Y|Y)\\ &\leq h(X+Y) \end{align*}

We can show the inequality is strict by observing that if $X$ and $Y$ are independent then $\text{Cov}(X,Y)=0$ which means that \begin{align*} \text{Cov}(X+Y,Y) &= \mathbb E[(X+Y-\mathbb E[X+Y]))(Y-\mathbb E[Y])]\\ &=\text{Cov}(X,Y)+\text{Var}(Y)\\ &=\text{Var}(Y)\\ &>0 \end{align*} So $X+Y$ and $Y$ are not independent hence $0<I(X+Y;Y)=h(X+Y)-h(X+Y|Y)$ which proves your statement.

$\text{Var}(Y)>0$ is a sufficient condition but may not be necessary.


Let's prove that $h(X|Y)=h(X+Y|Y)$. We can write \begin{align*} h(X|Y)&=-\int p_Y(y) \int p_{X|Y}(x|y) \log p_{X|Y}(x|y) dy\\ &=-\int p_Y(y) \int p_{X+Y|Y}(x+y|y) \log p_{X+Y|Y}(x+y|y) dy\\ &=h(X+Y|Y) \end{align*} By a change of variable $p_{X+Y|Y}(x+y|y)=p_{X|Y}(x|y)$.

$\endgroup$
4
  • $\begingroup$ Okay... but a constant doesn't have a density right? $\endgroup$
    – No One
    Commented Nov 8, 2018 at 16:30
  • $\begingroup$ Also $h(X|Y)=h(X+Y|Y)$ looks simple (One doesn't gain new information from $X+Y$ if they are given $Y$) but pretty hard to prove, Could you provide more details? $\endgroup$
    – No One
    Commented Nov 8, 2018 at 16:35
  • $\begingroup$ @NoOne For the last comment, see my edit. I'm not sure what you mean by "have a density", does that mean that $X$ and $Y$ are continuous random variables ? $\endgroup$
    – P. Quinton
    Commented Nov 9, 2018 at 8:37
  • $\begingroup$ Thanks! I meant I think some functions, like constants or indicator functions, don't have densities... But let me think about it... $\endgroup$
    – No One
    Commented Nov 10, 2018 at 2:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .