# The differential entropy of the sum of independent random variables

Let $$X$$ be a random variable with density $$f$$. The differential entropy is defined by $$h(X):=-\int_{\mathbb R^d} f(x)log(f(x))dx.$$

The conditional entropy is defined by replacing the density with conditional density.

Let $$X$$ and $$Y$$ be two independent random variables with densities. I want to show $$h(X+Y)> h(X).$$

I feel like we probably need the formula for the chain rule of conditional entropy somewhere, like $$h(X+Y)=h(X,X+Y)-h(X|X+Y)$$

You can do it directly like this \begin{align*} h(X)&=h(X|Y)\\ &=h(X+Y|Y)\\ &\leq h(X+Y) \end{align*}

We can show the inequality is strict by observing that if $$X$$ and $$Y$$ are independent then $$\text{Cov}(X,Y)=0$$ which means that \begin{align*} \text{Cov}(X+Y,Y) &= \mathbb E[(X+Y-\mathbb E[X+Y]))(Y-\mathbb E[Y])]\\ &=\text{Cov}(X,Y)+\text{Var}(Y)\\ &=\text{Var}(Y)\\ &>0 \end{align*} So $$X+Y$$ and $$Y$$ are not independent hence $$0 which proves your statement.

$$\text{Var}(Y)>0$$ is a sufficient condition but may not be necessary.

Let's prove that $$h(X|Y)=h(X+Y|Y)$$. We can write \begin{align*} h(X|Y)&=-\int p_Y(y) \int p_{X|Y}(x|y) \log p_{X|Y}(x|y) dy\\ &=-\int p_Y(y) \int p_{X+Y|Y}(x+y|y) \log p_{X+Y|Y}(x+y|y) dy\\ &=h(X+Y|Y) \end{align*} By a change of variable $$p_{X+Y|Y}(x+y|y)=p_{X|Y}(x|y)$$.

• Okay... but a constant doesn't have a density right? Commented Nov 8, 2018 at 16:30
• Also $h(X|Y)=h(X+Y|Y)$ looks simple (One doesn't gain new information from $X+Y$ if they are given $Y$) but pretty hard to prove, Could you provide more details? Commented Nov 8, 2018 at 16:35
• @NoOne For the last comment, see my edit. I'm not sure what you mean by "have a density", does that mean that $X$ and $Y$ are continuous random variables ? Commented Nov 9, 2018 at 8:37
• Thanks! I meant I think some functions, like constants or indicator functions, don't have densities... But let me think about it... Commented Nov 10, 2018 at 2:01