0
$\begingroup$

I wish to show that $H(X | Y, Z) \leq H(X | Y )$ for random variables $X,Y,Z$, where $H$ is the Shannon entropy. I know how to show $H(X | Y) \leq H(X)$ by using the Chain Rule on $H(X,Y)\leq H(X)+H(Y).$ I want to do something similar with $H(X,Y,Z)\leq H(X)+H(Y)+H(Z)$, but I'm stuck.

$\endgroup$
3
  • $\begingroup$ It might be helpful to include the work you did to show the simpler inequality. $\endgroup$ Apr 18, 2017 at 20:25
  • $\begingroup$ We have $H(Y)+H(X|Y)\leq H(X)+H(Y)$, giving the result. $\endgroup$ Apr 18, 2017 at 20:29
  • 1
    $\begingroup$ Well, that same argument applied to the second inequality gives \begin{align} H(X,Y,Z)&=H(X,Z|Y)+H(Y)\\&\leq H(X)+H(Y)+H(Z)\\\implies H(X,Z|Y)&\leq H(X)+H(Z)\end{align} But then I would think you can apply the chain rule once more to get $H(X,Z|Y)=H(X|Y,Z)+H(Z)$ which implies the desired result. $\endgroup$ Apr 18, 2017 at 20:41

1 Answer 1

1
$\begingroup$

The inequality in the "conditional case" follows the exact same steps as in the "unconditional case" since the same equations and inequalities hold, however, with all quantities involved conditioned on $Z$. In particular, it holds $$ H(X,Y|Z) = H(X|Y,Z)+H(Y|Z) \text{ (chain rule)} $$ and $$ H(X,Y|Z)\leq H(X|Z)+H(Y|Z) \text{ (conditional independence maximizes joint entropy)} $$

You may want to verify these formulas using the same mechanics as in the unconditional case, however, with the conditional version of the joint distribution of $X$ and $Y$ (that is, given $Z$) in place of the unconditional version.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .