# Proving an Entropy inequality

I wish to show that $H(X | Y, Z) \leq H(X | Y )$ for random variables $X,Y,Z$, where $H$ is the Shannon entropy. I know how to show $H(X | Y) \leq H(X)$ by using the Chain Rule on $H(X,Y)\leq H(X)+H(Y).$ I want to do something similar with $H(X,Y,Z)\leq H(X)+H(Y)+H(Z)$, but I'm stuck.

• It might be helpful to include the work you did to show the simpler inequality. Apr 18, 2017 at 20:25
• We have $H(Y)+H(X|Y)\leq H(X)+H(Y)$, giving the result. Apr 18, 2017 at 20:29
• Well, that same argument applied to the second inequality gives \begin{align} H(X,Y,Z)&=H(X,Z|Y)+H(Y)\\&\leq H(X)+H(Y)+H(Z)\\\implies H(X,Z|Y)&\leq H(X)+H(Z)\end{align} But then I would think you can apply the chain rule once more to get $H(X,Z|Y)=H(X|Y,Z)+H(Z)$ which implies the desired result. Apr 18, 2017 at 20:41

The inequality in the "conditional case" follows the exact same steps as in the "unconditional case" since the same equations and inequalities hold, however, with all quantities involved conditioned on $Z$. In particular, it holds $$H(X,Y|Z) = H(X|Y,Z)+H(Y|Z) \text{ (chain rule)}$$ and $$H(X,Y|Z)\leq H(X|Z)+H(Y|Z) \text{ (conditional independence maximizes joint entropy)}$$
You may want to verify these formulas using the same mechanics as in the unconditional case, however, with the conditional version of the joint distribution of $X$ and $Y$ (that is, given $Z$) in place of the unconditional version.