$ H(X_{1})\leqslant H(X_{2})\leqslant,...,\leqslant H(X_{n})$

I seem to have a problem of finding a lower bond of joint entropy. I proved that upper bound using the chain rule and the fact that information of discrete variables is positive.And I obtained result that

$H(X_{1}, X_{2},..., X_{n}) \leqslant $$\sum_{k=1}^{n} H(X_{k})$

Where should I start? I found result that lower bond is $max (H(X_{k}))$?

  • 1
    $\begingroup$ Hint: $H(Y|X) = H(X,Y)-H(X) \ge 0$. $\endgroup$
    – Math Lover
    Commented Apr 12, 2019 at 18:15

1 Answer 1


THe property $H(X_1,X_2) \ge H(X_1)$ should be obvious from the interpretation of entropy, as information content (the information provided by $X_1$ and $X_2$ together cannot be less than that provided by $X_1$ alone). And it can also be proved by the chain rule:

$$H(X_1,X_2) = H(X_1)+H(X_2 | X_1) \ge H(X_1)$$ because $H(X_2 | X_1) \ge 0$ (as any entropy).

We also can write $H(X_1,X_2) \ge H(X_2)$. Hence $H(X_1,X_2) \ge \max(H(X_1),H(X_2))$

You only need to generalize this to $n$ variables.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .