$ H(X_{1})\leqslant H(X_{2})\leqslant,...,\leqslant H(X_{n})$

I seem to have a problem of finding a lower bond of joint entropy. I proved that upper bound using the chain rule and the fact that information of discrete variables is positive.And I obtained result that

$H(X_{1}, X_{2},..., X_{n}) \leqslant $$\sum_{k=1}^{n} H(X_{k})$

Where should I start? I found result that lower bond is $max (H(X_{k}))$?

  • 1
    $\begingroup$ Hint: $H(Y|X) = H(X,Y)-H(X) \ge 0$. $\endgroup$ – Math Lover Apr 12 at 18:15

THe property $H(X_1,X_2) \ge H(X_1)$ should be obvious from the interpretation of entropy, as information content (the information provided by $X_1$ and $X_2$ together cannot be less than that provided by $X_1$ alone). And it can also be proved by the chain rule:

$$H(X_1,X_2) = H(X_1)+H(X_2 | X_1) \ge H(X_1)$$ because $H(X_2 | X_1) \ge 0$ (as any entropy).

We also can write $H(X_1,X_2) \ge H(X_2)$. Hence $H(X_1,X_2) \ge \max(H(X_1),H(X_2))$

You only need to generalize this to $n$ variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.