Does entropy $H(y)$ decrease as $H(x,y)$ decreases when $H(x)$ is fixed? Can't find any proof in Shannon's 1948 paper. Can you provide one or disproof?
Thank you.
P.S.
$H(x)$(or $H(y)$) is the entropy of messages produced by the discrete source $x$(or $y$).
$H(x,y)$ is the joint entropy.
They are all entities in information theory.
 A: Both cases are possible.
 H(X)            [=========================]
 H(Y)   [===============]                         (original)
 H(X,Y) [==================================]


 H(X)            [=========================]  constant 
 H(Y)        [=======]                        less than original
 H(X,Y)      [=============================]  decreases


 H(X)            [=========================]  constant 
 H(Y)        [=======================]        greater than original
 H(X,Y)      [=============================]  decreases

The assertion would be true only if we state that the mutual information $I(X;Y)$ is kept constant.
A: No it doesn't have to.
$H(X,Y) = H(X) + H(Y|X)$
To lower $H(X,Y)$ while keeping $H(X)$ fixed, you need to lower $H(Y|X)$. You can lower $H(Y|X)$ without lowering $H(Y)$ since $0 \leq H(Y|X) \leq H(Y)$ is a measure on how dependent X and Y is. If they are more dependent, there will be less entropy left in Y after you've learned X so $H(Y|X)$ is lower.
From the inequality above, you can also see that lowering $H(Y)$ would also lower $H(Y|X)$ which in turn lowers $H(X,Y)$, so lowering $H(Y)$ is one way but not the only way.
A: The only way H(X,Y) can be reduced is to increase the correlation between X and Y. Since X will be judged on the basis of Y which decoder already have. There must be strong correlation so that decoder can predict the X on basis of Y.
