I have a (perhaps simple) question about the chain rule for mutual information. The formula is given by

$$I(X_1, X_2, ..., X_n; Y) = \sum_{i=1}^{n} I(X_i; Y| X_{i-1}, X_{i-2}, ..., X_1)$$

My question is how to use it on the following equation:

$$I(X; Y_1, Y_2) = I(X; Y_1) + I(X; Y_2| Y_1) \hspace{3cm} (1)$$

I just get $I(X; Y_1, Y_2) = I(Y_1, Y_2; X) = I(X; Y_1) + I(Y_2; X|Y_1)$

With the definition of conditional information and the chain rule for entropy it follows that

$I(X,Y_2|Y_1) = H(X|Y_1) - H(Y_2,X|Y_1) + H(Y_2|Y_1)$


$I(Y_2;X|Y_1) = H(Y_2|Y_1) - H(X,Y_2|Y_1) + H(X|Y_1)$

This leads to the equation $H(Y_2, X|Y_1) = H(X, Y_2| Y_1)$, which is obviously not true. Where do I make a mistake? Also, how can i derive euqation (1) directly by the chain rule? I also followed the proof on wikipedia, which I understood. But I still don't see it from the chain rule for mutual information.

Thank you very much for your help and patience :)

  • $\begingroup$ "which is obviously not true". Obviously that's not true :-) .The equality is true, same as it's true that $H(A,B)=H(B,A)$. The conditioning applies to all the variables (which can be thought as a single multidimensional variable). This might help math.stackexchange.com/questions/481895 $\endgroup$ – leonbloy Dec 4 '17 at 21:18

The equation $H(Y_2, X|Y_1) = H(X, Y_2|Y_1)$ is true. More generally, you can change around the order of the random variables under the entropy functional without changing its value, which follows informally from the fact that $p_{(X,Y,Z)}(x,y,z) = p_{(Y,X,Z)}(y,x,z)$ and the definition of entropy: $$H(X,Y|Z) = -\sum p_{(X,Y,Z)}(x,y,z) \log \frac{p_{(X,Y,Z)}(x,y,z)}{p_Z(z)} \\ = -\sum p_{(Y,X,Z)}(y,x,z) \log \frac {p_{(Y,X,Z)}(y,x,z)}{p_Z(z)} = H(Y,X|Z).$$

By a very similar argument $I(X;Y_2|Y_1) = I(Y_2;X|Y_1)$ so your application of the chain rule is correct and (1) follows from the work you've already done: $$ I(X;Y_1,Y_2)=I(Y_1,Y_2;X)=I(X;Y_1)+I(Y_2;X|Y_1) = I(X;Y_1)+I(X;Y_2|Y_1).$$

  • $\begingroup$ Thank you Suhas! Unfortunately, I was too fast with my assumption ... but I get i know. $\endgroup$ – rabatt13 Dec 4 '17 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.