# Mutual Information Chain Rule

I have a (perhaps simple) question about the chain rule for mutual information. The formula is given by

$$I(X_1, X_2, ..., X_n; Y) = \sum_{i=1}^{n} I(X_i; Y| X_{i-1}, X_{i-2}, ..., X_1)$$

My question is how to use it on the following equation:

$$I(X; Y_1, Y_2) = I(X; Y_1) + I(X; Y_2| Y_1) \hspace{3cm} (1)$$

I just get $I(X; Y_1, Y_2) = I(Y_1, Y_2; X) = I(X; Y_1) + I(Y_2; X|Y_1)$

With the definition of conditional information and the chain rule for entropy it follows that

$I(X,Y_2|Y_1) = H(X|Y_1) - H(Y_2,X|Y_1) + H(Y_2|Y_1)$

and

$I(Y_2;X|Y_1) = H(Y_2|Y_1) - H(X,Y_2|Y_1) + H(X|Y_1)$

This leads to the equation $H(Y_2, X|Y_1) = H(X, Y_2| Y_1)$, which is obviously not true. Where do I make a mistake? Also, how can i derive euqation (1) directly by the chain rule? I also followed the proof on wikipedia, which I understood. But I still don't see it from the chain rule for mutual information.

Thank you very much for your help and patience :)

• "which is obviously not true". Obviously that's not true :-) .The equality is true, same as it's true that $H(A,B)=H(B,A)$. The conditioning applies to all the variables (which can be thought as a single multidimensional variable). This might help math.stackexchange.com/questions/481895 – leonbloy Dec 4 '17 at 21:18

The equation $H(Y_2, X|Y_1) = H(X, Y_2|Y_1)$ is true. More generally, you can change around the order of the random variables under the entropy functional without changing its value, which follows informally from the fact that $p_{(X,Y,Z)}(x,y,z) = p_{(Y,X,Z)}(y,x,z)$ and the definition of entropy: $$H(X,Y|Z) = -\sum p_{(X,Y,Z)}(x,y,z) \log \frac{p_{(X,Y,Z)}(x,y,z)}{p_Z(z)} \\ = -\sum p_{(Y,X,Z)}(y,x,z) \log \frac {p_{(Y,X,Z)}(y,x,z)}{p_Z(z)} = H(Y,X|Z).$$
By a very similar argument $I(X;Y_2|Y_1) = I(Y_2;X|Y_1)$ so your application of the chain rule is correct and (1) follows from the work you've already done: $$I(X;Y_1,Y_2)=I(Y_1,Y_2;X)=I(X;Y_1)+I(Y_2;X|Y_1) = I(X;Y_1)+I(X;Y_2|Y_1).$$