# Condition for $I(X, Y ; Z) = I(X; Z)$

$$I$$ is mutual information.

\begin{align*} I(X, Y ; Z) &= D_{KL} \left( P_{XYZ} || P_{XY} \otimes P_Z \right)\\ &= \sum_{x, y, z} P_{XYZ}(x, y, z) \log \left( \frac{P_{XYZ}(x, y, z)}{P_{XY}(x, y) P_Z(z)} \right) \\ &= \sum_{x, y, z} P_Y(y) P_{XZ|Y}(x, z | y) \log \left( \frac{ P_{XZ|Y}(x, z | y)}{ P_{X|Y}(x|y) P_Z(z)} \right) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, (1) \end{align*}

and \begin{align*} I(X; Z) &= \sum_{x, z} P_{XZ}(x, z) \log \left( \frac{P_{XZ}(x, z)}{P_X(x) P_Z(z)} \right) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, (2) \end{align*}

I need a necessary and sufficient condition for equality between $$(1)$$ and $$(2)$$.

It seems to me a sufficient condition would be:

$$\text{(X, Z) is independent of Y} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{(stat)}$$

Furthermore, intuitively, we know that the quantity $$I(X, Y ; Z)$$ means the amount of information $$(X, Y)$$ convey about $$Z$$

But if $$X = Y$$ , then $$X$$ conveys the same amount of information about $$Z$$ that $$(X, Y)$$ together do so

$$I(X, Y ; Z) = I(X; Z) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{if } X = Y \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{(stat2)}$$

So I believe the conditions in $$\text{(stat)}$$ and $$\text{(stat2)}$$ are sufficient but not necessary.

• Please define objects in the right order or give references. – dan_fulea Mar 26 at 4:23
• By the chain rule, $I(X,Y;Z) = I(X;Z) + I(Y;Z|X).$ So the equality holds iff $I(Y;Z|X) = 0.$ It is a standard result that this is equivalent to "$Y$ and $Z$ are conditionally independent given $X$" (this is also sometimes stated as 'the Markov chain $Y-X-Z$ holds'). Try to show this. – stochasticboy321 Mar 26 at 19:23
• @dan_fulea I don't understand. There's an information theory tag, and standard notation in information theory for random variables, mutual information, and KL divergence is used. What is undefined? – stochasticboy321 Mar 26 at 19:25
• @stochasticboy321 thanks. I was able to show it. Interestingly, the statement "$Y$ and $Z$ are conditionally independent given $X$" is implied by both $(a)$ $X = Y$ and $(b)$ $(X, Z)$ independent of $Y$ as mentioned in my question above. But the necessary & S condition is what you said. – ironX Mar 26 at 21:07
• @ironX Grand! Would you write an answer for the benefit of others who might have a similar question? – stochasticboy321 Mar 26 at 21:11

The chain rule for mutual information is $$I(X_1, X_2, ..., X_n; Y) = I(X_1; Y) + \sum_{i = 2}^n I(X_i;Y | X_{1}, ..., X_{i-1})$$

Applying this to $$I(X, Y; Z)$$, we get: $$I(X, Y; Z) = I(X; Z) + I(Y; Z | X)$$

Hence, $$I(X, Y; Z) = I(X; Z) \,\, \iff \,\, I(Y; Z| X) = 0$$

In words, $$I(Y; Z | X) = 0$$ means

1. $$Y$$ and $$Z$$ convey no information about each other given $$X$$, or
2. $$Y$$ and $$Z$$ are conditionally independent given $$X$$.

In Markov chain formulation, $$I(Y; Z | X) = 0$$ $$\iff$$ $$X, Y, Z$$ are related by the Markov chain $$Y - X - Z$$

The conditions

1. $$X = Y$$
2. $$(X, Z)$$ independent of $$Y$$

as mentioned in the original question are both sufficient conditions and both imply $$I(Y; Z | X) = 0$$.

But the necessary and sufficient condition is $$I(Y; Z | X) = 0$$.