0
$\begingroup$

$I$ is mutual information.

\begin{align*} I(X, Y ; Z) &= D_{KL} \left( P_{XYZ} || P_{XY} \otimes P_Z \right)\\ &= \sum_{x, y, z} P_{XYZ}(x, y, z) \log \left( \frac{P_{XYZ}(x, y, z)}{P_{XY}(x, y) P_Z(z)} \right) \\ &= \sum_{x, y, z} P_Y(y) P_{XZ|Y}(x, z | y) \log \left( \frac{ P_{XZ|Y}(x, z | y)}{ P_{X|Y}(x|y) P_Z(z)} \right) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, (1) \end{align*}

and \begin{align*} I(X; Z) &= \sum_{x, z} P_{XZ}(x, z) \log \left( \frac{P_{XZ}(x, z)}{P_X(x) P_Z(z)} \right) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, (2) \end{align*}

I need a necessary and sufficient condition for equality between $(1)$ and $(2)$.

It seems to me a sufficient condition would be:

$$\text{(X, Z) is independent of $Y$} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{(stat)}$$

Furthermore, intuitively, we know that the quantity $I(X, Y ; Z)$ means the amount of information $(X, Y)$ convey about $Z$

But if $X = Y$ , then $X$ conveys the same amount of information about $Z$ that $(X, Y)$ together do so

$$I(X, Y ; Z) = I(X; Z) \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{if } X = Y \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \text{(stat2)}$$

So I believe the conditions in $\text{(stat)}$ and $\text{(stat2)}$ are sufficient but not necessary.

$\endgroup$
  • $\begingroup$ Please define objects in the right order or give references. $\endgroup$ – dan_fulea Mar 26 at 4:23
  • 1
    $\begingroup$ By the chain rule, $ I(X,Y;Z) = I(X;Z) + I(Y;Z|X). $ So the equality holds iff $I(Y;Z|X) = 0.$ It is a standard result that this is equivalent to "$Y$ and $Z$ are conditionally independent given $X$" (this is also sometimes stated as 'the Markov chain $Y-X-Z$ holds'). Try to show this. $\endgroup$ – stochasticboy321 Mar 26 at 19:23
  • 1
    $\begingroup$ @dan_fulea I don't understand. There's an information theory tag, and standard notation in information theory for random variables, mutual information, and KL divergence is used. What is undefined? $\endgroup$ – stochasticboy321 Mar 26 at 19:25
  • $\begingroup$ @stochasticboy321 thanks. I was able to show it. Interestingly, the statement "$Y$ and $Z$ are conditionally independent given $X$" is implied by both $(a)$ $X = Y$ and $(b)$ $(X, Z)$ independent of $Y$ as mentioned in my question above. But the necessary & S condition is what you said. $\endgroup$ – ironX Mar 26 at 21:07
  • 1
    $\begingroup$ @ironX Grand! Would you write an answer for the benefit of others who might have a similar question? $\endgroup$ – stochasticboy321 Mar 26 at 21:11
1
$\begingroup$

The chain rule for mutual information is $$I(X_1, X_2, ..., X_n; Y) = I(X_1; Y) + \sum_{i = 2}^n I(X_i;Y | X_{1}, ..., X_{i-1})$$

Applying this to $I(X, Y; Z)$, we get: $$I(X, Y; Z) = I(X; Z) + I(Y; Z | X)$$

Hence, $$I(X, Y; Z) = I(X; Z) \,\, \iff \,\, I(Y; Z| X) = 0$$

In words, $I(Y; Z | X) = 0$ means

  1. $Y$ and $Z$ convey no information about each other given $X$, or
  2. $Y$ and $Z$ are conditionally independent given $X$.

In Markov chain formulation, $I(Y; Z | X) = 0$ $\iff$ $X, Y, Z$ are related by the Markov chain $$Y - X - Z$$

The conditions

  1. $X = Y$
  2. $(X, Z)$ independent of $Y$

as mentioned in the original question are both sufficient conditions and both imply $I(Y; Z | X) = 0$.

But the necessary and sufficient condition is $I(Y; Z | X) = 0$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.