Suppose X is a random variable generated from a distribution $f(X|L)$, where $L$ is another random variable that parameterizes the distribution. Further let $L'$ be a third random variable, which may be correlated (arbitrarily) with $X$ but is not equal to $L$. Is it true that the mutual information $I(X;L)$ is strictly (or weakly) greater than $I(X;L')$ ?

It does not seem to make some sense intuitively because if I say $L'$ is such that it can be perfectly determined with $X$, then obviously $X$ and $L'$ have very high mutual information.

Irrespective of whether the statement in question is true or false, can anyone point me to any theorems about comparing mutual informations of $X$ with two variables $L$ and $L'$. Even if there are results about mutual information with conditioning on a third variable, that will also be very helpful. I need any known results about the intuition that a variable which is generated from another variable is bound to have certain mutual information with it and that must be greater than the mutual information that it can have with any other (or perhaps any other in some limited space) variable ?

If more information about the motivation for this problem is required for understanding : I will try to put it in a context - Suppose that there are unbounded number of mixed data samples drawn from $k$ different distributions. Now there is a new data sample generated from one of these distribution that comes in and I would like to be able to say that in expectation, this new data sample must have more mutual information with the parameters of its own generating distribution than that of any other $k-1$ distributions'.

  • $\begingroup$ "It does not seem to make some sense intuitively". It does not make sense. The "parameter" can be arbitrariy "weak", so nothing can be said in general. $\endgroup$ – leonbloy May 3 '17 at 13:24
  • $\begingroup$ Not strict: what if $L'=-L$? Not even weak: what if $L'=X$? $\endgroup$ – Clement C. May 3 '17 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.