From Wikipedia we can read:

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables.

in fact, $I(X;Y)=I(Y;X)$. But we know that MI can measure also non-linear dependence. To clarify that concept, I made this Venn diagram to describe what I know about dependence, linear correlation and causality in probability and statistics.

dependence, linear correlation and causality

$Y = X^2$ is an example of dependence between two RVs, that is not contained in the set of correlation, and cannot be detected by Pearson's coefficient. In that case, mutual information will be greater than zero suggesting a mutual dependence. But that's not true! I mean: Y depends on X but not viceversa.

If Mutual Information measures dependence, why is it symmetric, while dependence is not?


It's not true that $Y$ depends on $X$ and not vice versa. If $Y=X^2$, then $X=\sqrt Y$.

Edit in response to the comments:

Two random variables are either dependent or independent; there is no such thing as one variable being dependent on another. You may be confusing this with the concept of one variable being a function of another. Indeed, in $Y=X^2$ you'd be right to say that $Y$ is a function of $X$ but $X$ is not a function of $Y$, since the sign of $X$ is not determined by $Y$.

To simplify things and focus on this loss of sign, consider $Y=|X|$ instead. If you know $X$, you know $Y$, and if you know $Y$, you know $X$ up to the sign. If the distribution of $X$ is symmetric, the information in the sign is exactly one bit. The mutual information (measured in bits) is

$$ I(X;Y)=H(X)-H(X\mid Y)=H(X)-1 $$

or equivalently

$$ I(Y;X)=H(Y)-H(Y\mid X)=H(Y)\;. $$

The mutual information is the information that you get about one variable by learning the value of the other. If you learn the value of $X$, you know $Y$; if you learn the value of $Y$, you know $X$ up to a sign; the information gain is the same in both cases, since there's one bit of information more in $X$ than in $Y$.

  • $\begingroup$ Sure! But is not the limited domain of the inverse function a "problem" in term of statistical dependence? We are loosing all the negative values of the old $X$... $\endgroup$ – floatingpurr Jul 28 '16 at 12:16
  • $\begingroup$ So, is the key concept: dependence is always mutual (ie symmetrical)? $\endgroup$ – floatingpurr Jul 28 '16 at 12:18
  • 3
    $\begingroup$ @superciccio14: I expanded the answer in response to your comments. My impression is that you're taking the word "dependence" in the Wikipedia article too literally. Dependence, used in the technical sense, is all or nothing and applies to a pair of variables, so it's symmetric by definition. The more interesting thing here, which isn't obvious by definition, is that the information gain is also symmetric, i.e. you gain as much information about $X$ by learning the value of $Y$ as you gain about $Y$ by learning the value of $X$. $\endgroup$ – joriki Jul 28 '16 at 13:08
  • 1
    $\begingroup$ Perfect explanation. Thanks! A last question: How do we know that "the information in the sign is exactly one bit"? $\endgroup$ – floatingpurr Jul 28 '16 at 13:26
  • 4
    $\begingroup$ @superciccio14: It's a binary piece of information with equal probability for either possibility; that's almost the definition of a bit. The information (measured in bits) of the distribution of the sign is $-\frac12\log_2\frac12-\frac12\log_2\frac12=1$. $\endgroup$ – joriki Jul 28 '16 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.