I always thought that mutual information and information gain refer to the same thing, however looking at Wikipedia:



I see that Information gain is something completely different and asymmetrical,

what are the differences in practice ? when should I choose one of the other ?


We know that $H(X)$ quantifies the amount of information that each observation of $X$ provides, or, equivalently, the minimal amount of bits that we need to encode $X$ ($L_X \to H(X)$, where $L_X$ is the optima average codelength - first Shannon theorem)

The mutual information $$I(X;Y)=H(X) - H(X \mid Y)$$ measures the reduction in uncertainity (or the "information gained") for $X$ when $Y$ is known.

It can be written as $$I(X;Y)=D(p_{X,Y}\mid \mid p_X \,p_Y)=D(p_{X\mid Y} \,p_Y \mid \mid p_X \,p_Y)$$ wher $D(\cdot)$ is the Kullback–Leibler divergence or distance, or relative entropy... or information gain (this later term is not so much used in information theory, in my experience).

So, they are the same thing. Granted, $D(\cdot)$ is not symmetric in its arguments, but don't let confuse you. We are not computing $D(p_X \mid \mid p_Y)$, but $D(p_X \,p_Y\mid \mid p_{X,Y})$, and this is symmetric in $X,Y$.

A slightly different situation (to connect with this) arises when one is interested in the effect of knowing a particular value of $Y=y$ . In this case, because we are not averaging on $y$, the amount of bits gained [*] would be $ D(p_{X\mid Y} \mid \mid p_X )$... which depends in $y$.

[*] To be precise, that's actually the amount of bits we waste when coding the conditioned source $X\mid Y=y$ as if we didn't knew $Y$ (using the unconditioned distribution of $X$)

| cite | improve this answer | |
  • 1
    $\begingroup$ Often for something to be a distance I think it should be symmetric. That could be the reason to call it divergence instead. $\endgroup$ – mathreadler Mar 5 '16 at 18:23
  • $\begingroup$ @mathreadler Yes, that's mentioned here en.wikipedia.org/wiki/… $\endgroup$ – leonbloy Mar 5 '16 at 18:31
  • $\begingroup$ So the answer is...they are the same? $\endgroup$ – information_interchange Apr 18 '19 at 0:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.