Suppose I have two (discrete) random variables $(X,Y)$ with some joint distribution $P$. The mutual information $I(X;Y)$ is informally defined as the reduction is the remaining entropy in $X$ once the realization of $Y$ is known, ie

$$I(X;Y) = H(X) - H(X|Y)$$

The units of all the information quantities (entropy and mutual information) are typically bits. I am interested in computing a normalized version of the mutual information which takes a value in $[0,1]$ and is defined as the proportion of reduction in entropy of $X$ once the realization of $Y$ is known, i.e.

$$I'(X;Y) = \frac{I(X;Y)}{H(X)}$$

Are there any references in the literature which use this normalized version $I'(X;Y)$ (which is obviously not symmetric)?

The reason I am interested in this question is to use mutual information to compare the strength of dependencies between random variables (parallel to how correlation is a normalization of covariance for measuring the strength of linear dependence).

If $I(X,Y) = 4$ and $I(W,Z)$ = 1000, this alone does not facilitate a comparison about the relative strength of dependency (are $(X,Y)$ or $(W,Z)$ more strongly dependent?) without knowing the entropies of $X,Y,W,Z$.

  • 1
    $\begingroup$ I found the so-called "information gain ratio" popping up in decision tree learning (C45 algorithm, etc.). en.wikipedia.org/wiki/… $\endgroup$ – Bernhard Jul 9 '15 at 8:35

Measures of this type have been studied extensively, starting with Shannon himself. Yao (2003) (See Table 6.1) has a decent overview. Your particular definition is not common at all. It's probably because of the asymmetry and also because it offers no real insights. What does it mean to have $I'(X;Y)=4.2$ bits?.

Kvalseth (1987) uses it, but only to build the much more useful symmetrized version $I'(X;Y)=2 I(X;Y)/[H(X)+H(Y)]$.

I'd personally use either this measures, known as normalized mutual information, or the variation of information $VI(X,Y)= H(X,Y)-I(X;Y)$. The latter is a metric, so it has nice intuitive properties.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.