Mutual information as a fraction of entropy?

Question

Suppose I have two (discrete) random variables $(X,Y)$ with some joint distribution $P$. The mutual information $I(X;Y)$ is informally defined as the reduction is the remaining entropy in $X$ once the realization of $Y$ is known, ie

$$I(X;Y) = H(X) - H(X|Y)$$

The units of all the information quantities (entropy and mutual information) are typically bits. I am interested in computing a normalized version of the mutual information which takes a value in $[0,1]$ and is defined as the proportion of reduction in entropy of $X$ once the realization of $Y$ is known, i.e.

$$I'(X;Y) = \frac{I(X;Y)}{H(X)}$$

Are there any references in the literature which use this normalized version $I'(X;Y)$ (which is obviously not symmetric)?

The reason I am interested in this question is to use mutual information to compare the strength of dependencies between random variables (parallel to how correlation is a normalization of covariance for measuring the strength of linear dependence).

If $I(X,Y) = 4$ and $I(W,Z)$ = 1000, this alone does not facilitate a comparison about the relative strength of dependency (are $(X,Y)$ or $(W,Z)$ more strongly dependent?) without knowing the entropies of $X,Y,W,Z$.

Answer 1

Measures of this type have been studied extensively, starting with Shannon himself. Yao (2003) (See Table 6.1) has a decent overview. Your particular definition is not common at all. It's probably because of the asymmetry and also because it offers no real insights. What does it mean to have $I'(X;Y)=4.2$ bits?.

Kvalseth (1987) uses it, but only to build the much more useful symmetrized version $I'(X;Y)=2 I(X;Y)/[H(X)+H(Y)]$.

I'd personally use either this measures, known as normalized mutual information, or the variation of information $VI(X,Y)= H(X,Y)-I(X;Y)$. The latter is a metric, so it has nice intuitive properties.