# Using mutual information to estimate correlation between a continuous variable and a categorical variable

As for the title, the idea is to use mutual information, here and after MI, to estimate "correlation" (defined as "how much I know about A when I know B") between a continuous variable and a categorical variable. I will tell you my thoughts on the matter in a moment, but before I advice you to read this other question/answer on CrossValidated as it contains some useful information.

Now, because we cannot integrate over a categorical variable we need to discretize the continuous one. This can be done quite easily in R, but no matter which language you use there is an important decision that has to be taken. That is, one has to decide a priori the number of "bins" (discrete states) before any discretization can be done.

The main problem, however, is another one: MI ranges from 0 to ∞, as it is an unstandardised measure which unit is the bit. That makes very difficult to use it as a correlation coefficient. This can be partly solved using global correlation coefficient, here and after GCC, which is a standardized version of MI; GCC is defined as follow:

Reference: the formula is from Mutual Information as a Nonlinear Tool for Analyzing Stock Market Globalization by Andreia Dionísio, Rui Menezes & Diana Mendes, 2010.

GCC ranges from 0 to 1, and therefore can easily be used to estimate the correlation between two variables. Problem solved, right? Well, kind of. Because all this process depends heavily on the number of 'bins' we decided to use during the discretization. Here the results of my experiments:

On the y-axis you have GCC and on the x-axis you have the number of 'bins' I decided to use for discretization. The two lines refers to two different analyses I conducted on two different (although very similar) datasets. It is possible to find the complete procedure plus the code used here.

It seems to me that the usage of MI in general and GCC in particular is still controversial. Yet, this confusion may be the result of a mistake from my side. Either the case, I'd love to hear your opinion on the matter (also, do you have alternative methods to estimate correlation between a categorical variable and a continuous one?).

-
You are looking for a "dependence" measure. GCC is one of them but it is difficult to estimate it. First of all, you need an error bar to indicate how confident your estimates are. Secondly, you probably want something that can be reliably estimated. There are many dependence measures out there...perhaps somebody can list them for you. Also, how many discrete categories do you have? And how many data points per category? – Memming Jan 29 '14 at 17:06