0
$\begingroup$

I have two continuous random variables X and Y and I know that their mutual information is quite small (less than 0.1 nats).

I understand that for discrete random variables, mutual information, and in particular, its normalized version with respect to the maximum entropy among X and Y gives an adequate characterization of the "amount of dependence" between X and Y:

$$\frac{I(X;Y)}{\min[H(X),H(Y)]}$$

However, for continuous random variables, their corresponding entropies are typically infinite and the above ratio goes to zero. Is there a way to characterize the "amount of dependence" between such two continuous random variables based on mutual information and argue that one can consider them independent without introducing a significant error in his/her analysis?

$\endgroup$
2
$\begingroup$

This "measure" of dependence between $X$ and $Y$ you are proposing, apart from being unconventional, does not work as you expect. The ratio can be arbitrarily close to zero even in cases of dependent variables.

For example, consider the binary random variable $X \in \{a, b\}$ with $\mathbb{P}(X=a)=\mathbb{P}(X=b)=1/2$ and the random variable $Y$ which, given $X=a$, takes one of the values in $\{1,2,\ldots,M\}$ with equal probability and, given $X=b$, takes one of the values in $\{M+1,M+2,\ldots,2 M\}$ with equal probability. Note that $X$ and $Y$ are "highly dependent" in the sense that knowledge of $Y$ implies perfect knowledge of $X$.

It is easy to see that $I(X;Y)=1$ (bits). Also note that $Y$ is uniformly distributed over $\{1,2,\ldots,2M\}$, therefore, $H(Y)=\log_2 (2M) \rightarrow \infty$ for $M \rightarrow \infty$, resulting in a ratio arbitrary close to zero.

$\endgroup$
  • $\begingroup$ Oops sorry Stelios! I made a typo (I corrected it). The idea of normalizing the mutual information I(X;Y) to its maximum value which is min{H(X), H(X)} is a common method for characterizing the dependence between two discreet random variables. However, in my case the variables are continuous so this ratio does not give any indication. Would you happen to know of an acceptable way to quantify this dependence (preferably using mutual information) for the continuous case? $\endgroup$ – Sotiris Apr 3 '18 at 0:06
  • $\begingroup$ Hi @Sotiris. I am not aware of a metric that quantifies dependence with mutual information. Maybe you could also try asking the signal prcessing stack exchange. I also don't see why "the entropies are typically infinite in the continuous case". This table list the most common entropies, which are not infinite in general $\endgroup$ – Stelios Apr 3 '18 at 7:56
  • $\begingroup$ Hi Stelios! I will do what you suggest and ask in signal processing! The above-mentioned quantity is a common way to characterize the dependence between two discrete random variables. Check for example this: en.wikipedia.org/wiki/Mutual_information#Normalized_variants The table concerns the differential entropies h(X) of the random variables (which can be positive or negative). The actual entropy of the discretized random variable X_Δ is about H(Χ_Δ) = n + h(X) which tends to infinity as the number of bins n goes to infinity. $\endgroup$ – Sotiris Apr 4 '18 at 20:14
  • $\begingroup$ @Sotiris Make sure in your question to explicitly mention that you are considering the discretized approximation of the continuous variable(s) and the corresponding entropy definition (since, otherwise, one expects that your are considering the differential entropy). It would also help to give some motivation of why you consider this form of entropy, as it is not at all convenient, due to the exact issue you mention, which makes it impractical in general (not only for the measure calculation you are interested in). $\endgroup$ – Stelios Apr 4 '18 at 21:06
  • $\begingroup$ Thank you Stelios! My focus is not on the entropy of the discretized random variable itself. It is on finding a way to characterize the dependence between two continuous random variables in the same way as this dependence is characterized by the normalized mutual information for discrete variables. The problem is that this ratio cannot be extended as it is to continuous random variables. The use of differential entropy in the denominator is meaningless and the use of discrete variable entropy goes to infinity. $\endgroup$ – Sotiris Apr 5 '18 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.