1
$\begingroup$

I have a problem where the mutual information between dependent random variables seems to decrease as their correlation increase, which goes against my intuition of mutual information. I'd appreciate if anyone could explain this phenomenom or where my reasoning is flawed.

Setup:

I have a system of $n$ continuous and dependent Gaussian variables calculated using a Gaussian Process (these variables come from a dataset of $n$ elements). The covariance of the joint distribution, $\mathbf{\Sigma} \in \mathbb{R}^{n \times n}$, is calculated using a covariance function, specifically the squared exponential function/kernel (reference):

$$ k_{SE}(x,x')=\sigma^2\exp{(-\frac{(x-x')^2}{2l^2})}$$

where $x, x'$ are elements in my dataset and $\sigma^2$ and $l$ are hyperparameters representing the kernel variance and lenghtscale, respectively. My understanding is that a greater $l$ leads to higher covariance values in $\mathbf{\Sigma}$ and thus, a higher correlation between the $n$ random variables of the system.

I calculate the mutual information between two subsets of variables, $X_1$ and $X_2$. These are two non-overlapping partitions of the original set of variables. Let $\mathbf{\Sigma}_{12}$ be the covariance matrix of the joint of these two subsets, and let $\mathbf{\Sigma}_1$ and $\mathbf{\Sigma}_2$ represent the covariance matrix of the marginal distributions. I compute mutual information the following way:

$$ I(X_1;X_2) = H(X_1) + H(X_2) - H(X_1,X_2) $$

which, in words, translates to summing the entropy of each variable subset and subtracting the entropy of the joint. Each $H(X_i)$ is calculated using the Gaussian entropy formula:

$$ H(X_i) = 0.5\ln(\det(2\pi e \mathbf{\Sigma}_i)) $$

The issue / question:

I have observed that with increasing lenghtscale parameter (and thus, with increasing correlation between random variables), the mutual information will increase then decrease beyond a certain threshold. What causes this to happen? Am I missing something? Thanks.

An example:

The following plot illustrates the behavior I am talking about. Mutual information starts decreasing with a lenghtscale of more than 1.

mutual information vs lenghtscale

This figure was generated with the following python script:

import numpy as np
import matplotlib.pyplot as plt

def k(x,xp,l,s):
    """Squared exponential kernel"""
    return s**2 * np.exp(-(x-xp)**2/(2*l**2))

def MI(C_1,C_2,C_12):
    """Mutual information between Gaussian variables"""
    H_1 = 0.5*np.linalg.slogdet(2*np.pi*np.e*C_1)[1]
    H_2 = 0.5*np.linalg.slogdet(2*np.pi*np.e*C_2)[1]
    H_12 = 0.5*np.linalg.slogdet(2*np.pi*np.e*C_12)[1]
    return H_1 + H_2 - H_12


n = 10  # size of the dataset
s = 1   # stdv of the kernel function

# Dataset
X = np.random.normal(size=n)

# >> X
# array([-0.55037423,  0.03189518, -1.64622258,  0.56081342,  0.80264256,
#         0.41539512,  0.66173243,  0.12141524,  0.70379963,  0.54954399])

# The lenghtscales to simulate
ls = np.linspace(0.1,10,100)

MIs = []
for l in ls:

    # The covariance matrix of the system
    C = np.zeros((n,n))
    for i, x in enumerate(X):
        for j, xp in enumerate(X):
            C[i,j] = k(x,xp,l,s)
    
    # Add small constant along diagonal for numerical stability
    C = C + 0.0001*np.eye(C.shape[0])

    # The covariance matrix of variable subsets
    # (here, the first five and last three variables of the system)
    n_1 = [0,1,2,3,4]
    n_2 = [7,8,9]

    C_1 = C[n_1][:,n_1]
    C_2 = C[n_2][:,n_2]
    C_12 = C[n_1+n_2][:,n_1+n_2]
    
    # The mutual information between both subsets of variables
    MIs.append(MI(C_1,C_2,C_12))

# Plot
fig = plt.figure(dpi=100)
plt.plot(ls, MIs)
plt.xlabel("lenghtscale")
plt.ylabel("Mutual Information")
plt.title("Variation of MI with increasing lenghtscale")
$\endgroup$
2
  • $\begingroup$ Thanks for the reply! My understanding is that the greater the correlation between two random variables (i.e. the greater the lengthscale), the more learning about one variable should tell us about the other. As such, my intuition is that a greater correlation should lead to a higher mutual information. It's unclear to me what is happening in the example above. As you pointed out, it means that the entropies of the marginals is greater/lower than the entropy of the joint for different lenghtscale values, which I need to wrap my head around. $\endgroup$
    – olamarre
    Commented Feb 21, 2022 at 4:14
  • $\begingroup$ Actually, I plotted each entropy value ($H(X_1)$, $H(X_2)$ and $H(X_1,X_2)$) and it looks like every entropy value decreases with an increasing lenghtscale parameter. The joint entropy decreases at a faster rate than the marginal entropies, and the rate of decrase of each slows down with greater lenghtscales. So it appears that an increasing correlation decreases all entropies. Interesting, I wonder if there is an intuitive explanation for this. Also, the entropy values all dip below zero (which I think is normal given that I calculate the differential entropy of continuous variables). $\endgroup$
    – olamarre
    Commented Feb 21, 2022 at 4:23

1 Answer 1

2
$\begingroup$

Intuitively, as the correlation increases, the entropy must decreases. This is because correlation reduce the degree of freedom within the space described by the marginal distributions. As the mutual information is concerned, since $I(X_1;X_2) = H(X_2) − H(X_2|X_1) \geq 0$, we have $H(X_2) \geq H(X_2|X_1)$. Thus conditioning reduces entropy. So we can conclude that $H(X_1;X_2)=H(X_1)+H(X_2|X_1) \leq H(X_1)+H(X_1)$, i.e. the joint entropy is upper bounded by the sum of the marginal entropies. When the joint entropy decreases it must diminish more rapidly than the sum of $H(X_1)$ and $H(X_2)$. The mutual information increases as uncertainties reduce. However, because the only constraint we have is that $H(X_1;X_2) \leq H(X_1)+H(X_1)$, the mutual information can exhibit a maximum. The entropy $h(X)$ of a continuous random variable $X$ with density $f(x)$ is called differential entropy. The differential entropy can be negative. For instance if you consider a random variable distributed uniformly in the range $S=[0,a]$ with density $1/a$ within the support $S$, you can realize that for $a<1$ the differential entropy is negative.

$\endgroup$
1
  • $\begingroup$ Thank you so much for the explanation! $\endgroup$
    – olamarre
    Commented Feb 22, 2022 at 20:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .