1
$\begingroup$

For a Gaussian random variable of unitary variance, the entropy is given by $H = 0.5\log(2 \pi e)$. Also, the mutual information (MI) between correlated Gaussian random variables is given by $MI = -0.5\log(1 - \rho^2)$, where $\rho$ denotes correlation. With these equations in mind, I can think of values for $\rho$ with which the MI is larger than the joint entropy between the two random variables. Am I missing something here or the MI may indeed be larger than the joint entropy?

$\endgroup$

1 Answer 1

1
$\begingroup$

For a Gaussian random variable of unitary variance, the entropy is given by $H=0.5 \log(2 \pi e)$.

Not quite. That's the differential entropy, which is a very different thing from the Shannon entropy. You cannot compare in a meaningful mutual information with differential entropies.

In particular, calling $h()$ the differential entropies, it's true that

$$I(X;Y) = h(X) - h(X|Y)$$

but it's not true that $h(X|Y) \ge 0$. Hence, the mutual information can be larger that the (differential) entropy of any variable (or the joint two variables).

See also here and here and here...

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .