Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

$X$ is a random variable with normal distribution, assume $Y=X$, what is the mutual information $I(X;Y)$?

I guess that $h(Y|X)=0$ since when $X$ is known, $Y$ is completely known, so $$I(X;Y)=h(Y)-h(Y|X)=h(Y)=\frac{1}{2}\log 2\pi e\sigma^2$$ nat.

But, I was told I was wrong! and a numerical computation also shows that the value of $$I(X;Y) \neq \frac{1}{2}\log 2\pi e\sigma^2$$ Where is my mistake? Please help me out of this problem, thanks a lot! (Please note that $X$ and $Y$ are both continuous).

share|cite|improve this question
up vote 0 down vote accepted

If $(X_1,X_2)$ is a Gaussian vector each variable having the same variance ($\sigma^2$) with covariance matrix $K$, then $$h(X_1,X_2)=\frac{1}{2}\log((2\pi e)^2|K|)=\frac{1}{2}\log((2\pi e)^2 \sigma^4(1-\rho^2))$$ where $\rho$ is the correlation coefficient.

Now $I(X_1:X_2)=2h(X_1)-h(X_1,X_2)=-\frac{1}{2}\log(1-\rho^2)$.

When $X_1=X_2$, $\rho=1$ and hence $I(X:X)=\infty$.

share|cite|improve this answer
so h(X|X) should be $-\infty$ instead of 0 as in discrete case. and if i run that program using more and more samples, the program will produce larger and larger outcome. Thank you! – Zhou Heng Jun 20 '12 at 7:23
This answer is wrong. It should be unselected. The answer by @avitus is the right one. – becko Apr 10 '15 at 18:21

For any r.v. $X$


To see this, put $Y=X$ in $I(X,Y)=H(X)-H(X|Y)$ and use

$$H(X|X)=0.~~ (*)$$

In summary, the mutual information of $X$ with itself is just its self information $H(X)$, as $H(X|X)$, i.e. the residual average information carried by $X$ conditional $X$ is zero, i.e. $(*)$ holds.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.