Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

$X$ is a random variable with normal distribution, assume $Y=X$, what is the mutual information $I(X;Y)$?

I guess that $h(Y|X)=0$ since when $X$ is known, $Y$ is completely known, so $$I(X;Y)=h(Y)-h(Y|X)=h(Y)=\frac{1}{2}\log 2\pi e\sigma^2$$ nat.

But, I was told I was wrong! and a numerical computation also shows that the value of $$I(X;Y) \neq \frac{1}{2}\log 2\pi e\sigma^2$$ Where is my mistake? Please help me out of this problem, thanks a lot! (Please note that $X$ and $Y$ are both continuous).

share|improve this question

2 Answers 2

up vote 1 down vote accepted

If $(X_1,X_2)$ is a Gaussian vector each variable having the same variance ($\sigma^2$) with covariance matrix $K$, then $$h(X_1,X_2)=\frac{1}{2}\log((2\pi e)^2|K|)=\frac{1}{2}\log((2\pi e)^2 \sigma^4(1-\rho^2))$$ where $\rho$ is the correlation coefficient.

Now $I(X_1:X_2)=2h(X_1)-h(X_1,X_2)=-\frac{1}{2}\log(1-\rho^2)$.

When $X_1=X_2$, $\rho=1$ and hence $I(X:X)=\infty$.

share|improve this answer
    
so h(X|X) should be $-\infty$ instead of 0 as in discrete case. and if i run that program using more and more samples, the program will produce larger and larger outcome. Thank you! –  Zhou Heng Jun 20 '12 at 7:23

For any r.v. $X$

$$I(X,X)=H(X).$$

To see this, put $Y=X$ in $I(X,Y)=H(X)-H(X|Y)$ and use

$$H(X|X)=0.~~ (*)$$

In summary, the mutual information of $X$ with itself is just its self information $H(X)$, as $H(X|X)$, i.e. the residual average information carried by $X$ conditional $X$ is zero, i.e. $(*)$ holds.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.