# Mutual information of two random variables

I am wondering how one does calculate the mutual information for the following setup. Suppose we have two random variables, $$X_0=\{x_1,x_2,x_3\}$$ $$X_1=\{x_1,x_2,x_3,x_4\}$$ where $P(X_0=x_1)=P(X_0=x_2)=P(X_0=x_3)=1/3$ and $P(X_1=x_1)=P(X_1=x_2)=P(X_1=x_3)=P(X_1=x_4)=1/4$

• Are $X_0$ and $X_1$ independent? – Josu Etxezarreta Martinez May 4 '18 at 9:37
• @Josu Is it possible to know the answer for both independent and dependent cases? – Wiliam May 4 '18 at 9:39

Note that $X_1=\{X_0,x_4\}$. It follows that

\begin{align} I(X_0;X_1) &= H(X_0)-H(X_0 \mid X_1)\\ &=H(X_0)-H(X_0 \mid X_0, x_4)\\ &=H(X_0)\\ &=\log(3), \end{align}

You can also find the same result starting from $I(X_0;X_1) = H(X_1)-H(X_1 \mid X_0)$. I will leave this as an exercise for you.

• Thanks. How did you get $H(X_0 \mid X_0, x_4)=0$? – Wiliam May 4 '18 at 12:30
• @William I expect you understand that $H(X_0|X_0)=0$ with the intuition that, conditioned on $X_0$, there is, of course, no uncertainty on $X_0$. It trivially follows that, conditioned on $X_0$ and any other (arbitrary) variable as well, there is still no uncertainty on $X_0$. – Stelios May 4 '18 at 18:03

If $X_0$ and $X_1$ are independent, then the answer to the question is straightforward as

$I(X_0,X_1)=H(X_0)-H(X_0|X_1)=H(X_0)-H(X_0)=0$, where the second step is done due to the fact that the random variables are independent. Note that the mutual information is always zero when the random variables are independent.

For the case where the variables are not independent, the answer is not straightforward, as it depends on the conditional probability distribution $p_{X_0|X_1}(x_0|x_1)$ and so $H(X_0|X_1)\neq H(X_0)$.

Consequently, an answer cannot be given if the conditional probability of both random variables is unknown.

• Thanks but I gave the probabilities, and the conditional probability is given by $P(X_0|X_1)=\frac{P(X_0,X_1)}{P(X_0)}$ I just do not know how to calculate is if the out comes for $X_0$ and $X_1$ are different. – Wiliam May 4 '18 at 9:52
• Ok, you have defined conditional probability. Anyway you have not given the distribution $P(X_0,X_1)$, so nothing can be said yet about the mutual information. – Josu Etxezarreta Martinez May 4 '18 at 9:57
• I am having difficulties in making the distribution $P(X_0,X_1)$. I just know that $P(X_0=x_i)=\{1/3,1/3,1/3\}$ and $P(X_1=x_i)=\{1/4,1/4,1/4,1/4\}$ How does one calculate the $P(X_0,X_1)$ for such setup? – Wiliam May 4 '18 at 10:02
• I have even asked for this here: (math.stackexchange.com/questions/2764714/…) but no good answer... – Wiliam May 4 '18 at 10:03
• The thing is that only knowing the marginals you cannot know how those random variables are related. You need to know their relationship via $P(X_0,X_1)$, and that thing cannot be calculated with the information you give. The only thing you can do is to assume something, for example that the variables are indeed independent, but you cannot obtain a general answer to the question with the information you give. – Josu Etxezarreta Martinez May 4 '18 at 10:10