# Shannon entropy and independent random variables

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific problem I'm referring to:

$\space$

Consider three independent random variables $u$, $v$, and $w$ with entropies $H_u$, $H_v$, $H_w$. Let,

$$X \equiv (U,\ V)$$ $$Y \equiv (V,\ W)$$

What is $H(X, Y)$, $H(X | Y)$, and $I(X; Y)$?

$\space$

Here's what I've come up with so far:

Since the random variables $u$, $v$, and $w$ are independent,

$$P(X) = P(U)P(V)$$ $$P(Y) = P(V)P(W)$$

And since $$P(X, Y) = P(X|Y)P(Y)$$ $$P(X|Y) = \frac{P(Y|X)P(U)}{P(W)}$$

But I'm not sure how to progress further from here... The solution my instructor provided for this particular problem in the textbook I'm using (Information Theory, Inference, and Learning Algorithms) is that:

$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$

$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$

And with this result the solution is fairly easy to derive.

$\space$

What I'm wondering is, where did the $x_2 = y_1$ come from, and how were the results for those probability distributions come to be? The approach I was taking was causing me to go in circles without any real results.

Thank you.

## 1 Answer

The $x_2 = y_1$ comes from the fact that $X$ and $Y$ share the component $V = X_2 = Y_1$. Thus, given $Y = (y_1,y_2) = (v,w)$ it is impossible for $X = (x_1,x_2) = (u,v)$ to have $x_2 \neq y_1$ since this would actually mean that $v \neq v$. So, the only cases where you have non-zero probability are the ones with $x_2 = y_1$. For this cases then, given $Y$ the behaviour of $X$ reduces to the behaviour of $U$ (since $V$ is indirectly given) and thus you have

$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$

For the same reasons, when $x_2 = y_1$, the event $\big (X=x=(u,v), Y=y=(v,w) \big)$ is the same as the event $(U=u, V=v, W=w)$ and thus you have

$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$

Note however that, in order to find H(X|Y), H(X,Y) and I(X;Y) you should be better of working with entropy and mutual information properties rather working directly with the probability mass functions (not that it is much harder, now that you have the above results! ;) )

• Hey, thanks so much for the tips. This information theory class I'm taking actually has a probability and statistics course as a prerequisite which I didn't know about. My lack of background has me wondering even basic concepts haha. Thanks anyway. :) – Seankala Jun 16 '18 at 3:22
• Haha you are most welcome Sean! :) Keep going, information theory is a wonderful area! – Sotiris Jul 19 '18 at 20:44