# Shannon entropy and independent random variables

I had a question regarding a question on Shannon entropy I came across. It has to do with representing entropy in the form of their probability distributions, but let me elaborate. Here's the specific problem I'm referring to:

$\space$

Consider three independent random variables $u$, $v$, and $w$ with entropies $H_u$, $H_v$, $H_w$. Let,

$$X \equiv (U,\ V)$$ $$Y \equiv (V,\ W)$$

What is $H(X, Y)$, $H(X | Y)$, and $I(X; Y)$?

$\space$

Here's what I've come up with so far:

Since the random variables $u$, $v$, and $w$ are independent,

$$P(X) = P(U)P(V)$$ $$P(Y) = P(V)P(W)$$

And since $$P(X, Y) = P(X|Y)P(Y)$$ $$P(X|Y) = \frac{P(Y|X)P(U)}{P(W)}$$

But I'm not sure how to progress further from here... The solution my instructor provided for this particular problem in the textbook I'm using (Information Theory, Inference, and Learning Algorithms) is that:

$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$

$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$

And with this result the solution is fairly easy to derive.

$\space$

What I'm wondering is, where did the $x_2 = y_1$ come from, and how were the results for those probability distributions come to be? The approach I was taking was causing me to go in circles without any real results.

Thank you.

The $x_2 = y_1$ comes from the fact that $X$ and $Y$ share the component $V = X_2 = Y_1$. Thus, given $Y = (y_1,y_2) = (v,w)$ it is impossible for $X = (x_1,x_2) = (u,v)$ to have $x_2 \neq y_1$ since this would actually mean that $v \neq v$. So, the only cases where you have non-zero probability are the ones with $x_2 = y_1$. For this cases then, given $Y$ the behaviour of $X$ reduces to the behaviour of $U$ (since $V$ is indirectly given) and thus you have
$$P(X|Y) = \left\{ \begin{array}{c} P(U)\ (x_2 = y_1) \\ 0\ (else) \end{array} \right.$$
For the same reasons, when $x_2 = y_1$, the event $\big (X=x=(u,v), Y=y=(v,w) \big)$ is the same as the event $(U=u, V=v, W=w)$ and thus you have
$$P(X, Y) = \left\{\begin{array}{c} P(U)P(V)P(W)\ (x_2 = y_1) \\ 0\ (else) \end{array}\right.$$