# Calculating conditional entropy given two random variables

I have been reading a bit about conditional entropy, joint entropy, etc but I found this: $H(X|Y,Z)$ which seems to imply the entropy associated to $X$ given $Y$ and $Z$ (although I'm not sure how to describe it). Is it the amount of uncertainty of $X$ given that I know $Y$ and $Z$? Anyway, I'd like to know how to calculate it. I thought this expression means the following:

$$H(X|Y,Z) = -\sum p(x,y,z)log_{2}p(x|y,z)$$

and assuming that $p(x|y,z)$ means $\displaystyle \frac{p(x,y,z)}{p(y)p(z)}$, then \begin{align} p(x|y,z)&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}\frac{p(x,y)}{p(y)}\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}p(x|y) \\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}\frac{p(x,z)}{p(z)}p(x|y)\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}p(x|z)p(x|y) \end{align} but that doesn't really help.

Basically I wanted to get a nice identity such as $H(X|Y)=H(X,Y)-H(Y)$ for the case of two random variables.

Any help?

Thanks

-

$$H(X\mid Y,Z)=H(X,Y,Z)-H(Y,Z)=H(X,Y,Z)-H(Y\mid Z)-H(Z)$$ Edit: Since $\log p(x\mid y,z)=\log p(x,y,z)-\log p(y,z)$, $$H(X\mid Y,Z)=-\sum\limits_{x,y,z}p(x,y,z)\log p(x,y,z)+\sum\limits_{y,z}\left(\sum\limits_{x}p(x,y,z)\right)\cdot\log p(y,z).$$ Each sum between parenthesis being $p(y,z)$, this proves the first identity above.

-
Can you show me how to manipulate $p(x|y,z)$ to get that identity? I can see why it's reasonable to get $H(X,Y,Z)-H(Y,Z)$ from $H(X|Y) = H(X,Y)-H(Y)$ but I'd like to know how to work out that from probability distributions. – Robert Smith Jul 8 '12 at 22:54
See Edit. This uses nothing but the definition, really. – Did Jul 9 '12 at 5:58

Yes, entropy is often referred to as "uncertainty", so $H(X|Y)$ can be thought of as your uncertainty about $X$, given that you know $Y$. If it's zero, then we would say that knowing $Y$ tells us "everything" about $X$, and so on.

It might be easier to think in terms of just two variables, although your basic idea is right. You can see wikipedia for more explicit calculations.

-
Thanks. Yes, I think I understand the conditional entropy correctly, however, I find it a bit awkward with two "conditional variables", though. What about my calculation? Unfortunately, Wikipedia didn't help a lot because it doesn't provide $H(X|Y,Z)$. – Robert Smith Jul 8 '12 at 19:55
@Robert: As did said, you can use the chain rule on that wikipedia page to change $H(X|Y,Z)$ into an expression involving only $H(Y|Z)$ – Xodarap Jul 8 '12 at 22:19