2
$\begingroup$

I have been reading a bit about conditional entropy, joint entropy, etc but I found this: $H(X|Y,Z)$ which seems to imply the entropy associated to $X$ given $Y$ and $Z$ (although I'm not sure how to describe it). Is it the amount of uncertainty of $X$ given that I know $Y$ and $Z$? Anyway, I'd like to know how to calculate it. I thought this expression means the following:

$$H(X|Y,Z) = -\sum p(x,y,z)log_{2}p(x|y,z)$$

and assuming that $p(x|y,z)$ means $\displaystyle \frac{p(x,y,z)}{p(y)p(z)}$, then \begin{align} p(x|y,z)&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}\frac{p(x,y)}{p(y)}\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}p(x|y) \\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}\frac{p(x,z)}{p(z)}p(x|y)\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}p(x|z)p(x|y) \end{align} but that doesn't really help.

Basically I wanted to get a nice identity such as $H(X|Y)=H(X,Y)-H(Y)$ for the case of two random variables.

Any help?

Thanks

$\endgroup$
4
$\begingroup$

$$H(X\mid Y,Z)=H(X,Y,Z)-H(Y,Z)=H(X,Y,Z)-H(Y\mid Z)-H(Z)$$ Edit: Since $\log p(x\mid y,z)=\log p(x,y,z)-\log p(y,z)$, $$ H(X\mid Y,Z)=-\sum\limits_{x,y,z}p(x,y,z)\log p(x,y,z)+\sum\limits_{y,z}\left(\sum\limits_{x}p(x,y,z)\right)\cdot\log p(y,z). $$ Each sum between parenthesis being $p(y,z)$, this proves the first identity above.

$\endgroup$
  • $\begingroup$ Can you show me how to manipulate $p(x|y,z)$ to get that identity? I can see why it's reasonable to get $H(X,Y,Z)-H(Y,Z)$ from $H(X|Y) = H(X,Y)-H(Y)$ but I'd like to know how to work out that from probability distributions. $\endgroup$ – Robert Smith Jul 8 '12 at 22:54
  • $\begingroup$ See Edit. This uses nothing but the definition, really. $\endgroup$ – Did Jul 9 '12 at 5:58
1
$\begingroup$

Yes, entropy is often referred to as "uncertainty", so $H(X|Y)$ can be thought of as your uncertainty about $X$, given that you know $Y$. If it's zero, then we would say that knowing $Y$ tells us "everything" about $X$, and so on.

It might be easier to think in terms of just two variables, although your basic idea is right. You can see wikipedia for more explicit calculations.

$\endgroup$
  • $\begingroup$ Thanks. Yes, I think I understand the conditional entropy correctly, however, I find it a bit awkward with two "conditional variables", though. What about my calculation? Unfortunately, Wikipedia didn't help a lot because it doesn't provide $H(X|Y,Z)$. $\endgroup$ – Robert Smith Jul 8 '12 at 19:55
  • $\begingroup$ @Robert: As did said, you can use the chain rule on that wikipedia page to change $H(X|Y,Z)$ into an expression involving only $H(Y|Z)$ $\endgroup$ – Xodarap Jul 8 '12 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.