Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have been reading a bit about conditional entropy, joint entropy, etc but I found this: $H(X|Y,Z)$ which seems to imply the entropy associated to $X$ given $Y$ and $Z$ (although I'm not sure how to describe it). Is it the amount of uncertainty of $X$ given that I know $Y$ and $Z$? Anyway, I'd like to know how to calculate it. I thought this expression means the following:

$$H(X|Y,Z) = -\sum p(x,y,z)log_{2}p(x|y,z)$$

and assuming that $p(x|y,z)$ means $\displaystyle \frac{p(x,y,z)}{p(y)p(z)}$, then \begin{align} p(x|y,z)&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}\frac{p(x,y)}{p(y)}\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(z)}p(x|y) \\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}\frac{p(x,z)}{p(z)}p(x|y)\\&=\displaystyle \frac{p(x,y,z)}{p(x,y)p(x,z)}p(x|z)p(x|y) \end{align} but that doesn't really help.

Basically I wanted to get a nice identity such as $H(X|Y)=H(X,Y)-H(Y)$ for the case of two random variables.

Any help?


share|cite|improve this question
up vote 3 down vote accepted

$$H(X\mid Y,Z)=H(X,Y,Z)-H(Y,Z)=H(X,Y,Z)-H(Y\mid Z)-H(Z)$$ Edit: Since $\log p(x\mid y,z)=\log p(x,y,z)-\log p(y,z)$, $$ H(X\mid Y,Z)=-\sum\limits_{x,y,z}p(x,y,z)\log p(x,y,z)+\sum\limits_{y,z}\left(\sum\limits_{x}p(x,y,z)\right)\cdot\log p(y,z). $$ Each sum between parenthesis being $p(y,z)$, this proves the first identity above.

share|cite|improve this answer
Can you show me how to manipulate $p(x|y,z)$ to get that identity? I can see why it's reasonable to get $H(X,Y,Z)-H(Y,Z)$ from $H(X|Y) = H(X,Y)-H(Y)$ but I'd like to know how to work out that from probability distributions. – Robert Smith Jul 8 '12 at 22:54
See Edit. This uses nothing but the definition, really. – Did Jul 9 '12 at 5:58

Yes, entropy is often referred to as "uncertainty", so $H(X|Y)$ can be thought of as your uncertainty about $X$, given that you know $Y$. If it's zero, then we would say that knowing $Y$ tells us "everything" about $X$, and so on.

It might be easier to think in terms of just two variables, although your basic idea is right. You can see wikipedia for more explicit calculations.

share|cite|improve this answer
Thanks. Yes, I think I understand the conditional entropy correctly, however, I find it a bit awkward with two "conditional variables", though. What about my calculation? Unfortunately, Wikipedia didn't help a lot because it doesn't provide $H(X|Y,Z)$. – Robert Smith Jul 8 '12 at 19:55
@Robert: As did said, you can use the chain rule on that wikipedia page to change $H(X|Y,Z)$ into an expression involving only $H(Y|Z)$ – Xodarap Jul 8 '12 at 22:19

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.