Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When someone writes $H(X_1, X_2, X_3) = H(X_1) + H(X_2\mid X_1) + H(X_3\mid X_2, X_1)$, how should that last term be interpreted/read? As the joint entropy between 2 variables where variable 1 is $X_3\mid X_2$ and variable 2 is $X_1$? Or As the entropy of $X_3$ conditioned on both $X_2$ and $X_1$? In other words is it:

$$H[ (X_3\mid X_2) , (X_1) ]\text{ or }H[ (X_3) \mid (X_2, X_1) ]$$

Are they the same? If so could someone show me how? If not, could someone tell me the correct way to read/interpret/write both possible interpretations?

share|cite|improve this question
Entropy of $X_3$ conditioned on both $X_1,X_2$. – oldrinb May 29 '13 at 1:33
up vote 2 down vote accepted

The intended interpretation in this formula is $H(X_3\mid (X_2,X_1))$. Your other option won't work because $X_3\mid X_2$ is not a random variable.

share|cite|improve this answer
why is it not a RV in some smaller belief space? – user79950 May 29 '13 at 1:06
Don't confuse $H(X|Y)$ with $H(X|Y=y)$, they are different things (the confusion can arise because when speaking of mere conditional probabilities $P(X|Y)$ and $P(X|Y=y)$ are basically the same). When one writes, for example, $E(X|Y)$ or $\sigma_{X|Y}^2$, the result is a function of $Y$ ; but $H(X|Y)$ is not such a thing. – leonbloy May 30 '13 at 17:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.