Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've just started learning information theory and have stumbled upon a roadblock due to my apparently not understanding how summation works. I do not understand why this is true:

$- \sum\limits_{i,j}^{} p(x_i, y_j) \log p(y_j | x_i) - \sum\limits_{i,k}^{} p(x_i, z_k) \log p(z_k | x_i) = - \sum\limits_{i,j,k}^{} p(x_i, y_j, z_k) [ \log p(y_j | x_i) + \log p(z_k | x_i) ]$

Can someone explain the justification for this equality? My book is less than forthcoming and I'm not even sure what to search for.

share|cite|improve this question
Is it the case that $p(x_i,y_j)=p(x_i,z_k)=p(x_i,y_j,z_k)$ by any chance? Or perhaps $p(x_i,y_j,z_k)$ is explicitly defined to be $\dfrac{p(x_i,y_j)\log p(y_j|x_i)+p(x_i,z_k)\log p(z_k|x_i))}{\log p(y_j|x_i)+\log p(z_k|x_i)}$? Otherwise it doesn't make much sense to me either. – Rahul Jan 26 '13 at 22:47
No, none of the probabilities have explicit definitions. I keep checking the definitions and I can't escape the feeling that I'm missing something... The claim is that $H(Y|X) + H(Z|X) = -\sum\limits_{i,j,k} p(x_i, y_j, z_k)[\log p(y_j |x_i) + \log p (z_k | x_i)]$ and from here I replace $H(Y|X)$ and $H(Z|X)$ with the definition $H(Y|X) = - \sum\limits_{i,j} p(x_i, y_j) \log p(y_i|x_i)$. – evenex_code Jan 26 '13 at 22:58
up vote 0 down vote accepted

It simply follows from the marginalization, i.e.,

$\sum_{i,j,k} p(x_i, y_j, z_k) f(x_i, y_j) = \sum_{i,j} p(x_i, y_j) f(x_i, y_j)$

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.