Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've just started learning information theory and have stumbled upon a roadblock due to my apparently not understanding how summation works. I do not understand why this is true:

$- \sum\limits_{i,j}^{} p(x_i, y_j) \log p(y_j | x_i) - \sum\limits_{i,k}^{} p(x_i, z_k) \log p(z_k | x_i) = - \sum\limits_{i,j,k}^{} p(x_i, y_j, z_k) [ \log p(y_j | x_i) + \log p(z_k | x_i) ]$

Can someone explain the justification for this equality? My book is less than forthcoming and I'm not even sure what to search for.

share|improve this question
    
Is it the case that $p(x_i,y_j)=p(x_i,z_k)=p(x_i,y_j,z_k)$ by any chance? Or perhaps $p(x_i,y_j,z_k)$ is explicitly defined to be $\dfrac{p(x_i,y_j)\log p(y_j|x_i)+p(x_i,z_k)\log p(z_k|x_i))}{\log p(y_j|x_i)+\log p(z_k|x_i)}$? Otherwise it doesn't make much sense to me either. –  Rahul Jan 26 '13 at 22:47
    
No, none of the probabilities have explicit definitions. I keep checking the definitions and I can't escape the feeling that I'm missing something... The claim is that $H(Y|X) + H(Z|X) = -\sum\limits_{i,j,k} p(x_i, y_j, z_k)[\log p(y_j |x_i) + \log p (z_k | x_i)]$ and from here I replace $H(Y|X)$ and $H(Z|X)$ with the definition $H(Y|X) = - \sum\limits_{i,j} p(x_i, y_j) \log p(y_i|x_i)$. –  evenex_code Jan 26 '13 at 22:58
add comment

1 Answer 1

up vote 0 down vote accepted

It simply follows from the marginalization, i.e.,

$\sum_{i,j,k} p(x_i, y_j, z_k) f(x_i, y_j) = \sum_{i,j} p(x_i, y_j) f(x_i, y_j)$

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.