Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Suppose a Naive Bayes graphical model with binary random variables is given by $$P(y,x_1,x_2,...,x_n)=P(y)P(x_1|y)...P(x_n|y)$$

Attempting to calculate $I(x_1,...,x_n;y)$ raises the question: how can the entropy $H(P(x_1,x_2,...,x_n))$ be calculated efficiently? There must be a better way than computing an exponential number of entries.

share|cite|improve this question
I don't think there's an easy way. You will need to sum over all possible patterns. – Memming Aug 16 '13 at 14:50
@Memming - I find it strange that the effort needed is the same as when we know nothing about the marginal. Given the Naive Bayes assumption there are much less degrees of freedom to the marginal distribution. – Leo Aug 16 '13 at 21:38

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.