Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am working on deriving Naive Bayes for document classification.

Each document is represented by a binary vector $x^i$ where $i=1,..,N$ for N documents. In this vector a cell is set to 1 if that cell representing a word is present at least once in the document, and left zero otherwise. Let's say there are 50,000 words hence each binary vector has 50,000 elements.

multivar bernoulli for document words

The joint distribution for Naive Bayes is

$p(x_1,...,x_{50000}) = \prod_{d=1}^{50000} p(x_d)=\prod_{d=1}^{50000} \alpha_d^{x_d}(1-\alpha_d)^{1-x_d}$

Likelihood is

$L(\theta) = \prod_{i=1}^N \prod_{d=1}^{50000} p(x_d^i) = \prod_{i=1}^N \prod_{d=1}^{50000} \alpha_d^{x_d^i}(1-\alpha_d)^{1-x_d^i}$

The text I am reading suggests maximum likelihood solution for $\alpha_d$ is $\alpha_d = \frac{N_d}{N}$, where $N_d$ is the total of '1's for a dimension (word) across all documents, and N is the total number of documents. I am guessing this is obtained by taking derivative of the likelihood function, setting the result to zero, then solve for $\alpha_d$. One trick is I guess taking log of both sides, but even then, the algebra gets hairy pretty fast. Maybe I am missing something else. I would appreciate if someone could help with this derivation.

share|cite|improve this question
up vote 1 down vote accepted

With the loglikelihood

$$LL(\alpha_1,\ldots,\alpha_{50000}) = \sum_{i=1}^N \sum_{d=1}^{50000} {x_d^i}\log(\alpha_d)+(1-x_d^i)\log(1-\alpha_d) \; ,$$

it's pretty easy, you get the following equations to solve:

$$\sum_{i=1}^N\left(\frac{x_d^i}{\alpha_d}-\frac{1-x_d^i}{1-\alpha_d}\right) = 0$$

or after summing over $i$

$$\frac{N_d}{\alpha_d}-\frac{N-N_d}{1-\alpha_d} = 0 \; .$$

share|cite|improve this answer
Hi, what happens to $\sum_{d=1}^{50000}$? Can it be dropped because we have zero on the RHS? – BB_ML Apr 22 '11 at 13:20
When you compute the partial derivatives of the loglikelihood function, you do it with respect to one $\alpha_d$, therefore all the terms which do not contain it are zero. Of course, there is one partial derivative for each $\alpha_d$ and thus just as many equations. But they all look the same. Solving one basically solves them all. – Raskolnikov Apr 22 '11 at 13:31
i see.. thanks. – BB_ML Apr 22 '11 at 13:33

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.