I am working on deriving Naive Bayes for document classification.

Each document is represented by a binary vector $x^i$ where $i=1,..,N$ for N documents. In this vector a cell is set to 1 if that cell representing a word is present at least once in the document, and left zero otherwise. Let's say there are 50,000 words hence each binary vector has 50,000 elements.

multivar bernoulli for document words

The joint distribution for Naive Bayes is

$p(x_1,...,x_{50000}) = \prod_{d=1}^{50000} p(x_d)=\prod_{d=1}^{50000} \alpha_d^{x_d}(1-\alpha_d)^{1-x_d}$

Likelihood is

$L(\theta) = \prod_{i=1}^N \prod_{d=1}^{50000} p(x_d^i) = \prod_{i=1}^N \prod_{d=1}^{50000} \alpha_d^{x_d^i}(1-\alpha_d)^{1-x_d^i}$

The text I am reading suggests maximum likelihood solution for $\alpha_d$ is $\alpha_d = \frac{N_d}{N}$, where $N_d$ is the total of '1's for a dimension (word) across all documents, and N is the total number of documents. I am guessing this is obtained by taking derivative of the likelihood function, setting the result to zero, then solve for $\alpha_d$. One trick is I guess taking log of both sides, but even then, the algebra gets hairy pretty fast. Maybe I am missing something else. I would appreciate if someone could help with this derivation.


With the loglikelihood

$$LL(\alpha_1,\ldots,\alpha_{50000}) = \sum_{i=1}^N \sum_{d=1}^{50000} {x_d^i}\log(\alpha_d)+(1-x_d^i)\log(1-\alpha_d) \; ,$$

it's pretty easy, you get the following equations to solve:

$$\sum_{i=1}^N\left(\frac{x_d^i}{\alpha_d}-\frac{1-x_d^i}{1-\alpha_d}\right) = 0$$

or after summing over $i$

$$\frac{N_d}{\alpha_d}-\frac{N-N_d}{1-\alpha_d} = 0 \; .$$

| cite | improve this answer | |
  • $\begingroup$ Hi, what happens to $\sum_{d=1}^{50000}$? Can it be dropped because we have zero on the RHS? $\endgroup$ – BBSysDyn Apr 22 '11 at 13:20
  • 1
    $\begingroup$ When you compute the partial derivatives of the loglikelihood function, you do it with respect to one $\alpha_d$, therefore all the terms which do not contain it are zero. Of course, there is one partial derivative for each $\alpha_d$ and thus just as many equations. But they all look the same. Solving one basically solves them all. $\endgroup$ – Raskolnikov Apr 22 '11 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.