Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Let $\mathcal{D}=\{ (y_1,x_1), ..., (y_N,x_N) \}$ be i.i.d. training data. Let $\widetilde P(Y,X)$ be the empirical distribution. The goal is to maximize the likelihood of $P(\mathcal{D}|\;\Theta)$.

I am having difficulties with this line: $P(\mathcal{D}|\Theta) = \prod_{y,x} P(Y=y|\;X=x; \Theta)^{\widetilde P(Y=y,X=x)}$

Specifically, it is the exponent ${\widetilde P(Y=y,X=x)}$ which I do not understand why it is there.

Please help me understand why this is so or point me to some reference which could help me understand it.

One line of thought I had, is that perhaps it is the product over all possible $y,x$ (not only those in the training sample), then I would understand if it was

$P(\mathcal{D}|\Theta) = \prod_{y,x} P(Y=y|\;X=x; \Theta)^{\widetilde P(Y=y,X=x)|\mathcal{D}|}$

that would work if each $y,x$ occured in $\mathcal{D}$ and some occured more than once. But then again if not each $y,x$ is in $\mathcal{D}$, then for those the exponent would be 0 and thus the product would include the probability of that event despite it not being in the training data. Needless to say, I am lost. It's probably due to some (fundamental) misunderstanding on my side. Thankful for any help.

share|improve this question
1  
If the exponent is $0$, that factor in the product is $1$, so it makes no difference whether the product runs over all $x,y$ or only those in the training set. Regarding the factor of $|\mathcal D|$, I agree that it should be there. However, note that it doesn't change the maximization problem, since you can take it outside the product and it just becomes a constant factor multiplying the log likelihood. –  joriki Jan 9 '12 at 23:51
    
Thanks! I can't recall how I was thinking regarding the case when the exponent is 0 (indeed a silly mistake to make.) –  j-a Jan 10 '12 at 5:56
    
@joriki, I think your comment is an answer to this question, if you would make it an answer I would accept it. –  j-a Jan 10 '12 at 6:03
1  
You're welcome. I almost made this an answer originally; I can't recall why I didn't ;-) –  joriki Jan 10 '12 at 6:05

1 Answer 1

up vote 1 down vote accepted

If the exponent is $0$, that factor in the product is $1$, so it makes no difference whether the product runs over all $x,y$ or only those in the training set. Regarding the factor of $|\mathcal D|$, I agree that it should be there. However, note that it doesn't change the maximization problem, since you can take it outside the product and it just becomes a constant factor multiplying the log likelihood.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.