I understand the concept behind finding a Maximum Likelihood Estimator, but when I'm setting up the likelihood function, I'm having trouble understanding if I start with a summation to create the joint probability function, or if I start with a product.

I thought it depended on the original distribution - perhaps product for continuous random variables and summation for discrete. But then I saw in my notes we started with a product when finding the MLE for p in the binomial distribution, and used a summation for finding the MLE for $\theta$ in the exponential distribution.

Is there a guideline?

  • $\begingroup$ Or is it always the product and the goal is to manipulate it into a summation, using natural logs and laws of exponents usually? $\endgroup$ – Carolyn Oct 25 '16 at 2:44
  • 1
    $\begingroup$ You always start with the product - but due to some nice properties of exponents (especially for nice functions such as exponential family etc) you usually end up with a summation (assuming I understand your question). $\endgroup$ – Chinny84 Oct 25 '16 at 2:48
  • $\begingroup$ @Chinny84 thank you! I thought a little more into it $\endgroup$ – Carolyn Oct 25 '16 at 2:53
  • $\begingroup$ no worries! it is great you take the time to think about the technique instead of following it blindly - and in the real world this can cause a lot of problems. $\endgroup$ – Chinny84 Oct 25 '16 at 12:20
  • $\begingroup$ It would be better to describe a simple case you're having trouble with and we'll try to assist you. $\endgroup$ – Royi Jul 28 '17 at 9:21

If the observations are independent, then the joint density is the product of the marginal densities for distributions with densities (similarly for discrete random variables, the joint probability is the product of the marginal probabilities). This product can be taken as (proportional to) the likelihood you wish to maximise

You then have a choice:

  • maximise the product
  • take the logarithm of the product, turning it into a sum of logarithms, and then maximise the sum of the logarithms

It makes little theoretical difference which you use, since logarithms are continuous strictly increasing functions. Taking the derivative of the product gives $\frac{d}{dx} \prod_i f_i(x) = \left(\sum_i \frac{f_i^\prime(x)}{f_i(x)}\right)\left( \prod_j f_j(x)\right)$ while the derivative of the sums of the logarithms gives $\frac{d}{dx} \sum_i \log_e(f_i(x)) = \sum_i \frac{f_i^\prime(x)}{f_i(x)}$, and these will have the same zeros if the likelihood is positive. If the individual likelihoods involve multiplication, powers and exponentiation, then their logarithms may sometimes be easier to handle

TLDR: maximising the sum of the log-likelihoods gives the same result as maximising the product of likelihoods, but might be easier to manipulate in some cases


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.