3
$\begingroup$

In the statement for estimating parameters through the Maximum Likelihood Principle (MLE), there is no mention of whether to choose a local maximum or a global maximum. (In my very limited reading so far) From the examples given in various textbooks/lecture notes, it seems that we should choose the global maximum of the likelihood function for inference. Is this correct?

The reason I am asking is because I am dealing with some data whose likelihood seems to have several maxima. The parameter space is three dimensional, so I have no intuition about the situation. In this case how do I estimate the parameters properly - do I just look for the maximum in a small part of the parameter space? (The bounds could be established through guesses based on the data, for example.)

$\endgroup$
1
$\begingroup$

Many, but not all, likelihood functions we usually encounter have strictly convex logarithm (i.e., they're log-concave). Consequently, they have a unique stationary point and that is the global maximum. This doesn't mean that there might be cases where the likelihood has multiple local maxima. You always look for the global maximum in MLE. Keep in mind, however, that MLE is not necessarily a good estimator for all problems and there are common and interesting cases where MLE may produce an estimate with large error.

$\endgroup$
4
  • $\begingroup$ Can you tell me about the cases where MLE produces estimates with large errors? Is there a unifying feature for these? $\endgroup$ – Comp_Warrior Jul 6 '13 at 20:53
  • 1
    $\begingroup$ Check this and this for instance. $\endgroup$ – S.B. Jul 6 '13 at 21:20
  • $\begingroup$ Also, in some special cases like logistic or probit regression, at finite samples the log-likelihood could be very flat near the true parameter specially when data is not that noisy. This could lead either to the (global) maximum being non-unique or very far from the true optimum. You might only get accuracy in terms of the value of likelihood, but not necessarily the parameter accuracy. $\endgroup$ – S.B. Jul 6 '13 at 21:28
  • $\begingroup$ "we usually encounter have strictly convex logarithm" It turns out that the likelihood for the normal distribution, where both $\mu$ and $\sigma^2$ are unknown is not concave, neither is its $\log$... $\endgroup$ – Gabriel Romon Apr 14 '17 at 16:34
1
$\begingroup$

In general, working with MLE depends on the case under consideration.

Things with MLE can go exceptionally good (in presence of concave maximum likelihood function, for example) or very bad.

In this latter case, if you consider logistic regressions in presence of complete separation, then you can prove that the MLE will provide you with a solution which diverges. More precisely, the maximum of the maximum likelihood function lies at the boundary of the $n$-cone generated by the parameters. An additional problem in this case is represented by the fact that statistical softwares will usually provide you with a finite solution and a worning/error message. In other words, you need to work on the error/warning message to understand the divergence of the Newton Raphson algorithm.

$\endgroup$
1
$\begingroup$

Maximum likelihood estimation is based on the principle that you want to maximize the likelihoodfunction, i.e. the function that represents the likelihood of the data that are observed.

As you know, the likelihood function is defined as $$\mathcal{L}(\theta)=\prod\limits_{i=1}^n f(x_i; \theta)$$. So this is basically the product of every pdf where the observed value of $X$ is incorporated. This means, that if this function is maximized, that then it is very likely that, given the value of theta that you find after maximization, these observations are true (and, as they are in fact real data, this is what you want). Hence, the value that you find after maximizing is the value of theta that you will want to use as an estimator.

Therefore, what you are after is a global maximum as all that you want is your likelihood function to me maximized; it could very well happen that this occurs at a point which is not a local maximum. Consider for example the uniform distribution on $[0, \theta]$; with pdf $\frac{1}{\theta}$ for $X$ between $0$ and $\theta$. This function does not have a local maximum. Hence we look at the boundaries of the interval to find the global maximum and you would see that $x_{n:n}$ (the largest observed value of $X$) would be come our estimator. So this example shows that you can find an estimator for theta at a point which is not a local maximum.

$\endgroup$
1
$\begingroup$

The reason I am asking is because I am dealing with some data whose likelihood seems to have several maxima.

If you have several interior maxima it might be worth documenting what you are doing as that is an atypical case. In the previously mentioned degenerate cases of logistic regression the MLE runs off to infinity (or to the boundary, if you restrict the coefficients), but this is not really a breakdown of the MLE global maximization paradigm, it is from having multiple models that perfectly fit the data.

$\endgroup$
2
  • $\begingroup$ I'm working through a self gathered data set (Signal strength was found at different locations, i.e. distance from an access point). The procedure I am following is the hyperparameter estimation as presented by Ferris et al. in the paper "Gaussian Processes for Signal Strength-Based Location Estimation". When I use a gradient descent method to locate the maximum, I get huge values for the parameters which seems wrong. On the other hand when I performed a brute force approach over a small region of the parameter space, I get another, more reasonable (local?) maximum. $\endgroup$ – Comp_Warrior Jul 6 '13 at 21:00
  • 1
    $\begingroup$ There are situations such as good fit with a nearly zero covariance matrix, where the length-scale parameter $\ell$ in the paper will appear to be effectively infinite. I don't know if this is from the MLE going off to infinity or only the numerical attempts to locate it using particular methods (without limiting to a part of the parameter space). If this is Gaussian process regression then finally you should have a unique and well behaved solution by MLE, but that's assuming the covariance matrix is estimated accurately enough to be seen to be positive definite. $\endgroup$ – zyx Jul 7 '13 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.