The maximum likelihood estimators for expectation $\mu$ and variance $\sigma^{2}$ of a normal distribution are: $$\hat{\mu} = \frac{1}{n}\sum_{i=1}^{n} \Big(x_{i}\Big)$$ and $$\hat{\sigma^{2}} = \frac{1}{n}\sum_{i=1}^{n}\Big((\hat{\mu}-x_{i})^{2}\Big)$$ They are used so frequently in statistics that it is easy to think that they apply to all probability distributions. What would be the mean, variance (or other parameter of the distribution) estimators for some of the non-normal probability distributions?

  • $\begingroup$ The Wikipedia article for each distribution normally quotes the ML estimators for the parameters of the distribution. These can not always be given in closed form, e.g. Gamma distribution. $\endgroup$ – user121049 Nov 23 '17 at 16:19

Here is a list of Maximum Likelihood Estimators of some distributions:

                           Parameter      ML-Estimator

Binomial distribution $\qquad \qquad p \qquad \qquad \qquad \frac{X}{n}$

Poisson distribution $\qquad \qquad \lambda \qquad \qquad \qquad \overline X$

Geometric distribution $\qquad \qquad p \qquad \quad \large{ \qquad \frac{1}{\overline X+1}}$

Neg. Binomial Distribution $\qquad k,p \qquad \quad \quad $ ---

Exponential distribution $\qquad \quad \lambda \quad \qquad \qquad \frac1{\overline X}$

Gamma distribution $\qquad \qquad k,\lambda \quad \qquad \qquad $---

Uniform distribution $\qquad \quad N \ \text{or} \ \theta \quad \quad \qquad X_{n}=Max \ X_i $

Source: $\small{\text{Einführung in die Statistik: Analyse und Modellierung von Daten (9th Edition), Author: Rainer Schlittgen}}$,$\small{\text{p. 295}}$

It can be read off that not for all listed distributions a MLE can be determined.

  • $\begingroup$ Interesting list (+1). However, I wonder if a dash means 'no MLE' or 'no MLE in closed form'. For example, computational methods are needed to find MLEs for some gamma distributions. $\endgroup$ – BruceET Nov 23 '17 at 17:47
  • $\begingroup$ @BruceET Thanks. Good question. The book says that it exists no MLE. In such cases the MLE values can only be evaluated by numerical methods for only specific samples. But it is not mentioned how it would work. The book is a little bit squishy at this point. $\endgroup$ – callculus Nov 23 '17 at 18:05

Suppose $X(t)$ is a random variable depending on a parameter $t$ with probability or probability density function $f(t, x)$ (I will only refer to pdfs from now on but the beginning of the argument applies to the discrete case as well). Then the pdf of $n$ independent samples of $X(t)$ is given by $\prod_{i=1}^n f(t, x_i)$, so the maximum likelihood value of $t$ given samples $x_i$ is

$$\text{argmax}_t \prod_{i=1}^n f(t, x_i) = \text{argmax}_t \sum_{i=1}^n \log f(t, x_i).$$

This is as much as we can say with no further assumptions on the form of $f$. However, if we make the useful simplifying assumption that $f$ is an exponential family, so that $f(t, x_i)$ is proportional to $\exp (\eta(t) T(x_i) - A(t))$ for some functions $\eta, T, A$, then the argmax becomes

$$\text{argmax}_t \sum_{i=1}^n \left( \eta(t) T(x_i) - A(t) \right) = \text{argmax}_t \, \eta(t) \left( \frac{1}{n} \sum_{i=1}^n T(x_i) \right) - A(t).$$

Note that now the samples $x_i$ only affect this estimate via the sample average $\frac{1}{n} \sum_{i=1}^n T(x_i)$, which has become a sufficient statistic for $t$. For example, if $t$ is the mean of a normal distribution with known variance we have $T(x_i) = x_i$, and if $t$ is the variance of a normal distribution with known mean $\mu$ we have $T(x_i) = (x_i - \mu)^2$. (This discussion generalizes to the case that $t$ is multivariate but I want to stick to one variable for simplicity.)

Assuming that $\eta$ and $A$ are smooth, differentiating with respect to $t$ now gives

$$\boxed{ \eta'(t_{MLE}) \left( \frac{1}{n} \sum_{i=1}^n T(x_i) \right) = A'(t_{MLE}) }.$$

So the MLE of $t$ is given by applying the inverse of $\frac{A'(t)}{\eta'(t)}$, assuming that exists, to the sample average $\frac{1}{n} \sum_{i=1}^n T(x_i)$.

Example. For a normal distribution with unknown mean $t = \mu$ and known variance $\sigma^2$ the density is proportional to $\exp \left( - \frac{(x - \mu)^2}{2 \sigma^2} \right)$, which gives $A(\mu) = \frac{\mu^2}{2 \sigma^2}$, $\eta(\mu) = \frac{\mu}{\sigma^2}$, and $T(x) = x$. Applying the boxed equation then gives

$$\frac{1}{\sigma^2} \left( \frac{1}{n} \sum_{i=1}^n x_i \right) = \frac{\mu_{MLE}}{\sigma^2}.$$

Happily the $\sigma$s cancel and we get $\mu_{MLE} = \frac{1}{n} \sum_{i=1}^n x_i$ as expected.

Example. Consider a lognormal distribution whose logarithm is normally distributed with unknown mean $t = \mu$ and known variance $\sigma^2$. Then everything is the same as in the normal case except that we have $T(x) = \log x$. Hence we find that

$$\mu_{MLE} = \frac{1}{n} \sum_{i=1}^n \log x_i.$$

Taking exponentials we get that $\exp (\mu_{MLE})$ is given by the geometric mean of the samples $x_i$ (note that $\exp(\mu)$ is not the mean of the lognormal distribution, but the geometric mean). In this case the arithmetic mean of the samples will be strongly biased upward, which the geometric mean corrects for.

You can compute other examples for yourself using the table of exponential family distributions given in the linked Wikipedia article.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.