Maximum Likelihood Estimate of the Uniform Distribution?

Question

I am trying to find the Maximum Likelihood estimate of the $a$ parameter in the uniform distribution.

For a quick background, so far I am familiar with Bayesian models which use the prior probability times the likelihood of the data set, normalized, which will give the posterior distribution. In the past, I've seen the following:

$p(\theta)$ is the prior probability distribution

$p(D\mid\theta)$ is the likelihood of the training data set $D$

$p(\theta\mid D)$ is the posterior distribution which is proportional to $[p(\theta) \times p(D\mid\theta)]$

And we use the Beta distribution for the prior, and the Binomial distribution for the likelihood (which is convenient because we use the log likelihood to find the MLE).

But now, we are being asked to find the maximum likelihood of the Uniform distribution.

The distribution formula is: $ \operatorname{Unif}(x\mid a,b) = \frac{1}{2a}I(x \in [-a, a])$

where $I(\text{true}) = 1$ and $I(\text{false}) = 0$

Our data set is $D=\{x_1, \ldots , x_n\}$

My question is this: how do we find the MLE of the parameter $a$?

Thanks in advance

Answer 1

The density for each observation is $\displaystyle f_{X_i}(x) = \begin{cases} 1/2a & \text{if } -a\le x\le a, \\ 0 & \text{if } x<-a \text{ or } x>a. \end{cases}$

You didn't say your observations were independent but I will assume that was intended. The joint density is therefore $$ f_{X_1,\ldots,X_n} (x_1,\ldots,x_n) = \begin{cases} 1/(2a)^n & \text{if for every } i\in\{1,\ldots,n\} \text{ we have } -a\le x_i \le a, \\ 0 & \text{otherwise}. \end{cases} $$

The condition that for every $i\in\{1,\ldots,n\}$ we have $-a\le x_i\le a$ is the same as $\min\{x_1,\ldots,x_n\} \ge -a$ and $\max\{x_1,\ldots,x_n\}\le a.$ That condition on $\min$ is the same as $-\min\{x_1,\ldots,x_n\} \le a.$ So we need $a\ge\max$ and $a\ge -\min.$ I leave it as an exercise to show that $$ \Big( a\ge \max\{x_1,\ldots,x_n\} \text{ and } a\le -\min\{x_1,\ldots,x_n\} \Big) \text{ if and only if } a \ge \max\{|x_1|,\ldots,|x_n|\}. $$

Therefore the likelihood function is $$ L(a) = \begin{cases} 1/(2a)^n & \text{if } a \ge \max\{|x_1|,\ldots,|x_n|\}, \\ 0 & \text{otherwise.} \end{cases} $$ Now notice that $L(a)$ increases as $a$ decreases, until $a$ gets down to $\max\{|x_1|,\ldots,|x_n|\}.$ Therefore that maximum is the MLE.

Answer 2

The maximum-likelihood estimate of the maximum and minimum values in a data set ${\cal D}$ drawn from a Uniform Distribution are the maximum and minimum values in the data set.

If the distribution is indeed centered on $0$, then the maximum-likelihood estimate of $a$ is $\max |x|$ over all $x \in {\cal D}$ (for large data set).

Let the data set be $\{ x_1, x_2, \ldots x_n \}$. For a given $a>0$, the probability of a single point $x_i$ is ${1 \over 2 a}$ if $-a < x_i < a$ and $0$ otherwise. The log-likelihood is a function of the single variable, $a$:

${\cal L}({\cal D}; a) = \sum\limits_{i=1}^n \log \left( {1 \over 2 a} \right) = n \log \left( {1 \over 2 a} \right)$.

You wish to maximize this, which means you want to minimize $a$ (make the distribution as tight as possible, while capturing all the points).

You can complete the simple arithmetic underlying those statements by taking derivatives...

The reason we generally construct functions and take derivatives (such as this case) is because of the natural extension to the more natural, more powerful, and more realistic case of Bayesian estimation. In statistical estimation we generally employ prior information over the parameters. In the current problem you acknowledge that the problem could incorporate prior information, but you state no such prior information about $a$. But really, is it possible that $a = 10^9$ ahead of time? Unlikely.

So in full Bayesian estimation, you incorporate the prior information about the variable. So one potential prior over $a$ in the current problem would be $P(a) = 1/20$ for $a<0<20$... that is, a uniform prior over $a$. Then, the likelihood of $a$ is proportional to the product of its prior probability and the probabilities given the data set. Suppose you have a small data set whose highest data point is $x_i = 18$. In maximum-likelihood estimation, you'd get $a=18$ (as we've seen), and the derived distribution would be uniform between $-18$ and $+ 18$. But in the more-principled Bayesian learning, there would still be some contribution from your prior information that $a$ might be as high as 20.

When you go through all the math (which can, but need not, use derivatives and such...), your distribution will be uniform between -18 and +18 but still extend (at lower value) between -20 and +20.

In the limit of very large data sets, the prior information will be swamped by the data and you'll get a uniform distribution between $-\max |x_i |$ and $+\max | x_i |$, but in the more interesting case of small $n$, the information from the prior knowledge about $a$ will have an effect.