1
$\begingroup$

I am trying to find the Maximum Likelihood estimate of the $a$ parameter in the uniform distribution.

For a quick background, so far I am familiar with Bayesian models which use the prior probability times the likelihood of the data set, normalized, which will give the posterior distribution. In the past, I've seen the following:

$p(\theta)$ is the prior probability distribution

$p(D\mid\theta)$ is the likelihood of the training data set $D$

$p(\theta\mid D)$ is the posterior distribution which is proportional to $[p(\theta) \times p(D\mid\theta)]$

And we use the Beta distribution for the prior, and the Binomial distribution for the likelihood (which is convenient because we use the log likelihood to find the MLE).

But now, we are being asked to find the maximum likelihood of the Uniform distribution.

The distribution formula is: $ \operatorname{Unif}(x\mid a,b) = \frac{1}{2a}I(x \in [-a, a])$

where $I(\text{true}) = 1$ and $I(\text{false}) = 0$

Our data set is $D=\{x_1, \ldots , x_n\}$

My question is this: how do we find the MLE of the parameter $a$?

Thanks in advance

$\endgroup$
1
$\begingroup$

The maximum-likelihood estimate of the maximum and minimum values in a data set ${\cal D}$ drawn from a Uniform Distribution are the maximum and minimum values in the data set.

If the distribution is indeed centered on $0$, then the maximum-likelihood estimate of $a$ is $\max |x|$ over all $x \in {\cal D}$ (for large data set).

Let the data set be $\{ x_1, x_2, \ldots x_n \}$. For a given $a>0$, the probability of a single point $x_i$ is ${1 \over 2 a}$ if $-a < x_i < a$ and $0$ otherwise. The log-likelihood is a function of the single variable, $a$:

${\cal L}({\cal D}; a) = \sum\limits_{i=1}^n \log \left( {1 \over 2 a} \right) = n \log \left( {1 \over 2 a} \right)$.

You wish to maximize this, which means you want to minimize $a$ (make the distribution as tight as possible, while capturing all the points).

You can complete the simple arithmetic underlying those statements by taking derivatives...

The reason we generally construct functions and take derivatives (such as this case) is because of the natural extension to the more natural, more powerful, and more realistic case of Bayesian estimation. In statistical estimation we generally employ prior information over the parameters. In the current problem you acknowledge that the problem could incorporate prior information, but you state no such prior information about $a$. But really, is it possible that $a = 10^9$ ahead of time? Unlikely.

So in full Bayesian estimation, you incorporate the prior information about the variable. So one potential prior over $a$ in the current problem would be $P(a) = 1/20$ for $a<0<20$... that is, a uniform prior over $a$. Then, the likelihood of $a$ is proportional to the product of its prior probability and the probabilities given the data set. Suppose you have a small data set whose highest data point is $x_i = 18$. In maximum-likelihood estimation, you'd get $a=18$ (as we've seen), and the derived distribution would be uniform between $-18$ and $+ 18$. But in the more-principled Bayesian learning, there would still be some contribution from your prior information that $a$ might be as high as 20.

When you go through all the math (which can, but need not, use derivatives and such...), your distribution will be uniform between -18 and +18 but still extend (at lower value) between -20 and +20.

In the limit of very large data sets, the prior information will be swamped by the data and you'll get a uniform distribution between $-\max |x_i |$ and $+\max | x_i |$, but in the more interesting case of small $n$, the information from the prior knowledge about $a$ will have an effect.

$\endgroup$
  • $\begingroup$ Okay, but how do we work this out on paper (how do we prove it)? $\endgroup$ – Josh Jul 20 '17 at 21:32
  • $\begingroup$ ... and, the uniform distribution we were given is always centered at zero, so the max and min are linked to each other (a is the max, and -a is the min) $\endgroup$ – Josh Jul 20 '17 at 21:34
  • 1
    $\begingroup$ @Josh he means the maximum and minimum of the data set $D$, not of the interval $[-a,a]$. $\endgroup$ – angryavian Jul 20 '17 at 21:36
  • $\begingroup$ Ok, that makes sense, but how do you prove this (how do you work out whether a will be either the max of the dataset D or the min?) $\endgroup$ – Josh Jul 20 '17 at 21:37
  • 1
    $\begingroup$ @MichaelHardy you said you would "qualify" my first statement - which one was that? The first comment in this answer? $\endgroup$ – Josh Jul 20 '17 at 21:41
2
$\begingroup$

The density for each observation is $\displaystyle f_{X_i}(x) = \begin{cases} 1/2a & \text{if } -a\le x\le a, \\ 0 & \text{if } x<-a \text{ or } x>a. \end{cases}$

You didn't say your observations were independent but I will assume that was intended. The joint density is therefore $$ f_{X_1,\ldots,X_n} (x_1,\ldots,x_n) = \begin{cases} 1/(2a)^n & \text{if for every } i\in\{1,\ldots,n\} \text{ we have } -a\le x_i \le a, \\ 0 & \text{otherwise}. \end{cases} $$

The condition that for every $i\in\{1,\ldots,n\}$ we have $-a\le x_i\le a$ is the same as $\min\{x_1,\ldots,x_n\} \ge -a$ and $\max\{x_1,\ldots,x_n\}\le a.$ That condition on $\min$ is the same as $-\min\{x_1,\ldots,x_n\} \le a.$ So we need $a\ge\max$ and $a\ge -\min.$ I leave it as an exercise to show that $$ \Big( a\ge \max\{x_1,\ldots,x_n\} \text{ and } a\le -\min\{x_1,\ldots,x_n\} \Big) \text{ if and only if } a \ge \max\{|x_1|,\ldots,|x_n|\}. $$

Therefore the likelihood function is $$ L(a) = \begin{cases} 1/(2a)^n & \text{if } a \ge \max\{|x_1|,\ldots,|x_n|\}, \\ 0 & \text{otherwise.} \end{cases} $$ Now notice that $L(a)$ increases as $a$ decreases, until $a$ gets down to $\max\{|x_1|,\ldots,|x_n|\}.$ Therefore that maximum is the MLE.

$\endgroup$
  • $\begingroup$ Ok, this explanation definitely makes more sense. So you're saying that the PDF defined by f_X1,...,Xn(x1, ... ,xn) = {1/(2a)^n if ....... ... } is the likelihood of the data set {x1, ... , xn} given the parameter a, correct? $\endgroup$ – Josh Jul 21 '17 at 14:14
  • $\begingroup$ @Josh : No: The likelihood is a function of $a$ with $x_1,\ldots,x_n$ fixed, whereas the density is a function of $x_1,\ldots,x_n$ with $a$ fixed. $\qquad$ $\endgroup$ – Michael Hardy Jul 21 '17 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.