3
$\begingroup$

I am looking at parameters estimation for the uniform distribution in the context of MLEs. Now, I know the likelihood function of the Uniform distribution $U(0,\theta)$ which is $1/\theta^n$ cannot be differentiated at $\theta$. The conclusion is that the estimate of $\theta$ is $\max(x_i)$, where $x_1,x_2,\ldots,x_n$ is the random sample. I would like a layman's explanation for this.

$\endgroup$
2
$\begingroup$

You asked for an intuitive explanation. So let's think about what happens when we take a sample $x_1, x_2, \ldots, x_n$ from a uniform $[0,\theta]$ random variable, where $\theta$ is a fixed but unknown parameter. From the data, we would like to calculate a statistic (a function of the data) that estimates the value of the parameter; moreover, this estimator $\hat \theta$ maximizes the likelihood of having observed the particular sample that we obtained.

Clearly, $\theta$ cannot be smaller than the largest observation we made (or else we could not have observed it!). But in order to "maximize the likelihood," we also note that $\hat\theta$ should not be unnecessarily large. For instance, if we observed $x_1 = 2, x_2 = 7, x_3 = 4$, we could pick $\hat\theta = 100$, but intuitively this is not a good estimate--the chance that we observed $2, 7, 4$ given that the true value of the parameter is $100$ is quite small. It is much more likely that, given the information we have, the true value of the parameter is close to $7$. In fact, we can show mathematically that the probability of observing the sample given $\theta > \max_i x_i$ is smaller than if $\theta = \max_i x_i$.

We can also characterize $\hat\theta = \max_i x_i$ in terms of sufficiency: notice that once we observe the largest value, the other values give no additional information about the true value of $\theta$. So in our example, once we saw the maximum of $7$, the observations $x_1 = 2$ and $x_3 = 4$ are irrelevant for the purposes of estimating $\theta$, because we already know that $\theta \ge 7$.

Of course, we must formalize all of this with the appropriate mathematics, but this is essentially the underlying intuition involved.

$\endgroup$
3
$\begingroup$

Your likelihood function is correct, but strictly speaking only for values of $\theta \ge x_\max$.

As you know the likelihood function is the product of the conditional probabilities $P(X_i=x_i|\theta)$.

$$L = P(X_1=x_1|\theta) \times P(X_2=x_2|\theta) \times \ldots \times P(X_2=x_2|\theta) $$

We can look at three different cases for your $\hat\theta$ = the estimate of $\theta$ that maximizes the likelihood.

(i) $\hat\theta_1 < \max(x_i)$

Let's say that $x_j=x_\max$. This gives $P(X_j=x_\max|\theta=\hat\theta_1) = 0$, because $x_\max$ is outside the support of the random variable, and so

$$L = P(X_1=x_1|\theta=\hat\theta_1) \times \ldots \times P(X_j=x_\max|\theta=\hat\theta_1) \times \ldots \times P(X_2=x_2|\theta=\hat\theta_1) = 0 $$

(ii) $\hat\theta_2 > \max(x_i)$

Here all the data is within the support so we have $L=1/\hat\theta_2^n$.

Clearly case (ii) is better than (i) because $1/\hat\theta_2^n > 0$.

(iii) $\hat\theta_3 = \max(x_i)$

Once again $L=1/\hat\theta^n$. But case (iii) is better than case (ii) because $\forall a,b:\; a>b>0 \;=>\; 1/a < 1/b$.

So out of the three cases, case (iii) maxmizes $L$. (Note that you can use calculus to show that $L$ is monotonically decreasing for $\theta < x_\max$ if you wish.)

$\endgroup$
2
$\begingroup$

${}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}$ enter image description here

$\endgroup$
1
$\begingroup$

A layman's explanation is that you observe realizations of a random variable, whose upper bound you do not know. You have no idea if it is $10$, $42$, $1337$, or so on. The only thing you know is that it is greater than $0$ and less than the bound you're interested in, $\theta$. Intuitively, it makes sense that your best guess is the greatest observation you have. Naturally, $\theta$ cannot be less than the greatest observation you have, because you have something greater than it. But if you would add an arbitrary (positive) number to your greatest observation as an estimate, you could very well exceed the true $\theta$. Thus, you settle for the greatest observed value as your estimate of $\theta$, because that is a pretty good guess.

If you add an indicator function to your pdf, it is even more clear that the likelihood is maximized at the maximum observation.

$\endgroup$
  • $\begingroup$ Well, if you're a bit bigger than the maximum, then theres less probability (in a small neighborhood) around each of the observations since the density becomes less tall than if you had chosen the maximum, so the likelihood has to be lower than the maximum. $\endgroup$ – Batman Feb 11 '14 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.