1
$\begingroup$

Let $\theta\in[0,1]$ be a parameter. Suppose that $Y$ is a random variable that takes value $\theta$ with probability $1/2$ and is uniformly distributed on $[0,1]$ with probability $1/2$. What is the maximum likelihood estimator of $\theta$ given independent observations $y_1,...,y_n$?

Stepping back from formalism, it seems very clear how to guess $\theta$ given a list of observations of $Y$. However, I don't see how a likelihood function can be defined for this family of distributions.

$\endgroup$
  • $\begingroup$ What is the meaning of '$Y$ is uniformly distributed on $[0,1]$ with probability $1/2$'? How is $\theta$ involved here? $\endgroup$ – StubbornAtom Apr 18 at 8:41
  • $\begingroup$ @StubbornAtom If you want a more technical description, take $X$ to be Bernoulli with $p=1/2$. Then define $Y_\theta = \theta X + (1 - X)U$ where $U$ is independent and uniformly distributed on $[0,1]$. $\endgroup$ – user3281410 Apr 18 at 18:13
  • $\begingroup$ Thank you...... $\endgroup$ – StubbornAtom Apr 18 at 20:03
1
$\begingroup$

Let's start by stating the intuitive answer:

Clearly the observation value which appears most often is the maximum likelihood estimate for $\theta$. There will be such a value with probability $1-\frac{n+1}{2^n}$ if $n \gt 1$ and with probability $1$ if $n=1$; otherwise, each observation value will appears only once with probability $\frac{n+1}{2^n}$ if $n \gt 1$ and they are each equally likely to be $\theta$ even though there is a positive probability that none of them are

Now let's try to express this in likelihood terms, noting that the probability a single observation is $\theta$ is $\frac12$ and the density for any observation is not $\theta$ is $\frac12$, though there is a fundamental distinction between probabilities and densities:

In a sense the likelihood is proportional to the product of the $n$ probability densities and probabilities associated with the observations. But providing that none of these are zero, any non-zero probability should conceptually lead to a higher likelihood than a probability density, e.g. combining $k$ probabilities and $n-k$ probability densities has an intrinsically higher likelihood than $k-1$ probabilities and $n-k+1$ probability densities. You cannot quite say that a probability corresponds to an infinite density or that a finite density corresponds to a zero probability, since you may have several of each, but this does provide a weak analogy to how probabilities dominate densities

So if you have a value $x_m$ which appears $k \gt 1 $ times together $n-k$ other distinct observations, if $\theta = x_m$ then the likelihood is proportional to $\frac1{2^n}$ with $k$ probabilities and $n-k$ densities while if $\theta \not = x_m$ then the likelihood is proportional to $\frac1{2^n}$ with $1$ or $0$ probabilities and $n-1$ or $n$ densities. The $\theta = x_m$ case then has a higher intrinsic likelihood than any $\theta \not = x_m$, making $x_m$ the maximum likelihood estimator

On the other hand if you have $n$ distinct observations $\{x_i\}$ then if any one of them is $\theta$ then the likelihood is proportional to $\frac1{2^n}$ with $1$ probability and $n-1$ densities while if $\theta \not \in \{x_i\}$ then the likelihood is proportional to $\frac1{2^n}$ with $0$ probabilities and $n$ densities. The former cases then have a higher intrinsic likelihood than the latter cases making each of the $\{x_i\}$ maximum likelihood estimators

$\endgroup$
  • $\begingroup$ I appreciate the response, but I am mostly interested in rigorous definitions that allow the MLE to be defined in terms of a solution to an optimization problem. We can't define a likelihood function because there is no dominating measure for this family of distributions. I believe you are saying that the MLE is the function $f(y_1,...,y_n)$ that selects whatever coordinate is equal to others the most times and returns an arbitrary coordinate if there are no repetitions. However, I think it would be hard to modify the same reasoning if the problem was more complicated. $\endgroup$ – user3281410 Apr 18 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.