5
$\begingroup$

Suppose we are given random variables $X_1,...,X_n$ that are uniformly distributed on the interval $[0,\theta]$, with $\theta >0$ unknown. I know that if the $X_1,...,X_n$ are furthermore independent, we can conclude that the Maximum Likelihood Estimation is given as $\hat{\theta} = \max\limits_i x_i$.

I was wondering if the same conclusion also holds if the random variables $X_1,...,X_n$ are not independent. My main problem is that I am unable to determine the Likelihood function, which I do not know if it is possible if the $X_i$ are not independent.

I would appreciate any help.

$\endgroup$
5
  • $\begingroup$ It’s the multivariate density $\endgroup$
    – JohnK
    Commented Jun 11 at 5:11
  • $\begingroup$ Intuitively, it doesn't seem to me like the samples being dependent would change the MLE ? The maximizer of the likelihood should still be the max of the $x_i$ because anything less than the max of the $x_i$ is not possible. The likelihood only exists for $\hat\theta$ = the maximum of the $x_i$ or greater. $\endgroup$
    – mark leeds
    Commented Jun 11 at 5:11
  • $\begingroup$ @markleeds I understand that intuitively, we should obtain the same MLE, however, I was not able to prove it. I also understand that it should be zero is the $x_i$ are less than $0$ or greater then $\theta$. I was not able to show that the function is then decreasing for $\theta > \max\limits_i x_i$ $\endgroup$
    – user007
    Commented Jun 11 at 5:24
  • 2
    $\begingroup$ There is no way to determine the likelihood function when you don't know the dependence structure of the $X_i$'s. $\endgroup$ Commented Jun 11 at 11:50
  • $\begingroup$ @user007: Based on the answer's below, it looks my comment is totally wrong. My apologies. I would delete my comment but then your comment and stubbornatom's comment would not make sense. $\endgroup$
    – mark leeds
    Commented Jun 11 at 12:51

2 Answers 2

5
$\begingroup$

The hypothesis that the sample maximum is the MLE, even if there are dependencies, is false. The MLE really depends on how the $X_i$ depend on each other. Consider this example:

$$ X_1 \sim U(0, \theta) $$ $$ X_n \sim U(0, \theta/2)\,\, \textit{if}\,\, x_1<\theta/2 $$ $$ X_n \sim U(\theta/2, \theta)\,\, \textit{if}\,\, x_1>\theta/2 $$ for $n>1$. You can see that, marginally, each $X_i$ is $U(0, \theta)$, but they are not independent.

Now suppose that you observe $x_1 = 0.1$ and $\max {x_i} = 0.9$. The likelihood of $\theta$ being 0.9 is zero, since, being $x_1<0.9/2$, every subsequent $x_i$ should have lied in the $[0, 0.45]$ interval. The MLE here is 1.8, twice the maximum observed value.

$\endgroup$
4
  • $\begingroup$ nicola: neat example. I didn't realize that dependence meant that the marginal distribution can be different for each $X_i$. $\endgroup$
    – mark leeds
    Commented Jun 11 at 12:49
  • $\begingroup$ Yes, I considered the marginally uniform a requirement for each $X_i$. The MLE depends on the joint probability function. $\endgroup$
    – nicola
    Commented Jun 11 at 13:00
  • 1
    $\begingroup$ @nicola Apologies, not sure what I was thinking. This works, +1 $\endgroup$ Commented Jun 11 at 14:25
  • $\begingroup$ @nicola thank you for the simple and easy to understand answer $\endgroup$
    – user007
    Commented Jun 12 at 11:40
3
$\begingroup$

By definition, when $X_1,\dots, X_n$ are dependent, the MLE is $X_{(n)}=\max\limits_i X_i$ if

A: and $X_1,\dots, X_n$ have a joint pdf $f$

B: The parameter $\theta$ only affects the univariate marginal distributions of the joint distribution of $(X_1,\dots, X_n)$.

C: The joint pdf of $\left ( \frac{X_1}{\theta},\dots, \frac{X_n}{\theta} \right )$ is increasing in each of its arguments.

Condition A is considered so that the analysis of the likelihood function becomes easier in the following (the ML method can be used for a more general case where the density is defined as the Radon-Nikodym derivative of the probability distribution relative to a dominating measure other than Lebesgue measure). Condition B is used to make sure that the distribution of $\left ( \frac{X_1}{\theta},\dots, \frac{X_n}{\theta} \right )$ is independent of $\theta$.

Under these conditions, for $0\le x_i \le \theta, i=1,\dots,n$, we have

$$f \left ( x_1,\dots, x_n \right )=\left ( \frac{1}{\theta} \right )^n g \left ( \frac{x_1}{\theta},\dots, \frac{x_n}{\theta} \right ) $$

where $g: [0,1]^n\to \mathbb R_{\ge 0}$ is the joint pdf of $\left ( \frac{X_1}{\theta},\dots, \frac{X_n}{\theta} \right ).$ Then, we have

$$l(\theta)=\left ( \frac{1}{\theta} \right )^n g \left ( \frac{x_1}{\theta},\dots, \frac{x_n}{\theta} \right ) I(\theta \ge x_{(n)})$$

where $g$ is independent of $\theta$ under condition B. Hence, under C, the MLE of $\theta$ is $X_{(n)}$.


Condition B is equivalent to that the copula $C$ associated with $X_1,\dots, X_n$ is independent of $\theta$, that is, in the following Sklar's representation:

$$F_{X_1,\dots, X_n} \left ( x_1,\dots, x_n \right )=C\left ( F_{X_1}(x_1),\dots, F_{X_n}(x_n) \right ),$$

the copula $C$ is independent of $\theta$. In this case, condition A is equivalent to that the copula $C$ has a joint pdf $g$, and condition C means that $g$ is increasing in each argument.

Note that in the independence case, for $u\in [0,1]^n$, we have $$C(u_1,\dots, u_n)=u_1×\cdots×u_n,$$ with $g(u_1,\dots, u_n)=1,$ and all the three conditions A, B, and C are satisfied.


Counterexample in lack of C:

For $n=2$, consider observations $x_1=x_2=5$ where the joint copula of $\frac{X_1}{\theta}, \frac{X_2}{\theta}$ is assumed to be the Frank copula [1] with parameter $-2$. Then, as you can see below from the plot of $l(\theta)$ [2], the maximum likelihood estimate of $\theta$ is a number greater than $x_{(2)}=5$.

enter image description here

$\endgroup$
4
  • $\begingroup$ I am not sure A is required to use the maximum likelihood method: if $X_1 \sim U[0, \theta]$ and $X_i=X_1$ for all $i$ then there is still a likelihood $\big($proportional to $\frac1\theta\, \mathbf I_{[0\le X_1\le \theta]}\big)$, but no joint density. $\endgroup$
    – Henry
    Commented Jun 11 at 17:07
  • $\begingroup$ @Henry Thanks for your comment! To use ML method, we need the density of data, which can be a joint pdf (in A, I used this case), a joint pmf, or more generally the Radon-Nikodym derivative of the probability distribution relative to a dominating measure. For the example you gave, we may use the general case. $\endgroup$
    – Amir
    Commented Jun 11 at 17:50
  • $\begingroup$ @Amir thank you for the detailed answer. The reason I accepted nicola's answer is, that the counterexample is simpler $\endgroup$
    – user007
    Commented Jun 12 at 11:41
  • $\begingroup$ @user007 You are welcome! I appreciate your kind consideration! I like Nicola's answer too. Actually, it could be guessed that the claim couldn't not hold generally, so I was not seeking to only present a counterexample. My answer provides sufficient conditions under which the claim holds. Hope weaker conditions will be presented. $\endgroup$
    – Amir
    Commented Jun 12 at 12:10

Not the answer you're looking for? Browse other questions tagged .