Suppose that $X_1,...,X_m$ and $Y_1,...,Y_n$ are independent exponential random variables with $X_i\sim EXP(\lambda)$ and $Y_j\sim EXP(\theta \lambda)$. This implies that $E(X_i)=1/\lambda$ and $E(Y_j)=1/{\lambda\theta}$.

Let $\hat\theta_{m,n}$ be the maximum likelihood estimator of $\theta$. Find an expression for the approximate standard error of $\hat\theta_{m,n}$. Assume that $m$ and $n$ are large and use asymptotic theory.

I end up calculating that the MLE is $\hat\theta_{m,n}=\frac{\bar X}{\bar Y}$, where X and Y bar are just the sample means of the respective distributions. I'm not one hundred percent confident in this, but it seems right.

So from here, I need to find the variance of this expression; particularly, the limiting variance; but here's the issue; this MLE depends on both $m$ and $n$! In my experience, I've never had to manipulate an MLE like this; so the typical way to do this (when you have just one sample and one parameter):

$$\sqrt{n}(\hat\theta-\theta)\to^d N(0,I_1^{-1}(\theta))$$

doesn't make sense any more! Or at least, not to me. So how can I possibly "find an expression for the approximate standard error of $\hat\theta_{m,n}$?"

  • $\begingroup$ How do the $X_i$ variables have any relevance to $\theta$? $\endgroup$
    – Michael
    Apr 6, 2018 at 4:54
  • $\begingroup$ I see your point; so should it simply be left as an expression of m? If so, why would it say "assume that $m$ is large?" $\endgroup$ Apr 6, 2018 at 5:37
  • $\begingroup$ I think you'll find that the $x_i$ variables are relevant insofar as they give information on $\lambda$, which then assists in estimation of $\theta$ through the $y_i$ variables. When you write out the likelihood function and calculate the MLE, it is a function of both sets of observed values (the form given by the OP is correct). $\endgroup$
    – Ben
    Apr 6, 2018 at 6:04
  • $\begingroup$ @Ben : It is not clear what is given info and what is to be estimated. It looks like $\lambda$ is a known constant, whereas $\theta$ is to be estimated. Though, in that case the $\{X_i\}$ values are irrelevant, so likely your interpretation is the intended one. It would not hurt for the problem to be clarified. $\endgroup$
    – Michael
    Apr 6, 2018 at 6:29
  • $\begingroup$ @Michael, the lambda is not a known constant in this context; this I am certain of. $\endgroup$ Apr 6, 2018 at 6:46

1 Answer 1


Checking the MLE: From your specification of the problem, your log-likelihood function is:

$$\begin{equation} \begin{aligned} \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}(\theta, \lambda) &= \sum_{i=1}^m \ln p (x_i | \lambda) + \sum_{i=1}^n \ln p (y_i | \theta, \lambda) \\[8pt] &= \sum_{i=1}^m (\ln \lambda - \lambda x_i) + \sum_{i=1}^n (\ln \theta + \ln \lambda - \theta \lambda y_i) \\[8pt] &= m ( \ln \lambda - \lambda \bar{x} ) + n ( \ln \theta + \ln \lambda - \theta \lambda \bar{y}). \end{aligned} \end{equation}$$

This gives the score functions:

$$\begin{equation} \begin{aligned} \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \theta}(\theta, \lambda) &= n \Big( \frac{1}{\theta} - \lambda \bar{y} \Big), \\[8pt] \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \lambda}(\theta, \lambda) &= m \Big( \frac{1}{\lambda} - \bar{x} \Big) + n \Big( \frac{1}{\lambda} - \theta \bar{y} \Big). \end{aligned} \end{equation}$$

Setting both partial derivatives to zero and solving the resulting score equations yields the MLEs:

$$\hat{\theta}_{m,n} = \frac{\bar{x}}{\bar{y}} \quad \quad \quad \hat{\lambda}_{m,n} = \frac{1}{\bar{x}}.$$

(Note that in the case where $\bar{y} = 0$ the first of the score equations is strictly positive and so the MLE for $\theta$ does not exist.) This confirms your calculations of the MLE.

Adjusting the MLE to remove bias: Treating the MLE as a random variable we have:

$$\hat{\theta}_{m,n} = \frac{n}{m} \cdot \frac{\dot{X}}{\dot{Y}},$$

where $\dot{X} \equiv m \bar{X} \sim \text{Gamma} (m, \lambda)$ and $\dot{Y} \equiv n \bar{Y} \sim \text{Gamma} (n, \theta \lambda)$ are independent random variables. From this equation, the MLE is a scaled beta-prime random variable:

$$\hat{\theta}_{m,n} \sim \theta \cdot \frac{n}{m} \cdot \text{Beta-Prime}(m, n).$$

This estimator has expected value $\mathbb{E} (\hat{\theta}_{m,n}) = \frac{n}{n-1} \cdot \theta$, which means that it has positive bias. We can correct this bias by using the bias-adjusted MLE:

$$\tilde{\theta}_{m,n} = \frac{n-1}{n} \cdot \frac{\bar{X}}{\bar{Y}} \sim \theta \cdot \frac{n-1}{m} \cdot \text{Beta-Prime}(m, n).$$

Standard-Error of the adjusted MLE: The adjusted MLE is unbiased. It has variance:

$$\begin{equation} \begin{aligned} \mathbb{V}(\tilde{\theta}_{m,n}) &= \int \limits_0^\infty \Big( \theta \cdot \frac{n-1}{m} \cdot r - \theta \Big)^2 \text{Beta-Prime} ( r | m, n) dr \\[8pt] &= \theta^2 \cdot \frac{\Gamma(m) \Gamma(n)}{\Gamma(m+n)} \int \limits_0^\infty \Big( 1 - \frac{n-1}{m} \cdot r \Big)^2 r^{m-1} ( 1 + r )^{-m-n} dr \\[8pt] &= \theta^2 \cdot \frac{n+m-1}{m(n-2)}. \end{aligned} \end{equation}$$

The corresponding standard error is:

$$\text{se}(\tilde{\theta}_{m,n}) = \tilde{\theta}_{m,n} \cdot \sqrt{\frac{n+m-1}{m(n-2)}}.$$

Letting $\phi \equiv m/n$ and taking the limit as $n \rightarrow \infty$ we obtain the asymptotic approximation:

$$\text{se}(\tilde{\theta}_{m,n}) \approx \frac{\tilde{\theta}_{m,n}}{\sqrt{n-2}} \cdot \sqrt{\frac{1+\phi}{\phi}}.$$

This gives you both exact and approximate expressions for the standard error. I hope that is helpful. (Please make sure to review my algebra to make sure I haven't made a mistake!)

  • $\begingroup$ Likely your use of $m$ and $n$ is switched in the $\dot{X}$ and $\dot{Y}$ definitions. $\endgroup$
    – Michael
    Apr 6, 2018 at 6:37
  • $\begingroup$ Can you show me where? $\endgroup$
    – Ben
    Apr 6, 2018 at 6:59
  • $\begingroup$ You have $\dot{X} = m\overline{X}$, I thought it should be $\dot{X}=n \overline{X}$, but I see in fact you were correct. $\endgroup$
    – Michael
    Apr 6, 2018 at 13:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .