Asymptotic Standard Error of Estimator that is a Function of Two Samples

Question

Suppose that $X_1,...,X_m$ and $Y_1,...,Y_n$ are independent exponential random variables with $X_i\sim EXP(\lambda)$ and $Y_j\sim EXP(\theta \lambda)$. This implies that $E(X_i)=1/\lambda$ and $E(Y_j)=1/{\lambda\theta}$.

Let $\hat\theta_{m,n}$ be the maximum likelihood estimator of $\theta$. Find an expression for the approximate standard error of $\hat\theta_{m,n}$. Assume that $m$ and $n$ are large and use asymptotic theory.

I end up calculating that the MLE is $\hat\theta_{m,n}=\frac{\bar X}{\bar Y}$, where X and Y bar are just the sample means of the respective distributions. I'm not one hundred percent confident in this, but it seems right.

So from here, I need to find the variance of this expression; particularly, the limiting variance; but here's the issue; this MLE depends on both $m$ and $n$! In my experience, I've never had to manipulate an MLE like this; so the typical way to do this (when you have just one sample and one parameter):

$$\sqrt{n}(\hat\theta-\theta)\to^d N(0,I_1^{-1}(\theta))$$

doesn't make sense any more! Or at least, not to me. So how can I possibly "find an expression for the approximate standard error of $\hat\theta_{m,n}$?"

Answer 1

Checking the MLE: From your specification of the problem, your log-likelihood function is:

$$\begin{equation} \begin{aligned} \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}(\theta, \lambda) &= \sum_{i=1}^m \ln p (x_i | \lambda) + \sum_{i=1}^n \ln p (y_i | \theta, \lambda) \\[8pt] &= \sum_{i=1}^m (\ln \lambda - \lambda x_i) + \sum_{i=1}^n (\ln \theta + \ln \lambda - \theta \lambda y_i) \\[8pt] &= m ( \ln \lambda - \lambda \bar{x} ) + n ( \ln \theta + \ln \lambda - \theta \lambda \bar{y}). \end{aligned} \end{equation}$$

This gives the score functions:

$$\begin{equation} \begin{aligned} \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \theta}(\theta, \lambda) &= n \Big( \frac{1}{\theta} - \lambda \bar{y} \Big), \\[8pt] \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \lambda}(\theta, \lambda) &= m \Big( \frac{1}{\lambda} - \bar{x} \Big) + n \Big( \frac{1}{\lambda} - \theta \bar{y} \Big). \end{aligned} \end{equation}$$

Setting both partial derivatives to zero and solving the resulting score equations yields the MLEs:

$$\hat{\theta}_{m,n} = \frac{\bar{x}}{\bar{y}} \quad \quad \quad \hat{\lambda}_{m,n} = \frac{1}{\bar{x}}.$$

(Note that in the case where $\bar{y} = 0$ the first of the score equations is strictly positive and so the MLE for $\theta$ does not exist.) This confirms your calculations of the MLE.

---

Adjusting the MLE to remove bias: Treating the MLE as a random variable we have:

$$\hat{\theta}_{m,n} = \frac{n}{m} \cdot \frac{\dot{X}}{\dot{Y}},$$

where $\dot{X} \equiv m \bar{X} \sim \text{Gamma} (m, \lambda)$ and $\dot{Y} \equiv n \bar{Y} \sim \text{Gamma} (n, \theta \lambda)$ are independent random variables. From this equation, the MLE is a scaled beta-prime random variable:

$$\hat{\theta}_{m,n} \sim \theta \cdot \frac{n}{m} \cdot \text{Beta-Prime}(m, n).$$

This estimator has expected value $\mathbb{E} (\hat{\theta}_{m,n}) = \frac{n}{n-1} \cdot \theta$, which means that it has positive bias. We can correct this bias by using the bias-adjusted MLE:

$$\tilde{\theta}_{m,n} = \frac{n-1}{n} \cdot \frac{\bar{X}}{\bar{Y}} \sim \theta \cdot \frac{n-1}{m} \cdot \text{Beta-Prime}(m, n).$$

---

Standard-Error of the adjusted MLE: The adjusted MLE is unbiased. It has variance:

$$\begin{equation} \begin{aligned} \mathbb{V}(\tilde{\theta}_{m,n}) &= \int \limits_0^\infty \Big( \theta \cdot \frac{n-1}{m} \cdot r - \theta \Big)^2 \text{Beta-Prime} ( r | m, n) dr \\[8pt] &= \theta^2 \cdot \frac{\Gamma(m) \Gamma(n)}{\Gamma(m+n)} \int \limits_0^\infty \Big( 1 - \frac{n-1}{m} \cdot r \Big)^2 r^{m-1} ( 1 + r )^{-m-n} dr \\[8pt] &= \theta^2 \cdot \frac{n+m-1}{m(n-2)}. \end{aligned} \end{equation}$$

The corresponding standard error is:

$$\text{se}(\tilde{\theta}_{m,n}) = \tilde{\theta}_{m,n} \cdot \sqrt{\frac{n+m-1}{m(n-2)}}.$$

Letting $\phi \equiv m/n$ and taking the limit as $n \rightarrow \infty$ we obtain the asymptotic approximation:

$$\text{se}(\tilde{\theta}_{m,n}) \approx \frac{\tilde{\theta}_{m,n}}{\sqrt{n-2}} \cdot \sqrt{\frac{1+\phi}{\phi}}.$$

This gives you both exact and approximate expressions for the standard error. I hope that is helpful. (Please make sure to review my algebra to make sure I haven't made a mistake!)