# Maximum Likelihood Estimator Independent Exponentials

Here is the problem statement: Let $(X_i,Y_i)$ be a random sample from a distribution with pdf $$f(x,y;\theta)= \frac{1}{\theta^3}\exp\left(\frac{-x}{\theta}-\frac{y}{\theta^2}\right)\qquad 0<x, 0<y.$$ (a) Find the MLE for $\theta$.

(b) Give the asymptotic distribution of $\sqrt{n}(\hat{\theta}-\theta).$

Here are my thoughts: We basically have two random samples, $X\sim \exp(\theta)$ and $Y\sim \exp(\theta^2)$. This pdf is the joint pdf for $X$ and $Y$. The parameter has no impact on the support of the variables $X$ and $Y$, so calculus should suffice for this problem. $$L(x,y;\theta) = \prod_{i=1}^n f(x,y;\theta) = \frac{1}{\theta^{3n}}\exp\left(\frac{-1}{\theta^2}\sum_{i=1}^n \theta x_i+y_i\right),$$ so $$l(x,y;\theta) = -3n\ln(\theta)-\frac{1}{\theta^2}\sum_{i=1}^n \theta x_i + y_i,$$ which has partial $\theta$ derivative $$\frac{-3n\theta^2+\sum_{i=1}^n \theta x_i +2y_i}{\theta^3}.$$ Equating to zero and solving for $\theta$ requires solving a quadratic in $\theta$. Is there an easy way to see the solution, or is solving this quadratic necessary? Have I made any mistakes to this point?

Finding the MLE: You should be more careful with your sigma-notation in your working, by bracketing the terms in the sum. The simplest way to express the score log-likelihood here is in terms of the sample means $\bar{x} \equiv \tfrac{1}{n} \sum_{i=1}^n x_i$ and $\bar{y} \equiv \tfrac{1}{n} \sum_{i=1}^n y_i$, which gives you:

$$\ell_{\boldsymbol{x},\boldsymbol{y}}(\theta) = - n \Big( 3 \ln (\theta) + \frac{\bar{x}}{\theta} + \frac{\bar{y}}{\theta^2} \Big).$$

The corresponding score function and Hessian are:

\begin{aligned} \frac{d \ell_{\boldsymbol{x},\boldsymbol{y}}}{d \theta}(\theta) &= - \frac{n}{\theta^3} \Big( 3 \theta^2 - \bar{x}\theta - 2 \bar{y} \Big), \\[8pt] \frac{d^2 \ell_{\boldsymbol{x},\boldsymbol{y}}}{d \theta^2}(\theta) &= \frac{n}{\theta^4} \Big( 3 \theta^2 - 2 \bar{x} \theta - 6 \bar{y} \Big). \end{aligned}

Setting the score function to zero gives the quadratic score equation $3 \hat{\theta}^2 - \bar{x} \hat{\theta} - 2 \bar{y} = 0$. Solving this quadratic equation yields the unique positive critical-point:

$$\hat{\theta} = \frac{\bar{x} + \sqrt{\bar{x}^2 + 24 \bar{y}} }{6}.$$

Since this is the only positive root of the quadratic, this implies that the score function is positive for $\theta < \hat{\theta}$ and negative for $\theta > \hat{\theta}$, which means that it is a quasi-concave function. This critical point is therefore the unique maximising value and so it is the MLE.

Distribution of the MLE: Using the fact that $\mathbb{E}(\bar{x}) = \theta$ and $\mathbb{E}(\bar{y}) = \theta^2$ the Fisher Information (for the single observation) in this case is:

$$\mathcal{I}(\theta) = - \mathbb{E} \Bigg[ \frac{d^2 \ell_{x,y}}{d \theta^2}(\theta) \Bigg| \theta \Bigg] = - \frac{1}{\theta^4} \Big( 3 \theta^2 - 2 \theta^2 - 6 \theta^2 \Big) = \frac{5}{\theta^2}.$$

Using the standard asymptotic properties of MLEs you have the asymptotic distribution:

$$\sqrt{n} (\hat{\theta} - \theta) \xrightarrow{\text{Dist}} \text{N} \Big( 0, \frac{\theta^2}{5} \Big).$$

Hence, for large $n$ you have the distributional approximation:

$$\hat{\theta} \sim \text{N} \Big( \theta, \frac{\theta^2}{5n} \Big).$$