# Maximum Likelihood Estimate with Multiple Parameters

I am not very familiar with multivariable calculus, but something tells me that I don't need to be in order to solve this problem; take a look:

Suppose that $$X_1,...,X_m$$ and $$Y_1,...,Y_n$$ are independent exponential random variables with $$X_i\sim EXP(\lambda)$$ and $$Y_j\sim EXP(\theta \lambda)$$.

Find the $$MLE$$ of $$\lambda$$ and $$\theta$$.

Finding the MLE of $$\lambda$$ is simple; by ignoring the $$Y_j$$ altogether and just looking at the $$X_i$$, it turns out to be $$\sum x_i/m$$. However, for $$\theta$$, I am no longer sure since the distribution of $$Y_j$$ is also dependent on $$\lambda$$. I don't know if I need to go as far as finding the gradient or if I can somehow use my previous result, but either way, I honestly don't know how to do it.

The log likelihood is given by $(m+n)log(\lambda) + n log(\theta)-\lambda \sum x_i -\theta \lambda \sum y_i$

The MLE for $\lambda$ including both $X$ and $Y$ turns out to be the same as just using $X$. That wasn't obvious to me.

For $\theta$ you get $n/\theta = \lambda \sum y_i$ for which you just substitute for the MLE of $\lambda$.

Deriving the MLE: From your specification of the problem, your log-likelihood function is:

\begin{equation} \begin{aligned} \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}(\theta, \lambda) &= \sum_{i=1}^m \ln p (x_i | \lambda) + \sum_{i=1}^n \ln p (y_i | \theta, \lambda) \\[8pt] &= \sum_{i=1}^m (\ln \lambda - \lambda x_i) + \sum_{i=1}^n (\ln \theta + \ln \lambda - \theta \lambda y_i) \\[8pt] &= m ( \ln \lambda - \lambda \bar{x} ) + n ( \ln \theta + \ln \lambda - \theta \lambda \bar{y}). \end{aligned} \end{equation}

This gives the score functions:

\begin{equation} \begin{aligned} \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \theta}(\theta, \lambda) &= n \Big( \frac{1}{\theta} - \lambda \bar{y} \Big), \\[8pt] \frac{\partial \mathcal{l}_{\boldsymbol{x},\boldsymbol{y}}}{\partial \lambda}(\theta, \lambda) &= m \Big( \frac{1}{\lambda} - \bar{x} \Big) + n \Big( \frac{1}{\lambda} - \theta \bar{y} \Big). \end{aligned} \end{equation}

Setting both partial derivatives to zero and solving the resulting score equations yields the MLEs:

$$\hat{\theta} = \frac{\bar{x}}{\bar{y}} \quad \quad \quad \hat{\lambda} = \frac{1}{\bar{x}}.$$

(Note that in the case where $\bar{y} = 0$ the first of the score equations is strictly positive and so the MLE for $\theta$ does not exist.) As user121049 correctly points out, the MLE for $\lambda$ is the same as if you only used the $x_i$ values.

Both answers are good but in practice you'd also want to obtain some measure of precision for the estimates. One approach is to use a bootstrap. Below is another common approach.

Take the 2nd derivatives of the log of the likelihood function:

$$\frac{\partial ^2\text{logL}}{\partial \{\lambda ,\theta \}^2}=\left( \begin{array}{cc} \frac{\lambda ^2}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} & -\frac{\theta ^2 \lambda ^2 \bar{y}}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} \\ -\frac{\theta ^2 \lambda ^2 \bar{y}}{\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n} & \frac{\theta ^2 (m+n)}{n \left(\theta ^2 \lambda ^2 (-n) \bar{y}^2+m+n\right)} \\ \end{array} \right)$$

Take minus the inverse of that resulting matrix and then substitute in the maximum likelihood estimators. One obtains

$$\hat{\Sigma}=\left( \begin{array}{cc} \frac{1}{m \bar{x}^2} & -\frac{1}{m \bar{y}} \\ -\frac{1}{m \bar{y}} & \frac{\bar{x}^2 (m+n)}{m n \bar{y}^2} \\ \end{array} \right)$$

An estimate of the variance of $$\hat{\lambda}$$ is $$1/(m \bar{x}^2)$$ and an estimate of the variance of $$\hat{\theta}$$ is $$\frac{\bar{x}^2 (m+n)}{m n \bar{y}^2}$$. An estimate of the covariance is $$-\frac{1}{m \bar{y}}$$.