# MLE with parameters constrained

I am wondering how I can solve this problem more elegantly:

Let's say I have two set of i.i.d normal random variables with known variance $$\sigma^2=1$$: $$X_1, X_2, ..., X_m \sim N(\mu_1, 1)$$ $$Y_1, Y_2, ..., Y_n \sim N(\mu_2, 1)$$

And it's given that $$\mu_1 \leq \mu_2$$. So the MLE of such an observation should be:

$$L = \prod_{i=1}^m f(X_i | \mu_1) \prod_{j=1}^n f(Y_j|\mu_2)$$ Where $$f(\cdot|\mu)$$ is the pdf of normal random variable with $$\mu$$ as mean and 1 as variance.

Under the constraint of $$\mu_1\leq\mu_2$$, I can think of this as an optimization problem, and that is to optimize for the L ( or log(L)), subject to the constraint. But this is pretty hard to reach at some kind of closed form solutions.

I am wondering if there is some more clever way to process this problem?

Thanks

• With or without the constraint you have the same likelihood surface which in this case has a single maximum. With a constraint you just look at a restricted area. So if $\bar{x}_1\leq \bar{x}_2$, then $\bar{x}_1$ and $\bar{x}_2$ are the mle's. Otherwise, the mle's are found on the border and the mle's for both $\mu_1$ and $\mu_2$ will be equal to $(m \bar{x}_1+n \bar{x}_2)/(m+n)$.
– JimB
May 15, 2020 at 4:11

The joint likelihood of the parameters is actually $$\mathcal L(\mu_1, \mu_2 \mid \boldsymbol x, \boldsymbol y) \propto \exp\left(-\frac{m}{2}\left((\bar x - \mu_1)^2 + \hat \sigma_x^2\right) - \frac{n}{2}\left((\bar y - \mu_2)^2 + \hat \sigma_y^2 \right) \right) \mathbb 1 (\mu_1 \le \mu_2)$$ where $$\bar x$$, $$\bar y$$ are the respective sample means, and $$\hat \sigma_x^2$$, $$\hat \sigma_y^2$$ are the (biased) sample variances. After removing the remaining multiplicative factors and taking the logarithm, we get the log-likelihood on the sufficient statistics $$\ell (\mu_1, \mu_2 \mid \bar x, \bar y) \propto -\frac{m(\bar x - \mu_1)^2 + n(\bar y - \mu_2)^2}{2} + \log \mathbb 1 (\mu_1 \le \mu_2).$$ Since the first term is always negative unless $$(\mu_1, \mu_2) = (\bar x, \bar y)$$, this choice, when possible (i.e. $$\bar x \le \bar y$$), yields the maximum likelihood. If $$\bar x > \bar y$$, then this choice is not possible because $$\log 0 = -\infty$$. By considering the transformation $$(\mu_1', \mu_2') = (\sqrt{m} \mu_1, \sqrt{n} \mu_2),$$ we seek to minimize the squared distance $$(\mu_1' - \sqrt{m}\bar x)^2 + (\mu_2' - \sqrt{n} \bar y)^2$$ subject to the constraint $$\mu_1'/\sqrt{m} \le \mu_2'/\sqrt{n}$$. As the level sets of this function of $$(\mu_1', \mu_2')$$ comprise circles centered at $$(\sqrt{m} \bar x, \sqrt{n} \bar y)$$, we observe that the smallest radius occurs for a circle tangent to the boundary of the transformed parameter space $$\mu_1'/\sqrt{m} = \mu_2'/\sqrt{n}.$$ Back-transforming shows that this corresponds to the boundary of the original parameter space, $$\mu_1 = \mu_2$$. Now we can solve for the constrained MLE via simple differentiation. The critical points satisfy $$0 = \frac{d \ell(\mu_1, \mu_1 \mid \bar x, \bar y)}{d \mu_1} = \mu_1 (m+n) - (m \bar x + n \bar y),$$ hence the MLE is $$(\hat \mu_1, \hat \mu_2) = \begin{cases} (\bar x, \bar y), & \bar x \le \bar y \\ \left(\frac{m \bar x + n \bar y}{m+n}, \frac{m \bar x + n \bar y}{m+n}\right), & \bar x > \bar y. \end{cases}$$