# Multiple MLE's for a single parameter of multivariate likelihood function

Suppose I have a multivariate likelihood function, $L(\theta ,\lambda |\mathbf{x})$, where $\theta$ can take on continuous values, but $\lambda$ can only take 'count' values $(0,1,2,...)$, and $\mathbf{x}$ is the data. The MLE of $\lambda$ has 2 solutions, $\hat{\lambda}=11$ and $\hat{\lambda}=12$, since the the likelihood is equally greatest for those two $\lambda$ values, irrespective of the value of $\theta$.

To estimate the asymptotic standard error of $\theta$, I'm using the Fisher Information at the ML estimates (i.e. $\sqrt{I^{-1}(\hat{\theta} ,\hat{\lambda})^{11}}$ to estimate $se(\hat{\theta})$), but this is a function of $\hat{\lambda}$... which does not have a unique value!

Any ideas what to do in this case to find $se(\hat{\theta})$ using the Fisher information?

• I can't help you find something I don't think exists. If we can't differentiate the likelihood with respect $\lambda$, what right do we have to speak of the Fisher information? Commented Apr 28, 2018 at 4:39
• It is certainly possible there is no solution. But element (1,1) of the inverted Fisher Information does not depend on element (2,2) of the (non-inverted) Fisher Information, the latter of which involves the second differential wrt lambda (the value in the numerator cancels with the denominator). Since element (1,1) of the inverted Fisher information depends on lambda, I need to 'impute' a value for lambda--this would ordinarily would be the MLE of lambda--to get the approximate std error of theta. Commented Apr 28, 2018 at 5:07
• How can you pretend to invert a matrix when three (the mixed partials will also be a problem) of its four entries are undefined? Commented Apr 28, 2018 at 5:23
• I think the asymptotic normality theory for MLEs requires additional regularity assumptions that the parameters have a continuous support. If you know what $\lambda_{11}$ and $\lambda_{12}$ are then you could plug them in and evaluate the fisher information of the MLEs and compare their relative asymptotic efficiencies. This equates to a model selection problem of $\lambda_{11}$ versus $\lambda_{12}$, using the fisher informations as criteria for comparison. You need a measure for defining which is "better", for example smaller trace or determinant, to have a well defined comparison. Commented Apr 29, 2018 at 14:29

You can't calculate Fisher information on discrete spaces. I very briefly looked up to see if there was any sort of "first" or "second differences" Fisher information matrix for discrete spaces, and did not find anything. Fortunately I found these notes which I'll come back to later

An alternative is to evaluate the Fisher information only on $$\mathcal{I}(\theta) = E_{X}[(\frac{\partial}{\partial \theta}\mathcal{L}(\theta | X, \lambda))^2]$$ for a specific $$\lambda$$; where $$\mathcal{L}(\theta | X, \lambda)$$ denotes the log likelihood.

Supposing that $$\lambda \in \Lambda$$ and $$|\Lambda| = k$$ for some $$k\in\mathbb{Z}^+$$, then the problem equates to a model selection problem of which $$\lambda$$ gives the "best" Fisher information, with some appropriate definition of best.

Some ways to compare the Fisher information could be whichever has a smaller determinant, or whichever has the smaller trace, or smaller maximal eigenvalue; some notions of the "size" of a covariance matrix. Just calculate the determinant or whatever of each of the Fisher informations for $$\lambda \in \Lambda$$ and find which one is best.

If $$|\Lambda| = \infty$$ is a countably infinite set, then you can still maybe find a solution of the "loss function" (determinant, trace, etc.) are almost surely convex over the Fisher information matrices defined over $$\Lambda$$.

Letting $$f(\mathcal{I}(\theta)): \mathbb{R}^{p\times p} \rightarrow \mathbb{R}^+$$ be the loss function over information matrices. If it is true that $$\forall \lambda_a,\lambda_b \in \Lambda$$ and $$c \in [0,1]$$, denoting by $$\mathcal{I}(\theta)_{\lambda_{a}}$$ and $$\mathcal{I}(\theta)_{\lambda_{b}}$$ the corresponding information matrices for $$\lambda_a$$ and $$\lambda_b$$, that we have $$P_X$$ almost surely:

$$$$f(c\mathcal{I}(\theta)_{\lambda_{a}} + (1-c)\mathcal{I}(\theta)_{\lambda_{b}}) \leq cf(\mathcal{I}(\theta)_{\lambda_{a}}) + (1-c) f(\mathcal{I}(\theta)_{\lambda_{b}})$$$$

Then you can do a greedy search over $$\lambda$$ to find the best value conditional on whatever $$f$$ you chose as your loss function. I would imagine some loss functions satisfy this, but maybe not, I honestly am not too familiar with that kind of thing.

Of course, if you know what $$\lambda_{11}$$ and $$\lambda_{12}$$ are, then you can just evaluate them and compare.

A more interesting idea that I found out about while coming up with an answer for this question is given in Equation 4.3 of the previously linked research article. That paper has a section which describes the Fisher Information Approximation (FIA) as a model selection problem; which works if you frame the problem as choosing whichever $$\lambda$$ has the most desirable statistical properties.

Like AIC and BIC, this criteria has a penalty for models which are overly complex in the number of parameters; which doesn't matter in this context because all models have the same dimension of $$\theta \in \Theta$$. However, the FIA has an additional penalization term involving the Fisher information which is equivalent to:

$$$$\log\int_{\theta \in \Theta}\sqrt{\textrm{det}[\mathcal{I}(\theta)_{\lambda}]}$$$$

which they term the "geometric complexity." I thought I would try an write out a synopsis of this penalty, but it gets into some fairly technical experimental design and information geometry fairly quickly. Essentially, it seems that the minimizer of the determinant of the Fisher information find the most D-optimal model (as a function of $$\lambda$$); which (I'm kind of beyond my depth here) "results in maximizing the differential Shannon information content of the parameter estimates."

Additionally, the log term allows the penalty term to actually incentivize a model if it gives that the interactions among the elements of the MLE are "favorable" to estimation; which would be the case if $$\int_{\theta \in \Theta}\sqrt{\textrm{det}[\mathcal{I}(\theta)_{\lambda}]} < 1$$.

In a lot of ways this is not dissimilar to just finding the Fisher information which minimizes $$Q(\lambda) = \log\int_{\theta \in \Theta}\sqrt{\textrm{det}[\mathcal{I}(\theta)_{\lambda}]}$$ as your loss function, and because $$\lambda_{11}$$ and $$\lambda_{12}$$ both give the same value in the likelihood, and $$\theta_{\lambda_{11}}$$ and $$\theta_{\lambda_{12}}$$ are both of equal dimension, the terms separate from the geometric complexity term are equal. This means that using $$Q(\lambda)$$ as your loss is equivalent to minimizing the FIA.

The loss function interpretation is more straightforward, but I think the model selection criteria approach helps elucidate why using $$Q(\lambda)$$ as a loss is a better idea than just, for example, looking at the maximal variance of the inverse Fisher informations, or some other less specific method of comparison.