Likelihood ratio with unknown parameters

Suppose we have an observation vector $x$, where $f_X(x,\theta)$ is its probability distribution function which depends on the parameter $\theta$. The parameter $\theta$ is also a random variable. Under $H_0$ it is distributed as $p_{\theta_0}(\theta)$, and under $H_1$ it is distributed as $p_{\theta_1}(\theta)$. Now I want to construct a likelihood ratio test to decide between $H_0$ and $H_1$. Can I simply take expectations over $\theta_1$ and $\theta_0$ to construct my test? i.e. is the following test correct?

$$\frac{E_{\theta_1}[f_X(x,\theta)]}{E_{\theta_0}[f_X(x,\theta)]}\begin{array}{c} H_1 \\ > \\ < \\ H_0 \end{array} \lambda.$$

• You want to use Bayes Theorem to convert f(x|theta) into an expression of f(theta|x). Then find whether f(theta_1|x) > f(theta_2|x). – NicNic8 Dec 31 '17 at 0:13
• He/she could be interested in evaluating it in a more "frequentist" way, though I guess the Bayesian approach is more obvious. I'm messing around with the problem, it makes intuitive sense to me that integrating over $\theta$ with respect to the two distributions would make it more probable that $f_{X_0}(x) = \int_{\theta}f(x,\theta)P_{\theta_0}$ gives larger support to $x\sim \theta$ with $\theta \sim P_{\theta_0}$, and vice versa for $\theta \sim P_{\theta_1}$. There are other things you could do though, like picking $f(x|\hat{\theta})P_{\theta_0}(\hat{\theta})$ – Ryan Warnick Dec 31 '17 at 0:38
• So it means that there is no optimal way to solve this problem, and it is just preference? – Mah Dec 31 '17 at 2:09
• I'm writing up an answer, there's some relationships with what you are doing and some forms of hypothesis testing. – Ryan Warnick Dec 31 '17 at 2:37

The approach you mentioned is using Bayes factors to do inference on the model space.

Changing the notation a little bit, denote by $f(x|\theta) = P[X|\theta]$, and $f_{\theta_j}(\theta) = P[\theta| M_j]$ for $j \in \{0,1\}$. Where $M_j$ denotes one of the possible models to choose from.

Then $E_{\theta_j}[f(x|\theta)]$ is the same as $P[X|M_j] = \int_\theta P[X|\theta] P[\theta|M_j]d\theta$. Then your test statistic:

\begin{equation} \begin{split} t(X) & = \frac{E_{\theta_1}[f(x|\theta)]}{E_{\theta_0}[f(x|\theta)]}\\ & = \frac{P[X|M_1]}{P[X|M_0]} \end{split} \end{equation}

The goal then is to find $\lambda$ such that: \begin{equation} P_{M_0}[t(X) > \lambda] = P_{M_0}[P[X|M_1] > \lambda P[X|M_0]] \leq \alpha \end{equation}

while maximizing: \begin{equation} P_{M_1}[t(X) > \lambda] = P_{M_1}[P[X|M_1] > \lambda P[X|M_0]] \end{equation}

There are some other ways one could approach the problem. For instance, you could look at:

\begin{equation} \begin{split} t(X)& = \frac{\sup_\theta P[X|\theta]P[\theta|M_1]}{\sup_\theta P[X|\theta]P[\theta|M_0]} \end{split} \end{equation}

As a weighted likelihood ratio test, where $P[\theta | M_j]$ acts as a penalty or prior belief about what values of $\theta$ are more likely.

Because $P[X|\theta]P[\theta|M_j] \propto P[\theta|X,M_j]$, we actually are finding the Maximum-a-Posteriori (MAP) estimator.

Taking the log of the numerator and denominator in the previous expression:

\begin{equation} \begin{split} t(X)& = \frac{\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_1]}{\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_0]} \end{split} \end{equation}

We can see the MAP estimation problem is a form penalized maximum likelihood. We'd specify $\lambda$ to achieve a required significance level as in the Bayes factor approach.

It's not clear that the MAP estimation approach will be as effective of a test as the Bayes factor.

Scratch work to test this on a real problem Still kind of messing around with this, there are some algebra errors

To do an experiment, let $X|\theta \sim N(\theta,1)$ and $\theta|M_j \sim N(\mu_j,1)$.

Bayes Factor

First doing the Bayes factor approach $P[X|M_j] = \int_\theta N(X;\theta,1)N(\theta;\mu_j,1)d\theta \sim N(\mu_j,\sqrt{2})$.

This gives us that, letting $\lambda^* = \textrm{log}\lambda$: \begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] &= P_{M_0}[\textrm{log}P[X|M_1] -\textrm{log}P[X|M_0] > \lambda^*]\\ & = P_{M_0}[-\frac{1}{2}(X-\mu_1)^2 + \frac{1}{2}(X-\mu_0)^2 > \lambda^*]\\ & = P_{M_0}[-X^2+2\mu_1X - \mu_1^2 + X^2 - 2\mu_0X + \mu_0^2 > 2\lambda^*]\\ & = P_{M_0}[2X(\mu_1 - \mu_0) +(\mu_0^2 - \mu_1^2) > 2\lambda^*]\\ & = P_{M_0}[X > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{2(\mu_1 - \mu_0)}]\\ & = P_{M_0}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}]\\ \end{split} \end{equation}

Setting this equal to $\alpha$ we have:

\begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] &= P_{M_0}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \frac{\mu_0}{\sqrt{2}})} - \mu_0] = \alpha\\ & \Rightarrow 1 - P_{M_0}[Z < \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}] = \alpha\\ & \Rightarrow P_{M_0}[Z < \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}] = 1-\alpha\\ & \Rightarrow \lambda^* = \sqrt{2}(\mu_1-\mu_0)(\Phi^{-1}(1-\alpha) + \frac{\mu_0}{\sqrt{2}}) + \frac{(\mu_0^2 - \mu_1^2)}{2} \end{split} \end{equation}

Now to compute the power: \begin{equation} \begin{split} P_{M_1}[t(X) > \lambda] &= P_{M_1}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{22(\mu_1 - \mu_0)} - \frac{\mu_1}{2}]\\ \Rightarrow P_{M_1}[t(X) > \lambda] & = 1 - \Phi(\sqrt{2}\Phi^{-1}(1-\alpha) + \frac{1}{2}(\mu_0 - \mu_1))\\ & = \Phi(\sqrt{2}\Phi^{-1}(\alpha) + \frac{1}{2}(\mu_1 - \mu_0)) \end{split} \end{equation}

MAP Estimator

Compute $\lambda$:

\begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] & =P_{M_0}[\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_1] > \lambda \sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_0]]\\ & =P_{M_0}[\sup_\theta \frac{1}{n}\sum_{i=1}^n -(X_i - \theta)^2 -(\theta - \mu_1)^2 > \lambda \sup_\theta \frac{1}{n}\sum_{i=1}^n -(X_i - \theta)^2 -(\theta - \mu_0)^2]\\ & =P_{M_0}[\inf_\theta \frac{1}{n}\sum_{i=1}^n X_i^2 - 2X_i\theta +\theta^2 +\theta^2 - 2\mu_1\theta + \mu_1^2 < \lambda \inf_\theta \frac{1}{n}\sum_{i=1}^n X_i^2 - 2X_i\theta +\theta^2 +\theta^2 - 2\mu_0\theta + \mu_0^2]\\ & =P_{M_0}[\inf_\theta \bar{X_i^2} - 2\bar{X_i}\theta +\theta^2 +\theta^2 - 2\mu_1\theta + \mu_1^2 < \lambda \inf_\theta \bar{X_i^2} - 2\bar{X_i}\theta +\theta^2 +\theta^2 - 2\mu_0\theta + \mu_0^2]\\ & =P_{M_0}[\bar{X^2}+\mu_1^2 -\frac{(\bar{X}+\mu_1)^2}{2} < \lambda +\bar{X^2}+\mu_0^2 -\frac{(\bar{X}+\mu_0)^2}{2} ]\\ & =P_{M_0}[\frac{(\bar{X}+\mu_0)^2}{2}-\frac{(\bar{X}+\mu_1)^2}{2} < \lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[\frac{1}{2}[2\bar{X}(\mu_0 - \mu_1) + \(\mu_0^2 - \mu_1^2)] < \lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[2\bar{X}(\mu_0 - \mu_1) < 2\lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[\bar{X} < \frac{2\lambda +\mu_0^2 - \mu_1^2}{2(\mu_0 - \mu_1)} ]\\ & =P_{M_0}[Z < \frac{\sqrt{n}(2\lambda +\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_0 - \mu_1)} -\frac{\sqrt{n}\mu_0}{\sqrt{2}} ]\\ \end{split} \end{equation}

Solving for $\lambda$ which gives significance $\alpha$ we get: \begin{equation} \lambda = (\frac{\sqrt{2}}{\sqrt{n}}\Phi^{-1}(\alpha) + \mu_0)(\mu_0 - \mu_1) + \frac{(\mu_1^2 - \mu_0^2)}{2} \end{equation}

Now computing the power: \begin{equation} \begin{split} P_{M_1}[t(X) > \lambda] & =P_{M_1}[Z < \frac{\sqrt{n}(2\lambda +\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_0 - \mu_1)} -\frac{\sqrt{n}\mu_1}{\sqrt{2}} ]\\ & = \Phi^{-1}(\alpha) + \frac{\sqrt{n}}{\sqrt{2}}(\mu_0 - \mu_1) \end{split} \end{equation}