2
$\begingroup$

Suppose we have an observation vector $x$, where $f_X(x,\theta)$ is its probability distribution function which depends on the parameter $\theta$. The parameter $\theta$ is also a random variable. Under $H_0$ it is distributed as $p_{\theta_0}(\theta)$, and under $H_1$ it is distributed as $p_{\theta_1}(\theta)$. Now I want to construct a likelihood ratio test to decide between $H_0$ and $H_1$. Can I simply take expectations over $\theta_1$ and $\theta_0$ to construct my test? i.e. is the following test correct?

$$\frac{E_{\theta_1}[f_X(x,\theta)]}{E_{\theta_0}[f_X(x,\theta)]}\begin{array}{c} H_1 \\ > \\ < \\ H_0 \end{array} \lambda.$$

$\endgroup$
  • $\begingroup$ You want to use Bayes Theorem to convert f(x|theta) into an expression of f(theta|x). Then find whether f(theta_1|x) > f(theta_2|x). $\endgroup$ – NicNic8 Dec 31 '17 at 0:13
  • $\begingroup$ He/she could be interested in evaluating it in a more "frequentist" way, though I guess the Bayesian approach is more obvious. I'm messing around with the problem, it makes intuitive sense to me that integrating over $\theta$ with respect to the two distributions would make it more probable that $f_{X_0}(x) = \int_{\theta}f(x,\theta)P_{\theta_0}$ gives larger support to $x\sim \theta$ with $\theta \sim P_{\theta_0}$, and vice versa for $\theta \sim P_{\theta_1}$. There are other things you could do though, like picking $f(x|\hat{\theta})P_{\theta_0}(\hat{\theta})$ $\endgroup$ – Ryan Warnick Dec 31 '17 at 0:38
  • $\begingroup$ So it means that there is no optimal way to solve this problem, and it is just preference? $\endgroup$ – Mah Dec 31 '17 at 2:09
  • $\begingroup$ I'm writing up an answer, there's some relationships with what you are doing and some forms of hypothesis testing. $\endgroup$ – Ryan Warnick Dec 31 '17 at 2:37
1
$\begingroup$

The approach you mentioned is using Bayes factors to do inference on the model space.

Changing the notation a little bit, denote by $f(x|\theta) = P[X|\theta]$, and $f_{\theta_j}(\theta) = P[\theta| M_j]$ for $j \in \{0,1\}$. Where $M_j$ denotes one of the possible models to choose from.

Then $E_{\theta_j}[f(x|\theta)]$ is the same as $P[X|M_j] = \int_\theta P[X|\theta] P[\theta|M_j]d\theta$. Then your test statistic:

\begin{equation} \begin{split} t(X) & = \frac{E_{\theta_1}[f(x|\theta)]}{E_{\theta_0}[f(x|\theta)]}\\ & = \frac{P[X|M_1]}{P[X|M_0]} \end{split} \end{equation}

The goal then is to find $\lambda$ such that: \begin{equation} P_{M_0}[t(X) > \lambda] = P_{M_0}[P[X|M_1] > \lambda P[X|M_0]] \leq \alpha \end{equation}

while maximizing: \begin{equation} P_{M_1}[t(X) > \lambda] = P_{M_1}[P[X|M_1] > \lambda P[X|M_0]] \end{equation}

There are some other ways one could approach the problem. For instance, you could look at:

\begin{equation} \begin{split} t(X)& = \frac{\sup_\theta P[X|\theta]P[\theta|M_1]}{\sup_\theta P[X|\theta]P[\theta|M_0]} \end{split} \end{equation}

As a weighted likelihood ratio test, where $P[\theta | M_j]$ acts as a penalty or prior belief about what values of $\theta$ are more likely.

Because $P[X|\theta]P[\theta|M_j] \propto P[\theta|X,M_j]$, we actually are finding the Maximum-a-Posteriori (MAP) estimator.

Taking the log of the numerator and denominator in the previous expression:

\begin{equation} \begin{split} t(X)& = \frac{\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_1]}{\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_0]} \end{split} \end{equation}

We can see the MAP estimation problem is a form penalized maximum likelihood. We'd specify $\lambda$ to achieve a required significance level as in the Bayes factor approach.

It's not clear that the MAP estimation approach will be as effective of a test as the Bayes factor.

Scratch work to test this on a real problem Still kind of messing around with this, there are some algebra errors

To do an experiment, let $X|\theta \sim N(\theta,1)$ and $\theta|M_j \sim N(\mu_j,1)$.

Bayes Factor

First doing the Bayes factor approach $P[X|M_j] = \int_\theta N(X;\theta,1)N(\theta;\mu_j,1)d\theta \sim N(\mu_j,\sqrt{2})$.

This gives us that, letting $\lambda^* = \textrm{log}\lambda$: \begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] &= P_{M_0}[\textrm{log}P[X|M_1] -\textrm{log}P[X|M_0] > \lambda^*]\\ & = P_{M_0}[-\frac{1}{2}(X-\mu_1)^2 + \frac{1}{2}(X-\mu_0)^2 > \lambda^*]\\ & = P_{M_0}[-X^2+2\mu_1X - \mu_1^2 + X^2 - 2\mu_0X + \mu_0^2 > 2\lambda^*]\\ & = P_{M_0}[2X(\mu_1 - \mu_0) +(\mu_0^2 - \mu_1^2) > 2\lambda^*]\\ & = P_{M_0}[X > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{2(\mu_1 - \mu_0)}]\\ & = P_{M_0}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}]\\ \end{split} \end{equation}

Setting this equal to $\alpha$ we have:

\begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] &= P_{M_0}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \frac{\mu_0}{\sqrt{2}})} - \mu_0] = \alpha\\ & \Rightarrow 1 - P_{M_0}[Z < \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}] = \alpha\\ & \Rightarrow P_{M_0}[Z < \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_1 - \mu_0)} - \frac{\mu_0}{\sqrt{2}}] = 1-\alpha\\ & \Rightarrow \lambda^* = \sqrt{2}(\mu_1-\mu_0)(\Phi^{-1}(1-\alpha) + \frac{\mu_0}{\sqrt{2}}) + \frac{(\mu_0^2 - \mu_1^2)}{2} \end{split} \end{equation}

Now to compute the power: \begin{equation} \begin{split} P_{M_1}[t(X) > \lambda] &= P_{M_1}[Z > \frac{2\lambda^* - (\mu_0^2 - \mu_1^2)}{22(\mu_1 - \mu_0)} - \frac{\mu_1}{2}]\\ \Rightarrow P_{M_1}[t(X) > \lambda] & = 1 - \Phi(\sqrt{2}\Phi^{-1}(1-\alpha) + \frac{1}{2}(\mu_0 - \mu_1))\\ & = \Phi(\sqrt{2}\Phi^{-1}(\alpha) + \frac{1}{2}(\mu_1 - \mu_0)) \end{split} \end{equation}

MAP Estimator

Compute $\lambda$:

\begin{equation} \begin{split} P_{M_0}[t(X) > \lambda] & =P_{M_0}[\sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_1] > \lambda \sup_\theta \frac{1}{n}\sum_{i=1}^n\textrm{log}P[X_i|\theta]+\textrm{log}P[\theta|M_0]]\\ & =P_{M_0}[\sup_\theta \frac{1}{n}\sum_{i=1}^n -(X_i - \theta)^2 -(\theta - \mu_1)^2 > \lambda \sup_\theta \frac{1}{n}\sum_{i=1}^n -(X_i - \theta)^2 -(\theta - \mu_0)^2]\\ & =P_{M_0}[\inf_\theta \frac{1}{n}\sum_{i=1}^n X_i^2 - 2X_i\theta +\theta^2 +\theta^2 - 2\mu_1\theta + \mu_1^2 < \lambda \inf_\theta \frac{1}{n}\sum_{i=1}^n X_i^2 - 2X_i\theta +\theta^2 +\theta^2 - 2\mu_0\theta + \mu_0^2]\\ & =P_{M_0}[\inf_\theta \bar{X_i^2} - 2\bar{X_i}\theta +\theta^2 +\theta^2 - 2\mu_1\theta + \mu_1^2 < \lambda \inf_\theta \bar{X_i^2} - 2\bar{X_i}\theta +\theta^2 +\theta^2 - 2\mu_0\theta + \mu_0^2]\\ & =P_{M_0}[\bar{X^2}+\mu_1^2 -\frac{(\bar{X}+\mu_1)^2}{2} < \lambda +\bar{X^2}+\mu_0^2 -\frac{(\bar{X}+\mu_0)^2}{2} ]\\ & =P_{M_0}[\frac{(\bar{X}+\mu_0)^2}{2}-\frac{(\bar{X}+\mu_1)^2}{2} < \lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[\frac{1}{2}[2\bar{X}(\mu_0 - \mu_1) + \(\mu_0^2 - \mu_1^2)] < \lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[2\bar{X}(\mu_0 - \mu_1) < 2\lambda +\mu_0^2 - \mu_1^2 ]\\ & =P_{M_0}[\bar{X} < \frac{2\lambda +\mu_0^2 - \mu_1^2}{2(\mu_0 - \mu_1)} ]\\ & =P_{M_0}[Z < \frac{\sqrt{n}(2\lambda +\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_0 - \mu_1)} -\frac{\sqrt{n}\mu_0}{\sqrt{2}} ]\\ \end{split} \end{equation}

Solving for $\lambda$ which gives significance $\alpha$ we get: \begin{equation} \lambda = (\frac{\sqrt{2}}{\sqrt{n}}\Phi^{-1}(\alpha) + \mu_0)(\mu_0 - \mu_1) + \frac{(\mu_1^2 - \mu_0^2)}{2} \end{equation}

Now computing the power: \begin{equation} \begin{split} P_{M_1}[t(X) > \lambda] & =P_{M_1}[Z < \frac{\sqrt{n}(2\lambda +\mu_0^2 - \mu_1^2)}{\sqrt{2}2(\mu_0 - \mu_1)} -\frac{\sqrt{n}\mu_1}{\sqrt{2}} ]\\ & = \Phi^{-1}(\alpha) + \frac{\sqrt{n}}{\sqrt{2}}(\mu_0 - \mu_1) \end{split} \end{equation}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.