Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Suppose we have some random variable $X$ that ranges over some sample space $S$. We also have two probability models $F$ and $G$. Let $f(x)$ and $g(x)$ be the probability density functions for these distributions. Does the following quantity $$ \log \frac{f(x)}{g(x)} = \log \frac{P(F|x)}{P(G|x)}- \log \frac{P(F)}{P(G)}$$ basically tell us how much more likely model $F$ is the true model than model $G$?

share|improve this question
    
What is the difference between $\log\frac{f(x)}{g(x)}$ and the log likelihood ratio (LLR)? The value of the LLR does not depend on the a priori probabilities $P(F)$ and $P(G)$ at all. In other words, there is cancellation on the right side of your displayed equation which makes the a priori probabilities disappear, and the LLR does not have any information about the a priori probabilities. –  Dilip Sarwate Dec 20 '11 at 18:53

2 Answers 2

I am totally confused by the last comment made by Michael (the answer is ok, it is the link with logistic regression which went too far for me). Logistic regression is to be used when you have pairs of observations (X, Y) where Y is a binary variable (taking values in {0,1}) which is modeled as a Bernoulli variable $\mathcal{B}(p)$ the parameter of which depends of the value $x$ taken by $X$ : $\mathrm{logit}(p) = \beta_0 + \beta_1 x$. Here you don’t observe a variable Y taking value 1 when the model if F and 0 when it is G, the model is fixed beforehand and would not change along the observations... and you wouldn’t write $\mathrm{logit} P(F) = \beta_0 + \beta_1 x$. To me, this doesn‘t make any sense.

I will slightly reword Michael’s answer, just to give you some additionnal keywords. If you have a single observation $x$, then $f(x)$ is the likelihood of the model F, denote it by $L(F; x) = f(x)$, and $g(x)$ is the likelihood of the model G, denote it by $L(G;x) = g(x)$. As you stated, the likelihood ratio $L(F ; x)/L(G ;x) = f(x)/g(x)$ tells you how much the data support F against G.

If you have prior probabilities for F and G, denoted by P(F) and P(G) = 1 - P(F), then you can write posterior probabilities P(F|x) and P(G|x). You have $$ P(F | x) = { L(F ; x) P(F) \over L(F;x) P(F) + L(G;x) P(G)},$$ $$ P(G | x) = { L(G ; x) P(G) \over L(F;x) P(F) + L(G;x) P(G)},$$ and $$ {P(F | x) \over P(G |x) } = {P(F) \over P(G)} \times {L(F ; x) \over L(G ;x)}.$$ This is as Michael stated an application of Bayes’ theorem. The quantity P(F)/P(G) = P(F)/(1-P(F)) is called the odds of the model F. You can take the log of this last equality to get an additive statement, which is very usual (cf Michael’s answer). The quantity L(F;x)/L(G;x) is called a Bayes factor.

If you have $n$ independent observations $\mathbf{x} = x_1, \dots, x_n$, the same thing holds with $L(F ; \mathbf{x}) = \prod_i f(x_i)$ and $L(G ; \mathbf{x}) = \prod_i g(x_i)$.

share|improve this answer
    
I've deleted the logistic regression remark. In logistic regression there are multiple observed instances of success or failure, each corresponding to an observed $x$ value. In this case, there is only one one binary choice, between $F$ and $G$, and its outcome is not observed, but the conditional distribution of $X$ given $F$ or $G$ is known, and the conditional probabilities of $F$ and $G$ are sought. –  Michael Hardy Dec 20 '11 at 22:09

I was slightly puzzled by the notation, but I'm assuming that by $P(F)$ you mean the probability that $F$ is the right model, and $P(F\mid x)$ is the conditional probability that $F$ is the right model given the event $X=x$.

The identity you write is then a form of a special case of what is sometimes called Bayes' theorem.

If one assumes the right model must be either $F$ or $G$, but not both, then one can say that $$ \operatorname{logit} P(F) = \log \frac{P(F)}{1-P(F)} = \log(\operatorname{odds}(F)) $$ increases by $\log(f(x)/g(x))$ when it is observed that $X=x$.

So that quantity tells you by how much the logit of the probability increases when you observe the data.

share|improve this answer
    
So $\text{logit} \ P(F) = \log \frac{P(F|x)}{P(G|x)} - \log \frac{f(x)}{g(x)}$. The first term plays the term of the intercept right? We need it to be of the form $\text{logit}\ P(F) = \beta_0+ \beta_{1}x$? –  Tommy Dec 20 '11 at 3:26
    
Actually, $\operatorname{logit}(P(F\mid X=x))$ would be a function of $x$ that depends on what probability distributions $X$ has under $F$ and $G$. –  Michael Hardy Dec 21 '11 at 15:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.