Suppose we have some random variable $X$ that ranges over some sample space $S$. We also have two probability models $F$ and $G$. Let $f(x)$ and $g(x)$ be the probability density functions for these distributions. Does the following quantity $$ \log \frac{f(x)}{g(x)} = \log \frac{P(F|x)}{P(G|x)}- \log \frac{P(F)}{P(G)}$$ basically tell us how much more likely model $F$ is the true model than model $G$?
Tell me more
×
Mathematics Stack Exchange is a question and answer site for
people studying math at any level and professionals in related fields. It's 100% free, no registration required.
|
|
I am totally confused by the last comment made by Michael (the answer is ok, it is the link with logistic regression which went too far for me). Logistic regression is to be used when you have pairs of observations (X, Y) where Y is a binary variable (taking values in {0,1}) which is modeled as a Bernoulli variable $\mathcal{B}(p)$ the parameter of which depends of the value $x$ taken by $X$ : $\mathrm{logit}(p) = \beta_0 + \beta_1 x$. Here you don’t observe a variable Y taking value 1 when the model if F and 0 when it is G, the model is fixed beforehand and would not change along the observations... and you wouldn’t write $\mathrm{logit} P(F) = \beta_0 + \beta_1 x$. To me, this doesn‘t make any sense. I will slightly reword Michael’s answer, just to give you some additionnal keywords. If you have a single observation $x$, then $f(x)$ is the likelihood of the model F, denote it by $L(F; x) = f(x)$, and $g(x)$ is the likelihood of the model G, denote it by $L(G;x) = g(x)$. As you stated, the likelihood ratio $L(F ; x)/L(G ;x) = f(x)/g(x)$ tells you how much the data support F against G. If you have prior probabilities for F and G, denoted by P(F) and P(G) = 1 - P(F), then you can write posterior probabilities P(F|x) and P(G|x). You have $$ P(F | x) = { L(F ; x) P(F) \over L(F;x) P(F) + L(G;x) P(G)},$$ $$ P(G | x) = { L(G ; x) P(G) \over L(F;x) P(F) + L(G;x) P(G)},$$ and $$ {P(F | x) \over P(G |x) } = {P(F) \over P(G)} \times {L(F ; x) \over L(G ;x)}.$$ This is as Michael stated an application of Bayes’ theorem. The quantity P(F)/P(G) = P(F)/(1-P(F)) is called the odds of the model F. You can take the log of this last equality to get an additive statement, which is very usual (cf Michael’s answer). The quantity L(F;x)/L(G;x) is called a Bayes factor. If you have $n$ independent observations $\mathbf{x} = x_1, \dots, x_n$, the same thing holds with $L(F ; \mathbf{x}) = \prod_i f(x_i)$ and $L(G ; \mathbf{x}) = \prod_i g(x_i)$. |
||||
|
|
I was slightly puzzled by the notation, but I'm assuming that by $P(F)$ you mean the probability that $F$ is the right model, and $P(F\mid x)$ is the conditional probability that $F$ is the right model given the event $X=x$. The identity you write is then a form of a special case of what is sometimes called Bayes' theorem. If one assumes the right model must be either $F$ or $G$, but not both, then one can say that $$ \operatorname{logit} P(F) = \log \frac{P(F)}{1-P(F)} = \log(\operatorname{odds}(F)) $$ increases by $\log(f(x)/g(x))$ when it is observed that $X=x$. So that quantity tells you by how much the logit of the probability increases when you observe the data. |
|||||
|
