I'm interested in the following problem: I have a biased coin, which you measure, and make an estimate $p_{est}$ of the odds of getting a heads. I then flip the coin a random (say, Poisson(10)) number of times, and observe $n_H$ heads and $n_T$ tails. I'd like to "score" your prediction.

Current idea for a method: Use the prediction to set a Bayesian prior distribution on the real value of $p$. Then use the data $(n_H, n_T)$ to obtain a posterior distribution. Then use Kullback-Leibler to compare these distributions. What I wanted was a method which takes account of $n:=n_H+n_T$ which is somehow the amount of information I'm gaining from my experiment. If $n$ is small, then it seems reasonable to say that I learn less from the experiment, and so can assess your prediction less.

To give more details, I take a prior as $\operatorname{Beta}(tp_{est}, t(1-p_{est}))$ where $t>0$ somehow expresses my "confidence" in your prediction. As this is a conjugate prior to the binomial, my posterior is $\operatorname{Beta}(tp_{est}+n_H, t(1-p_{est})+n_T)$. I can then work out the Kullback-Leibler analytically, though it's rather messy.

Questions:

Is this idea in the literature?

There is an obvious dependence on $t$. If $t$ is very large, then the prior completely dominates the data, and the KL divergence is small. But what occurs for $t$ small seems complicated. How do we choose $t$ appropriately?

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.