# Scoring “predictions” using Kullback-Leibler

I'm interested in the following problem: I have a biased coin, which you measure, and make an estimate $p_{est}$ of the odds of getting a heads. I then flip the coin a random (say, Poisson(10)) number of times, and observe $n_H$ heads and $n_T$ tails. I'd like to "score" your prediction.

Current idea for a method: Use the prediction to set a Bayesian prior distribution on the real value of $p$. Then use the data $(n_H, n_T)$ to obtain a posterior distribution. Then use Kullback-Leibler to compare these distributions. What I wanted was a method which takes account of $n:=n_H+n_T$ which is somehow the amount of information I'm gaining from my experiment. If $n$ is small, then it seems reasonable to say that I learn less from the experiment, and so can assess your prediction less.

To give more details, I take a prior as $\operatorname{Beta}(tp_{est}, t(1-p_{est}))$ where $t>0$ somehow expresses my "confidence" in your prediction. As this is a conjugate prior to the binomial, my posterior is $\operatorname{Beta}(tp_{est}+n_H, t(1-p_{est})+n_T)$. I can then work out the Kullback-Leibler analytically, though it's rather messy.

Questions:

Is this idea in the literature?

There is an obvious dependence on $t$. If $t$ is very large, then the prior completely dominates the data, and the KL divergence is small. But what occurs for $t$ small seems complicated. How do we choose $t$ appropriately?