I have some data that looks like this:

Prob    Outcome
0.09    0
0.10    0
0.10    0
0.11    1
0.84    1
0.99    1
0.86    1
0.78    1
0.86    1
0.00    0

i.e. a bunch a probabilities each with a single test. What statisitcal test should I use to test the hypothesis that the probabilities are correct?

Further details: The data points are combat probabilities from the game Civilization IV, and I have over 3000 of them in my set. Thus, each probaility is generated using some unknown formula from different input data, depending on the relative strengths of the units in that battle.

It has been suggested that the outcomes do not accurately reflect the probabilities given: for instance, the computer player wins more often that it should, based on the probabilites displayed, which is what we want to test.

So there is a link insofar as we assume the probabilities displayed are generated using the same formula for each line. It's this unknown formula that we want to test for consistency with the actual results.

  • $\begingroup$ Are the tests independent ? $\endgroup$ – Stéphane Laurent May 2 '13 at 16:53
  • $\begingroup$ @Stéphane I think so, if I understand "independent" correctly. The outcomes are calculated using some formula by a computer, which also reports a probability for each test - I want to check if the reported probabilities match the results being given. $\endgroup$ – Luigi Plinge May 2 '13 at 16:57
  • $\begingroup$ If there's no link between the tests I'm afraid this is not possible. That would mean that the first line of your data is the outcome of one experiment with "expected" probability $0.09$. The second is the outcome of another experiment, independent of the first one, with expected probability $0.1$. And so on. So your problem is "linewise" (there is an experiment at each line, and the experiments are independent from each other), but at each line you only have one outcome hence you cannot test the hypothesis. $\endgroup$ – Stéphane Laurent May 2 '13 at 18:15
  • $\begingroup$ Are there some "ties" in the first column ? Actually you should describe more precisely how these data are generated. $\endgroup$ – Stéphane Laurent May 2 '13 at 18:17
  • $\begingroup$ @Stéphane I've added some more details... I hope this is clearer. $\endgroup$ – Luigi Plinge May 2 '13 at 18:55

This is essentially the problem posed by Jaynes as "the honest weatherman" in chapter 13 of his book PROBABILITY THEORY: THE LOGIC OF SCIENCE. https://classes.soe.ucsc.edu/ams221/Spring13/jaynes-2003-pages-397-through-425.pdf (Imagine your first column is predicted probabilities of rain, and the second column is 1 if it rained, 0 otherwise).

I.e. lots of individual experiments each with a different probability. How are we to assess the weatherman's performance? You can use a measure based on the entropy, defined as SUM(p*log(p)).

To cut a long story short, for each 1 score log(2p) and for each 0 score log(2(1-p)) where p is the probability that was predicted, and the log is to base 2.

Sum all the scores (call the sum S). A positive total means a performance better than chance, and a negative total means a performance worse than chance. A 'perfect' score would be N where N is the number of trials, so S/N is a measure of performance.

You should disallow p=0 and p=1, round them to say 0.01 and 0.99, or an incorrect prediction will blow up.

-- Added 23.05 BST on 4 June 2015 --

Luigi asks how to derive a test hypothesis in the 'weatherman' problem. Jaynes does not cover this, but I will try to answer in the spirit of Jaynes.

First, it does not make sense to assess the evidence for or against a single hypothesis. One must always compare two or more hypotheses. Let's call the probabilities $\mathbf{q}$ and the outcomes $\mathbf{r}$. The $\mathbf{q}$ represent a prediction (or model) of the weather (or whatever) and the $\mathbf{r}$ are the evidence. We must compare the $\mathbf{q}$ with an alternative model, say $\mathbf{q_0}$. A good choice for $\mathbf{q_0}$ is the model that has no knowledge whatever concerning the outcomes, and this is the model where $q_i = 0.5$ for all $i$. Then the Likelihood Ratio comparing these two hypotheses is $$ LR = \frac{P(\mathbf{r}|\mathbf{q})}{P(\mathbf{r}|\mathbf{q_0})} = \frac{\prod\limits_1 q_i \prod\limits_0 (1-q_i)}{0.5^N} $$ where $\prod\limits_1$ is a product over trials where $r_i = 1$ and $\prod\limits_0$ is a product over trials where $r_i = 0$, and $N$ is the number of trials. Now take logs to base 2 to get the log-likelihood-ratio: $$ LLR_2 = \sum\limits_1 {\log_2 q_i} + \sum\limits_0 {\log_2 (1 - q_i)} + N $$ This can also be written $\sum\limits_1 {\log_2 (2(q_i))} + \sum\limits_0 {\log_2 (2(1 - q_i))}$ which is the S of my original reply. However it is more usual to write a LLR to base 10: $$ LLR_{10} = \sum\limits_1 {\log_{10} q_i} + \sum\limits_0 {\log_{10} (1 - q_i)} + N \log_{10} 2 $$ Thus $LLR_{10}$ directly represents the "weight of evidence" for $\mathbf{q}$ over $\mathbf{q_0}$ measured in 'bans', a unit coined by Alan Turing and I.J. Good at Bletchley Park in 1940. A 'weight of evidence' of 0 means the prediction is no better than $\mathbf{q_0}$, while a value of say 3 means the evidence is $10^3 = 1000$ times more likely under $\mathbf{q}$ than $\mathbf{q_0}$. According to the likelihood principle, that is all that is relevant. You do not need to worry about the distribution of the LLR. For your figures I get $LLR_{10} = 1.6$ which represents odds of about 40:1 in favour of $\mathbf{q}$ over $\mathbf{q_0}$.

Personally I would want the weatherman to do substantially better than beating $\mathbf{q_0}$ before I would pay for his forecast. He should be able to beat a simple prediction such as assigning a constant probability $p$ for rain on all days, based on historial data for the season and place; or assigning a fixed $p$ for tomorrow's weather being the same as today's. You can compare any such hypothesis with $\mathbf{q_0}$ as above, and compare any pair of hypotheses by subtracting LLRs.

If you want to calculate a p-value you might look at the method suggested here: Statistics: How to measure how accurately probabilities are reported? . But why would you want to? I recommend that instead you read Jaynes until the feeling goes away :-)

  • $\begingroup$ Thanks, that sounds promising, and you describe how to determine a test statistic, but I wonder how now to derive an appropriate test hypothesis? I guess we'd expect a normal distribution with mean zero, but what's the variance? $\endgroup$ – Luigi Plinge Jun 3 '15 at 20:18
  • $\begingroup$ @luigi-plinge : That's a good question. It turns out that the score S can be interpreted as a LLR (log-likelihood ratio) for two hypotheses H1: the data (outcomes)were drawn from the probabilities. H2: the data were drawn from an alternative set of probabilities all equal to 0.5. I will post a proof later. $\endgroup$ – gareth Jun 4 '15 at 13:25

All right. This is probably not an "official" statistic tests. However

Consider this test.

For each entry, if the outcome is zero, add (1-prob) to the score. If the outcome is 1, add (prob) to the score.

Divide the final score by the number of entries.

There is a better method where you add a function of (prob) to the score. I don't remember what that function is.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.