If you roll a six sided die a bunch of times, and count how many times the number 1 shows up, you'd expect it to show up about 1/6 of the time.
Now if you roll this die 1000 times, and the number 1 shows up 600 times, you'd know something is amiss (maybe somebody tampered with the die)
I'm designing an AI for a computer game, and part of this AI requires it to try to calculate the probability of some binary (true / false) event occurring. After having calculated the probability, it gets feedback on the outcome of this probability event.
The catch is the program's prediction may be wrong -- for example, it might falsely calculate an event to have a 1/6 chance of occurring, while the actual probability of it occurring is 0.6.
How can I determine to what extent is the probability engine accurate? For instance, for the event that should occur with probability 0.6, the engine reporting the probability to be 0.7 would be rated much more accurate than one reporting the probability as 0.1.
Here's a sample of my data. So if the engine is accurate, 90% of the entries of the value 0.90 should be in the success category, while only 10% of them in the failure category.