I am playing a computer game where many actions have a set chance to succeed communicated to the player as a percent chance of success.

I have began to suspect the the displayed chances do not reflect the real probability of success in the game (much like you might suspect that a dice or coin is weighted). In order to prove this systematically I have written out a table where each entry shows the game's displayed success chance for a given action and then the actual result (succeed or fail) of the action as I play.

   % Chance of Success |  Result |
                   50% | succeed |
                   60% |    fail |
                   20% |    fail |
                   80% | succeed |
                   ... |     ... |

A simple analysis of the data I've collected reveals that the actual chance of success is far lower than the displayed one, but I'm concerned that my sample size is too small to be statistically significant.

So, framing this as an experiment where the null hypothesis is that "the game's displayed success values represent the true probability of success" and my hypothesis is that they don't, how can I go about calculating how statistically significant (ideally a p value) my results are?

  • $\begingroup$ Not sure this is clear. How can the probability of the outcome $1$, say, be both $.5$ and $.8$? What, exactly, are you measuring? $\endgroup$ – lulu Oct 28 '18 at 11:55
  • $\begingroup$ I can edit to clarify. There are only two outcomes 0 or 1 the (you could think of them as success or failure, heads or tails etc. where the probability to the left on the table is the chance (from 0 to 1) that the outcome will be a 1. $\endgroup$ – Joel Collins Oct 28 '18 at 12:01
  • $\begingroup$ I still don't get it. Like I say, the probability of getting a $1$ can't be two different values. Maybe it would help if you wrote out exactly what you are doing. $\endgroup$ – lulu Oct 28 '18 at 12:10
  • $\begingroup$ Alright, I've figured out the misunderstanding. I'll edit the question to be more clear and concrete from the start. $\endgroup$ – Joel Collins Oct 28 '18 at 12:28

You might do a "test of value of population mean", also known as a "one sample t-test". First calculate the average number of successes you should experience based on the site's stated probability of success; just sum up the "chance of success" in your table and divide by the number of rows. That is the assumed mean of the population. Then compare the assumed mean with the average number of successes you actually experience, which is the sample mean. The t-test will tell you whether your result is statistically significant or not.

You can find the formula for the t-test in many places, either in a statistics textbook or online.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.