0
$\begingroup$

I'm not sure if this is the right place to publish this question, but I am trying to understand something about hypothesis testing. recently in a lecture on the subject a professor emphasized how people can "cheat" at hypothesis testing to get the results they want by adjusting an alpha value or by continuing to take samples until their data matched the results they wanted.. I obviously want to avoid doing that.

i am working on designing a computer program that simulates a simple card game where the player draws three cards, if one of the cards is a 10, or he draws all odd numbered cards, he wins.

My question is this.. i can calculate the probability of the player winning on paper to be .3833 but how can i test that against the simulation? Lets say the program runs some large number of times, and i calculate the mean of the probability of the player winning to be some number x. If i run a two sided hypothesis test with alpha being .05, and the null gets rejected, why is it wrong to then run the test again with alpha = .1 to see what certainty level the test would fail to reject at? in other words, the program won't spit out the exact .3833 every time, so i want to prove the correctness of my program using a hypothesis test to see how close or far off it might be.

$\endgroup$
0
$\begingroup$

First off you need to be really clear about what your null hypothesis is. What exactly are you trying to establish? Is it that the mean is $0.3833$?
And once you do a test you should get a $p$-value, so there's no reason to do it again at another $\alpha$. Because $p$ is less than $\alpha$ for all $\alpha$ greater than $p$.

Maybe it would be useful to calculate the empirical standard deviation of your simulation to get an idea of the accuracy. Then you can compute the sample mean and an empirical confidence interval.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.