Sampling $1000$ birds from a population which has $10$ types of birds the expected outcome is $100$ birds of each type (this is for the sake of simplicity; general case is each distributed with Probability $P_i$ where $i = 1..10$).

Now I have written a program to randomly sample $1000$ numbers (read birds) with each type with equal probability (or probability $P_i$ where $i = 1..10$ in general case)

Let say expected outcome is $E$ and during any run of program the observed outcome is $S$. Now to write a unit test case for this program, I have performed Chi-Squared test to compared $S$ to $E$ and asserted that p-value of Chi-statistic is greater than 0.05. I have asserted this $n$ (say $100$) times in a loop.

It would not be correct to expect each of $n$ p-values to be $>0.05$. So is it a good idea to assert that "average of" $n$ p-values to be $>0.05$.

More generally, what is the right approach in designing tests here?

Is Kolmogorov-Smirnov two-sample test a better test here?

  • $\begingroup$ Under the null hypothesis and using 5% level, the p.value should be less than .05, 5% of the time. $\endgroup$ – BruceET Mar 1 '16 at 17:20
  • $\begingroup$ Thanks BruceET. Do you think that getting 95% of $n$ p-values to be $> 0.05$ is a robust test in above case. Here $n >= 100$ (i.e., large) $\endgroup$ – Gerry Mar 6 '16 at 23:03
  • $\begingroup$ Sorry, not sure. Chi-squared GOF tests sometimes behave strangely when you have so much data. I'm sure of my "Comment," but if I was sure I had good advice for the entire problem, I'd have given an "Answer." $\endgroup$ – BruceET Mar 6 '16 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.