I have a process by which I measure the times it take to do something, (run some code), and this time varies, a lot. I'm trying to optimize this code, making small changes that sometimes have small impacts on performance, and I want to try and see whether the change actually made a difference or not (And if so, how much of a difference)
What i'm doing right now is I run one code 10 times, time each of them, and take an average and a standard deviation. I do the same for the second code. And then... That's where I'm stuck. How can I compare two averages and their standard deviations, to know which one is greater and how greater, or to know they are actually the same (because the difference is not statistically significant)
1) Avg: 13577 Std Dev: 114
Avg: 13929 Std Dev: 220
2) Avg: 24759 Std Dev: 34
Avg: 24196 Std Dev: 110
The first case are two runs of the same code, so whatever I do to compare should tell me those two are just as fast. The second case are two runs of different codes, one of which had a "feature" disabled, which does make it a bit faster, I just don't know how much.
- How can I know whether one test was faster than the other?
- How can I know how much faster/slower?
- I'm doing 10 runs, "just because". How would I know if 10 runs is too little to know, and I need to have more runs per test? (or in other words, I believe, how do I know if my deviation is too big and I need to lower it?)
UPDATE: Turns out my question is an exact duplicate of this: