Tell me more ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.
up vote 0 down vote favorite

Suppose I create a benchmark and measure its speed repeatedly. I would expect each benchmark run to take the same amount of time but due to external factors, it doesn't. Instead, the speed of any given run can be approximated by a normal distribution. Assume I've run the benchmark thousands of times to have a high confidence in this approximation. (If a better distribution exists, I'm all ears.)

Now suppose I make a small change to the benchmark and run it again several times. How could I tell whether the timings of the modified benchmark represent a statistically significant speedup or slow down? Assume I can't afford thousands of runs of the modified benchmark.

I've read the usual sources: Wikipedia and MathWorld but it still doesn't make sense to me.

share|improve this question
Have you looked at the Wikipedia article on hypothesis testing? – André Nicolas Apr 24 '12 at 15:47
Let me see if I understand this. You have data on the time it takes two processes to run. The first has been run thousands of times, and the second significantly fewer. And you want to know if the mean of times for the first is significantly different from the mean of times for the second. – B R Apr 24 '12 at 15:51
Also, the time it takes something to run will never actually be a normal distribution (since it can never be negative). So you might consider a log transform. Though, if the data looks normal enough, you shouldn't worry about it. Also, by the Central Limit Theorem, even if the speed isn't approximated by a normal distribution, the mean of the speeds will be. – B R Apr 24 '12 at 15:57
@B R - That's right: compared to the mean of the speeds, the distribution is "normal enough." So I have a thousand data points like 0.01, -0.002, ... corresponding to fluctuations in benchmark run times as compared with the mean with no modification. Then I have five data points of the modified benchmark compared to the mean: -0.035, -0.03, -0.02 (those would correspond to speedups). How can I produce a measure of confidence that those values represent a real speedup given data like the mean and standard deviation of the unmodified benchmark? – GrantJ Apr 25 '12 at 13:50
@AndréNicolas - I reread that today and was lead to the Z-test article on Wikipedia. Is this problem a good application of the Z-test? – GrantJ Apr 25 '12 at 14:10
show 4 more comments

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.