Suppose I create a benchmark and measure its speed repeatedly. I would expect each benchmark run to take the same amount of time but due to external factors, it doesn't. Instead, the speed of any given run can be approximated by a normal distribution. Assume I've run the benchmark thousands of times to have a high confidence in this approximation. (If a better distribution exists, I'm all ears.)
Now suppose I make a small change to the benchmark and run it again several times. How could I tell whether the timings of the modified benchmark represent a statistically significant speedup or slow down? Assume I can't afford thousands of runs of the modified benchmark.
I've read the usual sources: Wikipedia and MathWorld but it still doesn't make sense to me.