# Page load speed analysis using standard deviation

This is my first ever math question so I'll try my best to make it useful for others. Here's the problem: I want to measure site load speed and decide if it is within satisfactory range or, roughly speaking, is good enough compared to what we have measured before.

Here's the method I use:

1. Measure site load speeds, say 3 times, for a control group.
2. Get values like 14022, 11505, 11795 - these are in milliseconds. Yes, this is a pretty slow website with a lot of resources so the full page load takes a while.
3. Detect the "satisfactory range" by computing standard deviation for a sample of these 3 values: stdev(14022, 11505, 11795) = 1377. I use a standard deviation of a sample formula that gives a slightly higher value.

By this point I understand the "history", i.e. how fast the website is expected to load based on previous observations, and the possible deviation, i.e. ± (1377 / 2) = ± 688.

What I will try to do next is to measure load speeds again after some update and check if anything has changed, taking into account the possible deviations and whether or not they're within the range.

1. Perform 3 new measurements to a treatment group
2. Get, say, values: 10494, 10197, 10612
3. Take the median value = 10494
4. Calculate it's difference from the mean average of the control group = 10494 - mean_average(14022, 11505, 11795) = 10494 - 12441 = -1947
5. Compare the result to the standard deviation of control group sample: abs(-1947) > 1377 5.1 Perhaps I'm wrong here and I have to compare it to the value I called "possible deviation" previously: 688.
6. The result I got suggests that the change has statistical significance because it differs from the average by more than a standard deviation or half of it.

So the conclusion I draw from the calculation above is: the changes that have been made to a website have increased its performance by ~2 seconds on average and are statistically significant.

However, I'm not sure if the reasoning I'm using is correct. Please correct me so I understand where I'm mistaken and perhaps answer the following questions:

• Do I understand standard deviation correctly, as a measure of possible difference from the "norm" that happens due to measurement method imperfection?
• If I do use it correctly, shouldn't I call a value of 1 standard deviation a sigma and only consider significant the changed that exceed 3 sigma in magnitude?
• Is it correct to calculate average of the sample to compare against? I do so because I want to count for possible outliers that have happend historically in my calculation.
• Is it correct to compare the median value of the treatment group because I want to compare a real value observed instead of imaginary average?

Thank you so much if you have made it till here, and looking forward for having your answers!

• I'd at least suggest to increase the sample size from 3 to a bi more, at least 5. May 31, 2018 at 20:37
• $6$ is wrong. You're not comparing the change to the standard deviation but to your estimate of the standard deviation from a small number of samples. To talk about statistical significance, you can't use the estimate as if it were the true value. May 31, 2018 at 20:51