13
$\begingroup$

After meassuring the answer time of a software system I calculated the standard deviation from the samples. The average time is about 200ms, the standard deviation $$\sigma = 300ms$$ According to the image below this should mean that 68.2% of all response times should be between -100ms and 500ms.

enter image description here Image:https://en.wikipedia.org/wiki/Standard_deviation

A negative response time makes obviously no sense. How should the part of the normal distribution be interpreted that is enclosed in the red box?

Sample data with similiar avg ~202 stddev ~337:

100
100
200
150
70
90
110
80
150
70
190
110
130
100
100
1500
$\endgroup$
  • 14
    $\begingroup$ Why you have assumed normal distribution? $\endgroup$ – luka5z Dec 16 '15 at 15:55
  • 6
    $\begingroup$ This means that hypothesis about normal distribution of answer time is a poor one. If your software has "distribution" parameter, you may try "Poisson". $\endgroup$ – Abstraction Dec 16 '15 at 15:56
  • $\begingroup$ Maybe try lognormal $\endgroup$ – luka5z Dec 16 '15 at 15:57
  • $\begingroup$ Related: This question $\endgroup$ – ccorn Dec 16 '15 at 15:57
  • 1
    $\begingroup$ Try to remember that normally distributions are not normal . . . $\endgroup$ – Keith Dec 16 '15 at 22:35
13
$\begingroup$

You have assumed a normal distribution for a data which cannot be negative. It doesn't make sense at all. You can use lognormal distribution instead. It is used in Black-Scholes model for pricing options. (Stock prices cannot be negative)

Obviously, I can't tell you if your sample fits such a distribution if I don't have access to the full dataset.

r-script:

require(MASS)
hist(x, freq=F)
fit<-fitdistr(x,"log-normal")$estimate
lines(dlnorm(0:max(x),fit[1],fit[2]), lwd=3)

(x is a sample vector)

enter image description here

Obviously, your sample is way too small here.

$\endgroup$
  • $\begingroup$ To follow up on the title question, how can the standard deviation be interpreted if the distribution isn't normal? For example, is there any meaning to the standard deviation calculated on samples from a lognormal distribution? $\endgroup$ – R.M. Dec 16 '15 at 18:29
  • 2
    $\begingroup$ @R.M. Standard deviation is one measure of dispersion. It has the same interpretation regardless of the distribution. Standard deviation informs you how much data drawn from a particular distribution will be, on average, "spread out" from its mean. Just look at the definition: $SD(X)=\sqrt{\mathbb{E}(X-\mathbb{E}X)^2}$. Note $Y=(X-\mathbb{E}X)^2$ is another random variable which describes squared distance of $X$ from its mean. If you calculate $\mathbb{E}Y$ you will know how much on average this distance is. We take square root to normalize (go back to original scale - not squared). $\endgroup$ – luka5z Dec 16 '15 at 18:47
3
$\begingroup$

As others have mentioned, the normal distribution doesn't make much sense here. We sometimes use a normal distribution, despite the possibility of negative values that don't make sense, when the probability of such values under the normal distribution is very small. But here the standard deviation is greater than the mean, so that probability is not small at all (about $1/4$).

A glance at your data suggests that a heavy-tailed distribution might be appropriate: much of the standard deviation comes from that one very large value of $1500$.

$\endgroup$
2
$\begingroup$

You can calculate the square root of the weighted sum of the squares of the differences from the sample mean for any set of data. All this calculation does is tell you how spread out the data are.

But, as you've suspected, saying that this dataset is normally distributed is likely nonsense.

$\endgroup$
2
$\begingroup$

There's another issue that the other answers are not addressing. In applications like this one you're often not interested in the standard deviation, since it is a non-robust statistic with a breakdown point of 0%, which means that for a large sample size changing a negligible fraction of the data can result in an arbitrary change in the statistic's value. Instead, consider using quantiles, common ones being the inter-quantile range, which are more robust statistics. Specifically, the $25$-th and $75$-th quantiles both have 25% breakdown point, because you need to change at least 25% of the data to arbitrarily affect them.

This is particular important in your consideration, because of a number of factors:

  1. Communication delays are often caused by one-time events that result in a down-time rather than a normal delay, and of course such down-times are very long in comparison. For example think of power outages, server crash, even sabotage...

  2. Even if there are no down-times in your data, other factors could have a significant impact on your measurements that are completely irrelevant to your application. For example, other processes running in the background might slow down your application, or memory caching might be improving the speed for some but not all runs. There might even be occasional hardware activity that affects the speed of your application only now and then.

  3. Usually people judge a system's responsiveness based on the average case, not the average of all cases. Most will accept that an operation might in a minority of the cases completely fail and never even return a response. An excellent example is the HTTP request. A small but nonzero proportion of packets get totally dropped from the internet and the request would have a theoretically infinite response time. Obviously people don't care and just press "Refresh" after a while.

$\endgroup$
  • $\begingroup$ Very well-suited. $\endgroup$ – A.S. Dec 17 '15 at 5:42
  • $\begingroup$ @A.S.: Thanks! I always find it surprising that people don't seem to teach such statistics, even though it is in fact used in practice. Another good thing is that the quantiles always exist and can be estimated even if the mean doesn't, same for the inter-quantile range. There are even CLT-like theorems for the quantiles. Yet none of my statistics teachers ever even mentioned quantiles in statistical analysis! $\endgroup$ – user21820 Dec 17 '15 at 6:18
  • 1
    $\begingroup$ Few academicians practice and for theoreticians variance is very convenient to operate on analytically - so it gets the most attention in non-applied courses. What I found most useful about your post wasn't so much a suggestion to use quantiles (there are other robust statistics) but all the tailored info relevant to performance of the actual physical/customer system. This is what should guide a choice of statistic/metric since doing any data analysis beyond histogramming before maximally fleshing out/thinking through peculiarities of the system at hand is futile and possibly misleading...// $\endgroup$ – A.S. Dec 17 '15 at 7:32
  • $\begingroup$ //... Just look above - somebody is already trying to fit a log-normal to a small sample from much bigger data set or suggest a heavy-tail distribution even though it's possibly a mixture based on whether a "one-time event" happened or not (which I wouldn't think of right away without reading your post). $\endgroup$ – A.S. Dec 17 '15 at 7:32
  • $\begingroup$ @A.S.: Yea I saw the other post, which is why I posted my answer since I've dealt with such one-time events many times before in timing tests. As for my suggestion to use quantiles, what other robust statistics do you have in mind that can do the job well in this case? I suggested using quantiles because it doesn't presume anything at all about the distribution, but I don't know much about other robust statistics, so I'd be glad if you could tell me a bit more! $\endgroup$ – user21820 Dec 17 '15 at 7:44
1
$\begingroup$

The issue here is that you fallback to the most infamous distribution -- the normal distribution -- to interpret your results. This is not the only one, and in your case is a poor choice: indeed, the normal distribution is symmetric around its mean, and has support $\mathbb{R}$: in your case, an asymmetric distribution putting probability mass only on the positive reals would make more sense, if you really want to compare against a "known" model of probability distributions.

(Note that the notion of variance does not depend on the normal distribution, nor on any particular distribution.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.