49
$\begingroup$

I have an HP 50g graphing calculator and I am using it to calculate the standard deviation of some data. In the statistics calculation there is a type which can have two values:

Sample Population

I didn't change it, but I kept getting the wrong results for the standard deviation. When I changed it to "Population" type, I started getting correct results!

Why is that? As far as I know, there is only one type of standard deviation which is to calculate the root-mean-square of the values!

Did I miss something?

$\endgroup$
94
$\begingroup$

There are, in fact, two different formulas for standard deviation here: The population standard deviation $\sigma$ and the sample standard deviation $s$.

If $x_1, x_2, \ldots, x_N$ denote all $N$ values from a population, then the (population) standard deviation is $$\sigma = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - \mu)^2},$$ where $\mu$ is the mean of the population.

If $x_1, x_2, \ldots, x_N$ denote $N$ values from a sample, however, then the (sample) standard deviation is $$s = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (x_i - \bar{x})^2},$$ where $\bar{x}$ is the mean of the sample.

The reason for the change in formula with the sample is this: When you're calculating $s$ you are normally using $s^2$ (the sample variance) to estimate $\sigma^2$ (the population variance). The problem, though, is that if you don't know $\sigma$ you generally don't know the population mean $\mu$, either, and so you have to use $\bar{x}$ in the place in the formula where you normally would use $\mu$. Doing so introduces a slight bias into the calculation: Since $\bar{x}$ is calculated from the sample, the values of $x_i$ are on average closer to $\bar{x}$ than they would be to $\mu$, and so the sum of squares $\sum_{i=1}^N (x_i - \bar{x})^2$ turns out to be smaller on average than $\sum_{i=1}^N (x_i - \mu)^2$. It just so happens that that bias can be corrected by dividing by $N-1$ instead of $N$. (Proving this is a standard exercise in an advanced undergraduate or beginning graduate course in statistical theory.) The technical term here is that $s^2$ (because of the division by $N-1$) is an unbiased estimator of $\sigma^2$.

Another way to think about it is that with a sample you have $N$ independent pieces of information. However, since $\bar{x}$ is the average of those $N$ pieces, if you know $x_1 - \bar{x}, x_2 - \bar{x}, \ldots, x_{N-1} - \bar{x}$, you can figure out what $x_N - \bar{x}$ is. So when you're squaring and adding up the residuals $x_i - \bar{x}$, there are really only $N-1$ independent pieces of information there. So in that sense perhaps dividing by $N-1$ rather than $N$ makes sense. The technical term here is that there are $N-1$ degrees of freedom in the residuals $x_i - \bar{x}$.

For more information, see Wikipedia's article on the sample standard deviation.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ @Imray: It doesn't. They refer to two different things. The expression $\sigma/\sqrt{n}$ is the standard deviation of the mean $\bar{x}$ of the sample data. The expression $s = \sqrt{\frac{1}{N-1} \sum_{i=1}^N (x_i - \bar{x})^2}$ is the standard deviation of the sample data (so not the mean of the sample data). $\endgroup$ – Mike Spivey Nov 14 '12 at 3:57
  • 3
    $\begingroup$ "Since x¯ is calculated from the sample, the values of xi are on average closer to x¯ than they would be to μ" -- oh wow, thank you. I've read the explanation based on df a number of times, but this suddenly makes it intuitively clear in a different way. $\endgroup$ – octern Sep 6 '14 at 19:46
  • 1
    $\begingroup$ @MikeSpivey: Nice explanation. But if I know population mean μ then shouldn't I divide population variance by n-1 too? $\endgroup$ – Durin Sep 28 '14 at 20:09
  • 3
    $\begingroup$ @Durin: No. If you actually know $\mu$ and you want to calculate $\sigma$ with all $N$ finite values from the population, then you're not estimating anything. In this case $\sigma$ should be calculated using the straight-up average of the squared-deviations - dividing by $n$ instead of using the estimation-bias corrected version that divides by $n-1$. $\endgroup$ – Mike Spivey Sep 28 '14 at 20:47
  • 3
    $\begingroup$ Could you give a link to the proof of "It just so happens that that bias can be corrected by dividing..."? $\endgroup$ – Vytenis Bivainis Jun 17 '18 at 22:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.