$n$ vs $n-1$ for the standard deviation

Question

Suppose that I went to Tasmania a few years before the "Tazie Tiger" (thylacine) became extinct. I sample say, $100$ thylacines and make some biometric measurements. To make the discussion concrete, let's make the data the skull widths at the widest point on the head.

From this data I can calculate the sample mean, $\bar x$, and the standard deviation, sigma.

I get confused by the use of $n-1$ in the denominator of $\sigma$. So, I go away, look at Wikipedia, find out about Bessel's correction, slowly work my way through the maths and eventually learn to accept that, since $\bar x$ is calculated from the data, I only have $99$ independent comparisons of $x-\bar x$.

This makes a kind of sense. If I only have $1$ observation, there are no other data to compare it with. If I have $2$ observations, I have exactly $1$ estimate of the spread of the data, which is the spread of the data, and so on. I even understand that the sum of the absolute deviations must equal zero as the mean has been calculated from the data.

Great! I go away, happy with my calculations.

A week later, someone tells me that I was so thorough with my survey that my "sample" was actually a census. I have taken measurements of absolutely every single living thylacine in the world. Suddenly, my sample average, $\bar x$ is actually the true population average $\mu$. Clearly, I now have to use the population standard deviation.

I go back to my calculation. The number of data is still $100$. The population average has still been calculated from the data. The sum of the absolute deviations still adds up to zero. I still have $n-1$ independent estimates of $x-\bar x$.

The data are identical. The algebra is identical. So when I calculate the standard deviation, why do I now divide by $n$, rather than by $n-1$? Where did the extra degree of freedom come from?

Answer 1

When you do your $n-1$ calculations, you are in effect trying to take account of the bias in the estimate of the variance caused by your observations being likely to be closer to the sample estimate of the mean than they are to the population mean, because the sample mean is based on the sample observations

• In the extreme case of a sample sized $1$, you observe no sample variance even if there is likely to be some.
• In the almost-as-extreme version of a sample sized $2$, half of the location of the sample mean is determined by each sample observation, pulling the sample mean towards it
• With larger samples the same thing can occur, though to a lesser extent

So when you know you have sampled the whole population, you no longer need to worry about this bias. Nor do you need to worry about the distribution: you know it precisely from your observations.

• In your sample sized $1$, you know there is no variance
• In your sample sized $2$ there is no bias introduced into the variance by the sample mean, as it is the population mean
• With larger samples there is also no bias introduced into the variance so you can use the population $n$-based calculation rather than the $n-1$ based one

Answer 2

I was so thorough with my survey that my "sample" was actually a census.

I believe this is where the misunderstanding lies. For purposes of Bessel's correction, a census is not a very thorough sample – it's either an incorrect sample or a highly unlikely sample.

Bessel's correction is exact for the case where you uniformly independently sample $n$ individuals from a population of $N$ individuals, and it's correct and exact also for the case $n=N$ (in fact even for $n\gt N$). It doesn't apply to the case where you sample a random subset of $n$ distinct individuals out of a population of $N$, but that's what you're implicitly applying it to. If you worked out the proper correction for the case that you're applying it to, you'd find that as the sample size $n$ increases from $1$ to $N$, the denominator gradually increases from $n-1$ to $n$.

This distinction is rarely explicitly made because Bessel's correction is typically applied in cases where e.g. $\sim\!10^3$ people are sampled out of a population of $\sim\!10^8$ and it doesn't matter much whether they're constrained to be a distinct subset or not.

As a simple example, consider a population of $N=2$. If you uniformly independently sample $n=2$ individuals, you have a probability of $0.5$ to get the same individual twice, with a variance estimate of $0$, and a probability of $0.5$ to get both individuals, with an uncorrected variance estimate given by the actual variance. Thus, despite $n=N$, the expected uncorrected variance estimate is $\frac{n-1}n=\frac12$ times the actual variance, consistent with Bessel's correction.

P.S.: It's actually straightforward to work out the correction for your case of sampling distinct subsets. With $\Delta^2$ the mean square difference of two distinct individuals, the population variance is

$$V=\frac{N(N-1)}{N^2}\Delta^2=\frac{N-1}N\Delta^2\;,$$

whereas the expected sample variance is

$$\hat V=\frac{n(n-1)}{n^2}\Delta^2=\frac{n-1}n\Delta^2=\frac{n-1}n\frac N{N-1}V\;.$$

Thus the correct correction factor in your case is $\frac n{n-1}\frac{N-1}N$. For $n\ll N$ (the domain in which Bessel's correction is usually applied), this is essentially $\frac n{n-1}$, but for $n=N$ it's $1$, as you expected.

Answer 3

Comments in addition to @Henry's excellent Answer. Some elementary textbooks use $\frac 1 n \sum_{i=1}^n (X_i - \bar X)^2$ to estimate population variance $\sigma^2.$ (One reference: Freedman, et al.) This is partly for simplicity (to avoid questions such as yours) and partly because there is no one "best" denominator in all practical situations.

However, for normal data, the estimate $S^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar X)^2$ has the advantage that $Q = \frac{(n-1)S^2}{\sigma^2} \sim \mathsf{Chisq}(\nu = n - 1),$ so that a 95% confidence interval (CI) for $\sigma^2$ is of the form $\left(\frac{(n-1)S^2}{U}, \frac{(n-1)S^2}{L}\right),$ where $L$ and $U$ cut 2.5% of the probability from the lower and upper tails, respectively of $\mathsf{Chisq}(\nu = n - 1),$ the chi-squared distribution with $n - 1$ degrees of freedom. Also, a 95% CI for the population mean $\mu$ is of the form $\bar X \pm t^*\frac{S}{\sqrt{n}},$ where $t^*$ cuts 2.5% of the probability from the upper tail of (the symmetrical) Student t distribution with $n - 1$ degrees of freedom.

Moreover, for any distribution having a variance $\sigma^2 < \infty,$ one has $E(S^2) = \sigma^2,$ so that $S^2$ (with denominator $n-1)$ is an 'unbiased' estimator or $\sigma^2.$ [The sample standard deviation $S$ is a slightly biased estimator of $\sigma,$ but the bias is quite small except for samples of very small size $n.$ In particular, for a normal sample of size $n$ one can show that $E(S) = \sigma \sqrt{\frac{2}{n-1}}\Gamma({\frac n 2})/\Gamma(\frac{n-1}{2}).$ For a normal sample of size $n = 10,$ this amounts to $E(S) \approx 0.923\sigma.]$

Answer 4

The division by $n-1$ rather than by $n$ is still valid if your sample was without replacement and independent and every individual still has the same chance of being chosen.

For example, suppose $100$ critters exist, and each time you draw an individual from the population to measure the weight of its left ear, every critter has an equal chance of being chosen and the draws are all independent of each other. Say you take a sample of $300$ that way. Since there are only $100$ critters, you must have chosen some of them at least three times and you may have missed some and you may have caught some of them six times, etc. If you can't identify individuals so you don't know how many times you caught each critter, then the Bessel-corrected estimate in which you divide by $300-1$ gives you an unbiased estimator of the population variance even if you caught every critter exactly once. But what you are getting an unbiased estimate of, is the variance calculated by a process in which you divide by $100,$ not by $100-1.$

And unbiasedness is over-rated. The thing to divide by if you want to minimize the mean-square error of estimation is actually $300+1,$ which gives you a biased estimator.