1
$\begingroup$

I've recently been going through Khan Academy Statistics, and I recently came across the fact that sample standard deviation is biased. Now, I know that there are many proofs online including several on math stack exchange, but I was wondering if someone could give me some intuition for why this happens instead of using technical methods (like Jensen's inequality)? Please keep in mind my mathematical/statistical knowledge is equivalent to that of a grade 11 student.

Any help would be greatly appreciated!

$\endgroup$
  • 1
    $\begingroup$ I would have thought the simple explanation was that the sample variance with the $\frac{1}{n-1}$ formulation is an unbiased estimator of the population variance but taking square roots is not a linear function and so the square root of an unbiased estimator is unlikely to be an unbiased estimator of the square root. $\endgroup$ – Henry Mar 13 at 0:45
  • $\begingroup$ Just to clarify, my question is about why the sample standard deviation is biased, not about why sample variance is unbiased because of n-1. $\endgroup$ – John A. Mar 13 at 14:00
0
$\begingroup$

Reproduced from my argument in an AoPS thread, also featuring a derivation of the sample variance:

The square root of this estimate for the variance is not an unbiased estimator of the standard deviation, because square roots and expected values don't commute.

A simple example: let $ X$ be the probability distribution which is $ 1$ or $ -1$ with equal probabilities $ \frac12$. It has mean zero and variance $ 1$. Sample it twice. Half the time, our samples are equal, and the variance estimate we get is zero. The other half of the time, our samples differ by $ 2$, and the variance estimate we get is $ 2$. The average of those estimates is $ 1$, confirming the fact that it's an unbiased estimator.

Now, what if we take the square root of this estimated variance and call it an estimate for the standard deviation? We get zero half the time and $ \sqrt{2}$ the other half, for an expected value of $ \frac{\sqrt{2}}{2}$, not equal to the original standard deviation.

This sort of thing is guaranteed to happen. The concavity of the square root means that the expected value of this standard deviation estimate will always be too low. Worse, we can't simply scale our way out of the problem, because the details of how low it is depend on the distribution. Try the same two-point estimate on a uniform distribution on $ [0,1]$; the average length between our two points is $ \frac13$, so the estimated standard deviation is $ \frac1{3\sqrt{2}}$. The real standard deviation is $ \frac1{2\sqrt{3}}$, a multiple of $ \frac{\sqrt{3}}{\sqrt{2}}$ larger.

There is no generic unbiased estimator for the standard deviation. We have to make do with a biased one, and the square root of our unbiased variance estimator is a good choice.

$\endgroup$
  • $\begingroup$ It is simply untrue that there is no unbiased estimator for the SD of a normal population based on a random sample of size $n.$ It is based on $E(S) = \sigma\sqrt{\frac{n}{n-1}}\frac{\Gamma(n/2)}{\Gamma((n-1)/2)}.$ One ref is Wikipedia. $\endgroup$ – BruceET Mar 13 at 5:38
  • 1
    $\begingroup$ I said that there's no generic unbiased estimator. If you know more information, such as "this comes from a normal distribution", that's a different matter. $\endgroup$ – jmerry Mar 13 at 5:40
  • $\begingroup$ Thanks for the help @jmerry, but I think that the level of complexity in your answer is beyond my current knowledge. Can you please try and give me a more intuitive explanation without more technical approaches (for example I haven’t really learned expected value yet). Thanks! $\endgroup$ – John A. Mar 13 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.