# Why does sample standard deviation underestimate population standard deviation?

Refering to this wikipedia page Unbiased estimation of standard deviation, it says that "it follows from Jensen's inequality that the square root of the sample variance is an underestimate".

I do know that for the concave square root function, Jensen's inequality says that the square root of the mean > mean of the square root.

So, how do we conclude that the square root of the sample variance underestimates population standard deviation?

Since we know from Jensen's inequality that square root of the mean > mean of the square root, does "square root of sample variance" somehow relate to "mean of the square root" while "population standard deviation" somehow relates to "square root of the mean"?

Given joriki's response about using a single sampling of data, I am now left with why $s=\sqrt{\frac{1}{N-1}\sum_{i=1}^N{(x_i-\overline{x})^2}}$ will underestimate pop std dev. In order to use Jensen's inequality (mean of the square root < square root of the mean). I need to somehow relate the expression for $s$ to "mean of square root". I do see the square root sign in the expression for $s$ but where is the "mean" of this square root quantity?

• Yes, you have it right. Recall that the sample variance is an unbiased estimator of the variance. This means that the expectation (mean) of $\frac{1}{n-1}\sum(X_i-\bar{X})^2$ is $\sigma^2$. The square root of this is $\sigma$, the right thing. Thus $\sigma$ is greater than the expectation of $\sqrt{\frac{1}{n-1}\sum(X_i-\bar{X})^2}$, which is the expectation (mean) of the sample standard deviation. If we use the expression just mentioned as our estimator for $\sigma$, then on average this estimator will be too small. Feb 8, 2012 at 6:52
• So, do you mean that if I were to perform an experiment where I take many samplings from a population: if I first compute the unbiased estimator for population variance from each sampling; then get the mean of all these values; followed by taking the square root of this mean; I will get the unbiased estimator for population standard deviation? Feb 8, 2012 at 7:21
• JohnC: It would not be an estimator. An unbiased estimator is a random variable whose mean is the right thing. You have a random variable the square root of whose mean is the right thing. Feb 8, 2012 at 7:29
• @JohnC: That two-stage process (dividing your total sample into several samples, computing the sample variance for each and then averaging) doesn't improve the estimation; in fact it makes it worse, since you're throwing away information about how much the groups deviated with respect to each other. You'll get the best estimate if you treat all your data as one big sample. The problem of the square root skewing the result is independent of this; approaches to dealing with it are discussed in the Wikipedia article you linked to. Feb 8, 2012 at 13:12
• @JohnC: You ask where is the mean. Suppose we use the already square rooted expression as an estimator of the population standard deviation $\sigma$. We imagine taking the mean in order to find out whether this estimator is biased or not. If the mean of this estimator is $\sigma$, then it is unbiased. The Jensen's Inequality argument shows that except in trivial cases, the mean is less than $\sigma$. This lets us conclude that the estimator is not unbiased, and even more, in what direction it errs on average. Feb 8, 2012 at 17:46

The mean is part of what it means for an estimator to be biased. You can't make the estimator unbiased by averaging over several estimates; to the contrary, you can show that it's biased by averaging over estimates and showing that the expected average isn't the value to be estimated. (You can reduce the bias and the variance of the estimator by averaging several estimates, but as discussed above you can do that even better by using all the data for one estimate.)

For example, if your population has equidistributed values $-1,0,1$, with variance $\frac23$, and you take a sample of $2$, you'll get variance estimates of $0$, $\frac12$ and $2$ with probabilities $\frac13$, $\frac49$ and $\frac29$, respectively, yielding the correct mean $\frac13\cdot0+\frac49\cdot\frac12+\frac29\cdot2=\frac23$, whereas the estimates for the standard deviation, $0$, $\sqrt{\frac12}$ and $\sqrt2$ average to $\frac13\cdot0+\frac49\cdot\sqrt{\frac12}+\frac29\cdot\sqrt2=\frac49\sqrt2\neq\sqrt{\frac23}$, with $\frac49\sqrt2\approx0.6285\lt0.8165\approx\sqrt{\frac23}$, an underestimate as expected. If you take a sample of $3$ instead, the mean improves to $\frac19\cdot0+\frac49\cdot\sqrt{\frac13}+\frac29\cdot\sqrt{\frac43}+\frac29\cdot1=\frac19(8\sqrt{\frac13}+2)\approx0.7354$.

Let's assume we're picking $n$ independent samples from the same (unknown) distribution. Thus, the samples $x_1, x_2, \dotsc, x_n$ are independent and identically distributed random variables with some unknown mean $\mu$ (which we may approximate by the sample mean $\bar x = \frac 1 n \sum x_i$) and standard deviation $\sigma$, which we wish to estimate.

As André Nicolas notes in his first comment, the sample variance $$\tilde \sigma^2 = \frac 1{n-1} \sum_{i=1}^n(x_i-\bar x)^2$$ is a random variable whose mean or expected value $\mathrm E[\tilde \sigma^2]$ is equal to the true variance $\sigma^2$ of the unknown distribution. Thus, $\tilde \sigma^2$ is an unbiased estimator of $\sigma^2$. However, because the square root function is concave, by Jensen's inequality the mean $\mathrm E[\tilde \sigma]$ of its square root $$\tilde \sigma = \sqrt{\tilde \sigma^2} = \sqrt{\frac 1{n-1} \sum_{i=1}^n(x_i-\bar x)^2}$$ is (except in trivial cases) less than the square root $\sigma$ of its mean $\mathrm E[\tilde \sigma^2] = \sigma^2$. Thus, $\tilde \sigma$ is an underestimate of the true standard deviation $\sigma$.

Let's warm up. Subject to the constraint $\sum_{i} r_i = 1$, we have the following. \begin{align*} \sqrt{r_1 a_1 + r_2 a_2 + r_3 a_3} &\ge r_1 \sqrt{a_1} + r_2 \sqrt{a_2} + r_3 \sqrt{a_3} \end{align*} Substituting $r_i = \frac{1}{3}$, $a_1 = 1$, $a_2 = 16$ and $a_3=25$ we have \begin{align*} \sqrt{\frac{1}{3} (1 + 16 + 25)} &\ge \frac{10}{3}. \end{align*}

Let $\phi$ be a concave function such as the square root function. Then, by Jensen's inequality, we have $$\phi(\mathbb{E}[x]) \ge \mathbb{E}[\phi(x)].$$ Further, $\mathbb{E}[x] = T$, the true mean, in an unbiased estimator. We desire to estimate $\phi(T)$. But we have $$\mathbb{E}[\phi(x)] \le \phi(T).$$ Hence, we get an 'underestimate' stochastically speaking.

Yes, I know this question is old, but someone might need true answer. Note that estimator of variance is noted as $$\hat{\sigma^2}$$, not $$\hat{\sigma}^2$$ which is squared estimator of standard deviation. To prove that thing you use inequality from https://www.probabilitycourse.com/chapter6/6_2_5_jensen%27s_inequality.php but note you need it for concave, not convex function. Because $$f(x) = \sqrt{x}$$ is concave, we have: $$E[\sqrt{\hat{\sigma^2}}] \le \sqrt{E[\hat{\sigma^2}]} = \sqrt{\sigma^2} = \sigma$$ $$E[\hat{\sigma^2}] = \sigma^2$$ because $$\hat{\sigma^2}$$ is unbiased estimator of variance.

It seems everybody is saying sample variance is an unbiased estimator; even the Wikipedia article implies this. Yet this seems to conflict with at least one paper and my own simulations.

Crawford & Howell, 1998:

"One problem with using the standard deviation of a small normative sample as if it were the parameter (F) is that, although the sample variance is a maximum likelihood estimator of F2 , the sampling distribution of the variance is positively skewed. This means that we will be more likely to underestimate F than to overestimate it. Thus we are more likely to overestimate z and the rarity of the observation.” The test proposed in this paper seems to be the gold standard for comparing single subjects to samples and was an improvement because it fixed this issue of variance underestimation.

Here are some simulations I ran. Interestingly, the variance estimates go from a Poisson distribution with n=2 to a more normal-looking distribution with n=10. Btw, the population variance in all these figures is approximately 1:

Average (mean) of variances amongst samples of n=2, with replacement

Average (mean) of variances amongst samples of n=10, with replacement

Average variance vs. sample size

Why do people think that happens? My guess is Wikipedia and the answers here assume the sample variance is $$\frac{1}{n-1}\sum (X_i-\bar{X})^2$$ where $$\bar{X}$$ is the population mean. But really the population mean is not going to be reflected in the sample. Samples with more extreme values will have more extreme means, closer to the extreme values. So the distance to the sample means will be less and the variance will be smaller.

Paper referenced: Crawford, J. R., & Howell, D. C. (1998). Comparing an Individual’s Test Score Against Norms Derived from Small Samples. The Clinical Neuropsychologist, 12(4), 482–486. doi:10.1076/clin.12.4.482.7241

• I am confused, is this question or an answer? Sep 22, 2021 at 5:31
• It's a question and also pointing out a seemingly flawed prevalent assumption in other answers here and Wikipedia: that variance is an unbiased estimator. I was wondering what others' thoughts are as to the reason variance is biased
– nnsk
Sep 22, 2021 at 17:14
• post a new question referring to this thread then. as of now, very few people would see your post as this is not a space for a question. Sep 23, 2021 at 1:57

Reality is that the sampling distribution of $$S$$ is skew to the right, more so for s from small samples, less so for $$S$$ from large samples.

Fact 1: the median of a distribution that is skew to the right is less than the mean of the distribution.

Fact 2: The median $$S$$ is the $$50^{th} \%$$ile.

Fact 3: the mean $$S$$ is equal to the population standard deviation.

Therefore, the mean has a percentile larger than $$50\%$$, and a single randomly drawn $$S$$ is more likely to underestimate sigma than to overestimate sigma, even though the mean $$S$$ is sigma. The discrepancy between the median s and the mean s vanishes as sample size approaches infinity, which is why $$t$$ converges to $$Z$$ for progressively larger sample sizes.