Intuitive Explanation of Bessel's Correction

When calculating a sample variance a factor of (N-1) appears instead of N (see http://en.wikipedia.org/wiki/Sample_variance#Population_variance_and_sample_variance ). Does anybody have an intuitive way of explaining this to students who need to use this fact but maybe haven't taken a statistics course?

-
This is essentially a duplicate of Sample Standard Deviation vs. Population Standard Deviation. –  Mike Spivey Sep 1 '11 at 19:04
But the answer there just says that there is a bias and here's how you correct it. Is there a way to explain the correction intuitively (yes that's a little vague). –  Boris Sep 1 '11 at 19:07
There is an attempt at an intuitive explanation in the second part of that answer. Does that help? –  Mike Spivey Sep 1 '11 at 19:08
The concrete example I cite in my answer below can be understood without even using algebra. –  Michael Hardy Sep 1 '11 at 20:02
The fact that correction for bias can sometimes be a very bad thing to do seems not widely understood among non-statisticians, including some who teach statistics. In this example of estimating variance, by the usual mean-squared error criterion, and assuming the sample is from a normally distributed population, the unbiased estimator is only slightly worse than the biased estimator, so it's not a good example to illustrate that point. But the idea that it is the unbiased estimator that is worse may fail to be as widely appreciated as it could be. –  Michael Hardy Sep 1 '11 at 20:11

http://en.wikipedia.org/wiki/Bessel%27s_correction

The Wikipedia article linked to above has a section (written by me) titled "The source of the bias". It explains it via a concrete example.

But note also that correcting for bias, when it can be done, is not always a good idea. I wrote this paper about that: http://www.math.umn.edu/~hardy/An_Illuminating_Counterexample.pdf

(My article begins near the bottom of the first page of this reprint.)

-
+1 for the interesting paper. –  Mike Spivey Sep 1 '11 at 21:10
Thank you. (....extra line because the software does not allow short comments....) –  Michael Hardy Sep 1 '11 at 21:21
I use dollars  –  The Chaz 2.0 Sep 1 '11 at 21:28
(to circumvent the minimum character requirement) And while I haven't seen anyone else use this, I'm not opposed to sharing it with such a generous (mathematical) contributor! –  The Chaz 2.0 Sep 1 '11 at 21:29
@TheChaz: Unicode zero width spaces work much better, and don't have the formatting issues like in your comment above. –  Ilmari Karonen Sep 16 '13 at 14:27
show 1 more comment

To me, the main idea is that the sample mean is not the distribution (or population) mean. The sample mean is "closer" to the sample data than the distribution mean, so the variance computed is smaller.

Suppose the distribution mean is $m_d$. The sum of $n$ variates (the sample) is $n m_s$, where $m_s$ is the sample mean. Recall that the mean and variance of a sum of variates are the sum of the means and sum of the variances of the variates. That is, the distribution mean of the sum of $n$ variates is $n m_d$ and the distribution variance of the sum of $n$ variates is $n v_d$. In other words, $$\mathrm{E}[(n m_s-n m_d)^2]=n v_d$$ or equivalently, $$\mathrm{E}[(m_s-m_d)^2]=\frac{1}{n}v_d$$ Let us compute the expected sample variance as \begin{align} &\mathrm{E}[v_s]\\ &=\mathrm{E}\left[\frac{1}{n}\sum_{k=1}^n(x_k-m_s)^2\right]\\ &=\mathrm{E}\left[\frac{1}{n}\sum_{k=1}^n(x_k-m_d)^2+2(x_k-m_d)(m_d-m_s)+(m_d-m_s)^2\right]\\ &=\mathrm{E}\left[\frac{1}{n}\sum_{k=1}^n(x_k-m_d)^2+2(m_s-m_d)(m_d-m_s)+(m_d-m_s)^2\right]\\ &=\mathrm{E}\left[\frac{1}{n}\sum_{k=1}^n(x_k-m_d)^2-(m_d-m_s)^2\right]\\ &=v_d-\frac{1}{n}v_d\\ &=\frac{n{-}1}{n}v_d \end{align} Thus, $$v_d=\frac{n}{n{-}1}\mathrm{E}[v_s]$$ This is why, to estimate the distribution variance, we multiply the sample variance by $\frac{n}{n{-}1}$. Thus, it appears as if we are dividing by $n{-}1$ instead of $n$.

-
+1, though for an intuitive explanation, you could have stopped after the first paragraph. –  Henry Sep 1 '11 at 22:51
@Henry: yes, I know. Since the derivation follows the idea of my intuitive explanation, I thought I would attach it for anyone who is interested. –  robjohn Sep 2 '11 at 3:27

The sum $\sum (x_i - a)^2$ is minimized when $a$ is the average of the $x_i$'s. The proof is a simple exercise in algebra (write sum as quadratic in $a$) or calculus (differentiate to get $\sum (x_i - a) = 0$).

Therefore, $\sum (x_i - \overline{x})^2 \leq \sum (x_i - \mu)^2$. The same inequality holds with averages in place of sums.

The average of the squared $(x_i - \mu)$'s is an unbiased estimate of the variance, since each term has that variance as its expected value. Replacing $\mu$ by $\overline{x}$ produces a smaller estimate of the variance, and the expected value of that estimate is lower than the expected value of the unbiased estimator with $\mu$. The latter is the variance, so the expression with $1/N$ will (on average) underestimate the variance.

To summarize: the use of $\overline{x}$ as a surrogate for $\mu$ causes a downward bias in estimating variance by $\sum (x_i - \overline{x})^2 / N$.

-

The question refers to "explaining this to students who need to use this fact but maybe haven't taken a statistics course". If they're more advanced than those who will understand the example that I mentioned that does not require algebra beyond expanding $(a+b)^2$, maybe a couple of other points of view are worth looking at.

We can write $$\begin{bmatrix}x_1 \\ \vdots \\ x_n\end{bmatrix} = \begin{bmatrix}\overline{x} \\ \vdots \\ \overline{x} \end{bmatrix} + \begin{bmatrix} x_1 - \overline{x} \\ \vdots \\ x_n - \overline{x}\end{bmatrix},$$ and notice that the two vectors being added are the orthogonal projections of the sum onto spaces of dimensions $1$ and $n-1$. The expected value of the first summand is $\mu$ times a column of $1$s, and the expected value of the second summand is $0$. So rotate the coordinate system so that this becomes $$\begin{bmatrix}u_1 \\ \vdots \\ u_n\end{bmatrix} = \begin{bmatrix} u_1 \\ 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix} + \begin{bmatrix} 0 \\ u_2 \\ u_3 \\ \vdots \\ u_n \end{bmatrix}.$$ The expected value of the first entry in the first summand is $\mu\sqrt{n}$. The expected value of every entry in the second summand is $0$. The expected value of the square of the norm of the second vector is $n-1$ times the expected value of the square of any of its entries. That's where the $n-1$ comes from. Notice that $$\underbrace{\sum_{i=1}^n (x_i - \overline{x})^2}_{n\text{ terms}} = \underbrace{\sum_{i=2}^n u_i^2}_{n-1\text{ terms}}.$$

If students do know some probability theory, the above can also explain why $\sum_{i=1}^n (X_i - \overline{X})^2/\sigma^2$ has a chi-square distribution with $n-1$ degrees of freedom when there are certain assumptions about normal distribution and about independence. (I use capital $X$ this time since it's a random variable.) It can also explain why $\overline{X}$ is actually independent of that chi-square random variable.

Another thing that is sometimes useful in thinking about this topic is the algebraic identity $$\sum_{i=1}^n (x_i - \mu)^2 = n(\overline{x} - \mu)^2 + \sum_{i=1}^n (x_i - \overline{x})^2$$ (where $\overline{x} = (x_1+\cdots+x_n)/n$). Clearly this implies that $$\sum_{i=1}^n (x_i - \mu)^2 \ge \sum_{i=1}^n (x_i - \overline{x})^2$$ with equality if and only if $\overline{x}=\mu$. This is of course the same thing as what was used in the concrete example in the Wikipedia article linked in my earlier answer, but stated in a way that will be understood by students who know more algebra than the expansion of $(a+b)^2$.

-
This bias is corrected by dividing by a smaller number ($n-1$ instead of $n$) so that the variance gets bigger, i.e. unbiased again.