61
$\begingroup$

What is the variance of the sample variance? In other words I am looking for $\mathrm{Var}(S^2)$.

I have started by expanding out $\mathrm{Var}(S^2)$ into $E(S^4) - [E(S^2)]^2$

I know that $[E(S^2)]^2$ is $\sigma$ to the power of 4. And that is as far as I got.

$\endgroup$
  • $\begingroup$ Your expressions are very difficult to read. You need to edit and present your question in a better way. $\endgroup$ – smanoos Oct 16 '11 at 4:44
  • 1
    $\begingroup$ One way of expressing $Var(S^2)$ is given on the Wikipedia page for variance. $\endgroup$ – Mike Spivey Oct 16 '11 at 4:50
  • $\begingroup$ It doesn't show how they derived it. $\endgroup$ – MathMan Oct 16 '11 at 4:51
  • $\begingroup$ The solution to the question is in many books. You can easily find it. $\endgroup$ – smanoos Oct 16 '11 at 5:17
  • 1
    $\begingroup$ There is a derivation on MathWorld's Sample Variance Distribution page. They use the "divide by $N$" convention rather than the "divide by $N-1$" convention, though, so you might have to adjust for that. $\endgroup$ – Mike Spivey Oct 16 '11 at 5:39
58
$\begingroup$

Maybe, this will help. Let's suppose the samples are taking from a normal distribution. Then using the fact that $\frac{(n-1)S^2}{\sigma^2}$ is a chi squared random variable with $(n-1)$ degrees of freedom, we get $$\begin{align*} \text{Var}~\frac{(n-1)S^2}{\sigma^2} & = \text{Var}~\chi^{2}_{n-1} \\ \frac{(n-1)^2}{\sigma^4}\text{Var}~S^2 & = 2(n-1) \\ \text{Var}~S^2 & = \frac{2(n-1)\sigma^4}{(n-1)^2}\\ & = \frac{2\sigma^4}{(n-1)}, \end{align*}$$

where we have used that fact that $\text{Var}~\chi^{2}_{n-1}=2(n-1)$.

Hope this helps.

$\endgroup$
  • 12
    $\begingroup$ Remember that $(n-1)S^2/\sigma^2$ is only guaranteed to be $\chi^2$ when the sample is taken from a normal distribution, though. $\endgroup$ – Mike Spivey Oct 16 '11 at 6:09
  • $\begingroup$ Thanks Mike. I've edited my answer to reflect what you said in your comment. $\endgroup$ – Nana Oct 16 '11 at 6:15
  • 2
    $\begingroup$ The question posed is a general one, whereas the answer is distribution-specific. Not appropriate, I am afraid. $\endgroup$ – wolfies Apr 26 '13 at 7:49
  • 1
    $\begingroup$ @afsdfdfsaf Perhaps, you should ask that as a separate question. $\endgroup$ – Nana Apr 12 '14 at 18:48
  • 4
    $\begingroup$ The answer is extremely useful, but would have been even more useful if someone could reference why (n−1)S2/σ2 is a Chi squared $\endgroup$ – moldovean Mar 9 '15 at 7:06
77
$\begingroup$

Here's a general derivation that does not assume normality.

Let's rewrite the sample variance $S^2$ as an average over all pairs of indices: $$S^2={1\over{n\choose 2}}\sum_{\{i,j\}} {1\over2}(X_i-X_j)^2.$$ Since $\mathbb{E}[(X_i-X_j)^2/2]=\sigma^2$, we see that $S^2$ is an unbiased estimator for $\sigma^2$.

The variance of $S^2$ is the expected value of $$\left({1\over{n\choose 2}}\sum_{\{i,j\}} \left[{1\over2}(X_i-X_j)^2-\sigma^2\right]\right)^2.$$

When you expand the outer square, there are 3 types of cross product terms $$\left[{1\over2}(X_i-X_j)^2-\sigma^2\right] \left[{1\over2}(X_k-X_\ell)^2-\sigma^2\right]$$ depending on the size of the intersection $\{i,j\}\cap\{k,\ell\}$.

  1. When this intersection is empty, the factors are independent and the expected cross product is zero.

  2. There are $n(n-1)(n-2)$ terms where $|\{i,j\}\cap\{k,\ell\}|=1$ and each has an expected cross product of $(\mu_4-\sigma^4)/4$.

  3. There are ${n\choose 2}$ terms where $|\{i,j\}\cap\{k,\ell\}|=2$ and each has an expected cross product of $(\mu_4+\sigma^4)/2$.

Putting it all together shows that $$\mbox{Var}(S^2)={\mu_4\over n}-{\sigma^4\,(n-3)\over n\,(n-1)}.$$ Here $\mu_4=\mathbb{E}[(X-\mu)^4]$ is the fourth central moment of $X$.

$\endgroup$
  • 1
    $\begingroup$ a related question on stats.SE asks provides a different solution, and asks for a reference, your input would be appreciated: stats.stackexchange.com/q/29905/2750 $\endgroup$ – Abe Jun 6 '12 at 16:56
  • 1
    $\begingroup$ @Abe Sorry, I don't have any references or worthwhile input. The above is a solution that I made up to teach my students. $\endgroup$ – user940 Jun 6 '12 at 19:07
  • 2
    $\begingroup$ thanks, an answer to the stats.SE question solved my confusion: the discrepancy was use of kurtosis ($\mu_4$, the fourth central moment) vs excess kurtosis ($\kappa = \frac{\mu_4}{\sigma^4} -3$); one reference is Mood Graybill and Boes, 1974, Introduction to the Theory of Statistics $\endgroup$ – Abe Jun 6 '12 at 22:39
  • 1
    $\begingroup$ @AntoniParellada It is just algebra. Expand the square in the double sum $\sum_{i=1}^n \sum_{j=1}^n (x_i-x_j)^2$ and simplify. It should be pretty clear from there, but if not let me know. $\endgroup$ – user940 Aug 3 '15 at 15:23
  • 5
    $\begingroup$ In the derivation, how do we see claims 2 and 3, i.e. that the expected value of $$\left[{1\over2}(X-Y)^2-\sigma^2\right] \left[{1\over2}(X-Y)^2-\sigma^2\right]$$ is $(\mu_4+\sigma^4)/2$, for X,Y i.i.d? $\endgroup$ – Emolga Jul 28 '17 at 11:35
5
$\begingroup$

There can be some confusion in defining the sample variance ... 1/n vs 1/(n-1). The OP here is, I take it, using the sample variance with 1/(n-1) ... namely the unbiased estimator of the population variance, otherwise known as the second h-statistic:

h2 = HStatistic[2][[2]] 

These sorts of problems can now be solved by computer. Here is the solution using the mathStatica add-on to Mathematica. In particular, we seek the Var[h2], where the variance is just the 2nd central moment, and express the answer in terms of central moments of the population:

CentralMomentToCentral[2, h2] 

enter image description here

We could just as easily find, say, the 4th central moment of the sample variance, as:

CentralMomentToCentral[4, h2]

enter image description here

$\endgroup$
  • $\begingroup$ Does your program also let you handle dependent random variables? Presumably, then the result would be in terms of higher-order covariances. $\endgroup$ – Thomas Ahle Aug 15 at 15:17
  • $\begingroup$ Yes - it works for dependent random variables too. There is a multivariate example at: math.stackexchange.com/questions/589865/… $\endgroup$ – wolfies 2 days ago
1
$\begingroup$

The best way to understand what the variance of a sample looks like is to derive it from scratch.

On the following site you will find the complete derivation (it goes over 70 steps) of the sample variance. It takes a bit of time to fully understand how it is working, but if one goes over the whole derivation several times it becomes quite clear.

You will also understand better, why the proposed sample variance estimator is unbiased.

http://economictheoryblog.wordpress.com/2012/06/28/latexlatexs2/

$\endgroup$

protected by Saad Oct 3 '18 at 4:10

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.