70
$\begingroup$

What is the variance of the sample variance? In other words I am looking for $\mathrm{Var}(S^2)$.

I have started by expanding out $\mathrm{Var}(S^2)$ into $E(S^4) - [E(S^2)]^2$

I know that $[E(S^2)]^2$ is $\sigma$ to the power of 4. And that is as far as I got.

$\endgroup$
  • $\begingroup$ Your expressions are very difficult to read. You need to edit and present your question in a better way. $\endgroup$ – smanoos Oct 16 '11 at 4:44
  • 2
    $\begingroup$ One way of expressing $Var(S^2)$ is given on the Wikipedia page for variance. $\endgroup$ – Mike Spivey Oct 16 '11 at 4:50
  • $\begingroup$ It doesn't show how they derived it. $\endgroup$ – MathMan Oct 16 '11 at 4:51
  • $\begingroup$ The solution to the question is in many books. You can easily find it. $\endgroup$ – smanoos Oct 16 '11 at 5:17
  • 1
    $\begingroup$ There is a derivation on MathWorld's Sample Variance Distribution page. They use the "divide by $N$" convention rather than the "divide by $N-1$" convention, though, so you might have to adjust for that. $\endgroup$ – Mike Spivey Oct 16 '11 at 5:39
68
$\begingroup$

Maybe, this will help. Let's suppose the samples are taking from a normal distribution. Then using the fact that $\frac{(n-1)S^2}{\sigma^2}$ is a chi squared random variable with $(n-1)$ degrees of freedom, we get $$\begin{align*} \text{Var}~\frac{(n-1)S^2}{\sigma^2} & = \text{Var}~\chi^{2}_{n-1} \\ \frac{(n-1)^2}{\sigma^4}\text{Var}~S^2 & = 2(n-1) \\ \text{Var}~S^2 & = \frac{2(n-1)\sigma^4}{(n-1)^2}\\ & = \frac{2\sigma^4}{(n-1)}, \end{align*}$$

where we have used that fact that $\text{Var}~\chi^{2}_{n-1}=2(n-1)$.

Hope this helps.

$\endgroup$
  • 13
    $\begingroup$ Remember that $(n-1)S^2/\sigma^2$ is only guaranteed to be $\chi^2$ when the sample is taken from a normal distribution, though. $\endgroup$ – Mike Spivey Oct 16 '11 at 6:09
  • 4
    $\begingroup$ The question posed is a general one, whereas the answer is distribution-specific. Not appropriate, I am afraid. $\endgroup$ – wolfies Apr 26 '13 at 7:49
  • 1
    $\begingroup$ @afsdfdfsaf Perhaps, you should ask that as a separate question. $\endgroup$ – Nana Apr 12 '14 at 18:48
  • 6
    $\begingroup$ The answer is extremely useful, but would have been even more useful if someone could reference why (n−1)S2/σ2 is a Chi squared $\endgroup$ – moldovean Mar 9 '15 at 7:06
  • 1
    $\begingroup$ @moldovean About as to why $(n−1)S^2/\sigma^2$ is a Ki2 distribution, I see it this way : $\sum(x_i-\overline{x})^2$ is the sum of the square value of N variables following normal distribution with expected value 0 and variance $\sigma^2$. Then, since all the $(x_i-\overline{x})/\sigma^2$ follow a normal standard distribution, $Y = \sum^N((x_i-\overline{x})/\sigma)^2 = \frac{1}{\sigma^2}\sum^N(x_i-\overline{x})^2 = \frac{(n-1)S^2}{\sigma^2}$ follows a ki2 with N degrees of freedom, and not with N-1 degrees of freedom. I don't know what I am missing... $\endgroup$ – mocquin Sep 27 '19 at 10:01
88
$\begingroup$

Here's a general derivation that does not assume normality.

Let's rewrite the sample variance $S^2$ as an average over all pairs of indices: $$S^2={1\over{n\choose 2}}\sum_{\{i,j\}} {1\over2}(X_i-X_j)^2.$$ Since $\mathbb{E}[(X_i-X_j)^2/2]=\sigma^2$, we see that $S^2$ is an unbiased estimator for $\sigma^2$.

The variance of $S^2$ is the expected value of $$\left({1\over{n\choose 2}}\sum_{\{i,j\}} \left[{1\over2}(X_i-X_j)^2-\sigma^2\right]\right)^2.$$

When you expand the outer square, there are 3 types of cross product terms $$\left[{1\over2}(X_i-X_j)^2-\sigma^2\right] \left[{1\over2}(X_k-X_\ell)^2-\sigma^2\right]$$ depending on the size of the intersection $\{i,j\}\cap\{k,\ell\}$.

  1. When this intersection is empty, the factors are independent and the expected cross product is zero.

  2. There are $n(n-1)(n-2)$ terms where $|\{i,j\}\cap\{k,\ell\}|=1$ and each has an expected cross product of $(\mu_4-\sigma^4)/4$.

  3. There are ${n\choose 2}$ terms where $|\{i,j\}\cap\{k,\ell\}|=2$ and each has an expected cross product of $(\mu_4+\sigma^4)/2$.

Putting it all together shows that $$\mbox{Var}(S^2)={\mu_4\over n}-{\sigma^4\,(n-3)\over n\,(n-1)}.$$ Here $\mu_4=\mathbb{E}[(X-\mu)^4]$ is the fourth central moment of $X$.

$\endgroup$
  • 1
    $\begingroup$ a related question on stats.SE asks provides a different solution, and asks for a reference, your input would be appreciated: stats.stackexchange.com/q/29905/2750 $\endgroup$ – Abe Jun 6 '12 at 16:56
  • 1
    $\begingroup$ @Abe Sorry, I don't have any references or worthwhile input. The above is a solution that I made up to teach my students. $\endgroup$ – user940 Jun 6 '12 at 19:07
  • 2
    $\begingroup$ thanks, an answer to the stats.SE question solved my confusion: the discrepancy was use of kurtosis ($\mu_4$, the fourth central moment) vs excess kurtosis ($\kappa = \frac{\mu_4}{\sigma^4} -3$); one reference is Mood Graybill and Boes, 1974, Introduction to the Theory of Statistics $\endgroup$ – Abe Jun 6 '12 at 22:39
  • 1
    $\begingroup$ @ByronSchmuland It's probably too basic, but I have problems with the first expression of variance as a pair of indices. Is there any way you can send a reference for this equation? Ty $\endgroup$ – Antoni Parellada Aug 3 '15 at 14:21
  • 6
    $\begingroup$ In the derivation, how do we see claims 2 and 3, i.e. that the expected value of $$\left[{1\over2}(X-Y)^2-\sigma^2\right] \left[{1\over2}(X-Y)^2-\sigma^2\right]$$ is $(\mu_4+\sigma^4)/2$, for X,Y i.i.d? $\endgroup$ – Emolga Jul 28 '17 at 11:35
6
$\begingroup$

There can be some confusion in defining the sample variance ... 1/n vs 1/(n-1). The OP here is, I take it, using the sample variance with 1/(n-1) ... namely the unbiased estimator of the population variance, otherwise known as the second h-statistic:

h2 = HStatistic[2][[2]] 

These sorts of problems can now be solved by computer. Here is the solution using the mathStatica add-on to Mathematica. In particular, we seek the Var[h2], where the variance is just the 2nd central moment, and express the answer in terms of central moments of the population:

CentralMomentToCentral[2, h2] 

enter image description here

We could just as easily find, say, the 4th central moment of the sample variance, as:

CentralMomentToCentral[4, h2]

enter image description here

$\endgroup$
  • $\begingroup$ Does your program also let you handle dependent random variables? Presumably, then the result would be in terms of higher-order covariances. $\endgroup$ – Thomas Ahle Aug 15 '19 at 15:17
  • $\begingroup$ Yes - it works for dependent random variables too. There is a multivariate example at: math.stackexchange.com/questions/589865/… $\endgroup$ – wolfies Aug 16 '19 at 21:54
1
$\begingroup$

Showing the derivation of $E(\left[{1\over2}(X-Y)^2-\sigma^2\right]^2) = (\mu_4+\sigma^4)/2$ of user940:

LHS:

$E(\left[{1\over2}(X-Y)^2-\sigma^2\right]^2) = E(\frac{1}{4}(X-Y)^4 - (X-Y)^2 \sigma^2 + \sigma^4) = E(\frac{1}{4}(X-Y)^4) - 2\sigma^2\sigma^2 + \sigma^4 = E(\frac{1}{4}(X-Y)^4) - \sigma^4 = \frac{1}{4}E(X^4 -4X^3Y +6X^2Y^2 -4XY^3 + Y^4) -\sigma^4 = \frac{1}{4}(2E(X^4) -8E(X)E(X^3) +6 E(X^2)(X^2)) - \sigma^4 = \frac{1}{2}(E(X^4)-4E(X)E(X^3) +3 E(X^2)(X^2) - 2\sigma^4)$

I use the fact that $E((x-y)^2) = 2\sigma^2$ here.

RHS:

$\require{cancel} (\mu_4+\sigma^4)/2 = \frac{1}{2}(E((X-\mu)^4) + \sigma^4) = \frac{1}{2}(E((X-E(X))^4) + \sigma^4) = \frac{1}{2}(E(X^4 -4X^3E(X) + 6X^2E(X)^2 -4XE(X)^3 + E(X)^4) + \sigma^4) = \frac{1}{2}(E(X^4 -4X^3E(X) + 6X^2E(X^2) - 6X^2\sigma^2 -4XE(X)(E(X^2)-\sigma^2) + (E(X^2)-\sigma^2)^2) + \sigma^4) = \frac{1}{2}(E(X^4) -4E(X)^3E(X) + 6E(X)^2E(X^2) - 6E(X)^2\sigma^2 -4E(X)^2(E(X^2)-\sigma^2) + (E(X^2)-\sigma^2)^2 + \sigma^4) = \frac{1}{2}(E(X^4) -4E(X)^3E(X) + 6E(X)^2E(X^2) - \cancel{6E(X)^2\sigma^2} -4E(X^2)E(X^2) +\cancel{4E(X^2)\sigma^2 +4E(X^2)\sigma^2} - 4\sigma^4 + E(X^2)^2-\cancel{2E(X^2)\sigma^2} + \sigma^4 + \sigma^4) = \frac{1}{2}(E(X^4) -4E(X)^3E(X) + 3E(X)^2E(X^2) - 2\sigma^4)$

I use the fact that $E(x) = \mu$ and that $E(x)^2 = E(x^2) - \sigma^2$

Now LHS = RHS.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.