2
$\begingroup$

I'm doing an experiment comparing 10,000 rolls of 5 dice (those 5 rolls were then summed, so the population mean is 17.5) and dividing them into sets of ten with sample size N = 10. The same experiment is repeated except with 3000 rolls, but the sample size decreases to N = 3. Both experiments have 1000 samples, just with different sample sizes.

Now, I took the AVERAGE sample standard deviation for each experiment and to my great confusion, the experiment with the SMALLER sample size had a SMALLER average sample standard deviation than the experiment with the LARGER sample size. I thought this was a fluke and re-generated the random numbers only to find the same result.

In other words, this means that the sample standard deviation for each sample of N =3 is on average smaller than each sample of N = 10. How is this possible?

Intuitively, one would think that with a larger sample size the "spread" between values would be decreased because any individual improbable value would have less effect on the spread. Sort of like for the mean. While for a smaller sample size, the variables would be less "controlled" and thus on average more spread out.

Is my intuition just wrong? And is this plain to see mathematically?

FOLLOW UP: I talked to my professor and he said that a weird phenomenon happens in statistics where sample standard deviations tend to be underestimations rather than overestimations of the population standard deviation. And that as sample size goes up, the sample standard deviation goes up because it becomes "more accurate."

Is this true? And why does that happen? Is there both an intuitive explanation as well as a mathematical proof?

$\endgroup$
6
  • $\begingroup$ Are you computing standard deviation or standard error? Think about the standard deviation you would see with $N = 1$. It would always be $0$. It's smaller for lower $N$ because your average is always as close as possible to the center of the specific data (as opposed to the distribution). $\endgroup$
    – Brian Tung
    Commented Sep 22, 2016 at 18:13
  • $\begingroup$ @Ian: The standard deviation of the mean would be, but not the distribution of the quantity itself. $\endgroup$
    – Brian Tung
    Commented Sep 22, 2016 at 18:14
  • $\begingroup$ Further question: How much smaller is the $N = 3$ standard deviation. Is it about $3/4$ of the $N = 10$ standard deviation? $\endgroup$
    – Brian Tung
    Commented Sep 22, 2016 at 18:16
  • $\begingroup$ Can you post the standard deviations you calculated? $\endgroup$
    – Joey Zou
    Commented Sep 22, 2016 at 18:16
  • $\begingroup$ Oh, yes, the standard deviation is normalized in just the right way that it will tend to oscillate around the population standard deviation as the sample size grows. $\endgroup$
    – Ian
    Commented Sep 22, 2016 at 18:22

4 Answers 4

2
$\begingroup$

Absent further clarification from the OP, here's what I think is happening:

Each sample consists of $N$ values drawn from a binomial distribution spanning the integer range $[5, 30]$. As a population, the draws from this binomial distribution have mean $17.5$ and standard deviation about $3.82$ (I think, I have to double check this).

With a sample size of just $N$, however, the average of the sample is not generally $17.5$. It will be some value which is overall closer to the sample than the population mean would be. Hence, the standard deviation of that $N$-count sample, treated as a population, will systematically underestimate the standard deviation of the population.

For example, with $N = 3$, if you draw $12, 14, 16$, you have an average of $14$. The standard deviation of the sample, treated as the population, is about $1.63$. But the RMS distance from the actual population mean of $17.5$ is $6.69$. The disparity arises from the sample average being closer to the data, overall, than the population mean.

The larger $N$ is, the smaller the expected disparity, and that's perhaps why you obtain a smaller standard deviation for $N = 3$ than you do for $N = 10$.


ETA: Dividing by $N-1$ instead of $N$ should produce an unbiased estimator for the population standard deviation.

$\endgroup$
1
$\begingroup$

The issue has to do with the choice of statistic you are calculating from your samples. If you are taking the sum of the sample, then for a sample of size $N = 10$, you will observe sums ranging from a minimum of $10$ to a maximum of $60$. But if your sample is of size $N = 3$, the range of the sum is only from $3$ to $18$.

$\endgroup$
1
  • $\begingroup$ I think you misunderstood, ill re-clarify in my question. $\endgroup$
    – Nova
    Commented Sep 22, 2016 at 18:06
1
$\begingroup$

There are two different issues here related to how you calculate variance and standard deviation from a sample drawn from a population with unknown mean $\mu$ and variance $\sigma^2$.

  1. An estimate of variance using $\frac{1}{n} \sum (x_i-\bar x)^2$ is biased in the sense that its expectation is $\frac{n-1}{n}\sigma^2$ rather than $\sigma^2$. This is caused by the $x_i$ tending on average to be closer to $\bar x$ than they are to $\mu$; if you had known $\mu$ then the larger $\frac{1}{n} \sum (x_i-\mu)^2$ would have been unbiased, but you did not. This leads to Bessel's correction and so it is common to use $s^2 = \frac{1}{n-1} \sum (x_i-\bar x)^2$ as an unbiased estimator of variance. (It is from a sample and in a sense also a random variable, so may still be too high or low, even though its expectation is $\sigma^2$.)

  2. If you use $s^2$ to estimate the variance $\sigma^2$ then it is natural to use $\sqrt{s^2}$ to estimate the standard deviation $\sigma$. But $\sqrt{\,}$ is not a linear function: it is concave, so $\mathbb E[\sqrt{s^2}] \lt \sqrt{\mathbb E[s^2]}=\sigma$. For large $n$ you have $s^2$ tending to be a closer estimate of $\sigma^2$ than it tends to be for small $n$, so this bias due to the shape of $\sqrt{\,}$ tends to be more substantial for values which are "more spread out", i.e. here for smaller $n$.

To illustrate this, a single fair six-sided die has expectation $\mu=\frac72=3.5$ and variance $\sigma^2=\frac{35}{12} \approx 2.916667$ and standard deviation $\sigma=\sqrt{\frac{35}{12}}\approx 1.707825$. The following table shows the expected values for $n$ dice for estimates using the average, the uncorrected estimate of variance, the Bessel-corrected estimate of variance, and the square-root of the Bessel-corrected estimate of variance (though remember that each sample involves randomness and will give different values, some higher and some lower).

$$\begin{array} \,n & E\left[X_i\right] & E\left[\frac{1}{n} \sum (X_i-\bar X)^2\right] & E\left[\frac{1}{n-1} \sum (X_i-\bar X)^2\right] & E\left[\sqrt{\frac{1}{n-1} \sum (X_i-\bar X)^2}\right] \\ 1 & 3.5 & 0 & \frac00 & \frac00 \\ 2 & 3.5 & 1.458333 & 2.916667 & 1.374930 \\ 3 & 3.5 & 1.944444 & 2.916667 & 1.556217 \\ 4 & 3.5 & 2.187500 & 2.916667 & 1.618076 \\ 5 & 3.5 & 2.333333 & 2.916667 & 1.646405 \\ 6 & 3.5 & 2.430556 & 2.916667 & 1.661957 \\ 7 & 3.5 & 2.500000 & 2.916667 & 1.671558 \\ 8 & 3.5 & 2.552083 & 2.916667 & 1.677984 \\ \lim\limits_{\to \infty} & 3.5 & 2.916667 & 2.916667 & 1.707825 \end{array} $$

and this illustrates your observation that "the experiment with the smaller sample size had a smaller average sample standard deviation than the experiment with the larger sample size".

$\endgroup$
0
$\begingroup$

I believe Henry's answer gives the correct explanation. However, sometimes it helps to visualise the answer. Thus, here the plot of the (at first glance) counter-intuitive result enter image description here

As claimed by the OP, if we calculate the standard deviation of the sets, we find an increasing function -- red dots. In contrast, if we first calculate the variance of the sets and then rescale the result (by taking the square root), we obtain an unordered function -- blue dots.

What I have done:

  1. Roll $n_{dice}=5$ random dices.
  2. Calculate the sum of these five dices, and let's call the sum a single "set".
  3. Generate $n_{set}$ "sets" and calculate their standard deviation (and variance). Store these results in two variables.
  4. Repeat this $n_{sim}=10\,000$ times.
  5. Calculate the average value of the two variables stored in step 3. The only parameter which we change in the procedure is the value of $n_{set}$ in step 3. We start with $n_{set}=3$ and increase it till $n_{set}=10$.

R code, which generates the plot:

nDices = 5
nSets  = 3:10
nSim   = 10e3

data.std = vector(mode="numeric", length=nSim)
data.var = vector(mode="numeric", length=nSim)
Mean_SD  = vector(mode="numeric", length=length(nSets))
Mean_VAR = vector(mode="numeric", length=length(nSets))
for ( iSet in 1:length(nSets) ){
    nSet      = nSets[[iSet]]
    randomSum = vector(mode="numeric", length=nSet)
    for ( iSim in 1:nSim ){
        # A single "set" consists of nDices=5 dices. 
        # We first calculate the SUM of each "set", and then 
        # we calculate the standard deviation of nSet sets.
        randomSums = replicate(nSet, 
                               sum(sample(x=1:6, size=nDices, replace=TRUE)))
        data.std[[iSim]] = sd(randomSums)
        data.var[[iSim]] = var(randomSums)
    }
    # Finally, we calculate the average value of the standard deviation:
    Mean_SD[[iSet]]  = mean(data.std)
    Mean_VAR[[iSet]] = mean(data.var)
    #cat(paste0('nSet=', nSet, ' => E[Sd[randomSum]] = ', Mean_SD[[iSet]], '\n'))
}

## Plot
dataFrame = data.frame(nSets = c(nSets, nSets),
                       Mean_SD  = c(Mean_SD, sqrt(Mean_VAR)), 
                       grp = factor(rep(c('SD', 'sqrt(VAR)'), each=length(nSets))))
library(ggplot2)
gg = ggplot(dataFrame, aes(x=nSets, y=Mean_SD, col=grp)) + 
    geom_point(alpha=0.4) + 
    xlab('Number of averages sets')
print(gg)
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .