Estimating population standard deviation with sample standard deviation

At 4:30 of this video the author decided to estimate the standard deviation of the population with sample standard deviation (sample size was $100$).

In the next video, the author mentioned that it was reasonable because the sample size greater than $30$. Well, what tells us that we could estimate standard deviation in this way? Why is $30$ that magical boundary? Does it have anything to do with Central Limit Theorem? (I guess not, because we don't calculating the standard deviation of the mean, so it's not related in any way).

• The key is in the bias of the estimator, so depending on the percentage you accept, it changes. If you were working in some medical field where you work with confidence levels of $99.9\%$, $30$ would be clearly too few cases. If you are ok with confidence levels around $95\%$, that's way more acceptable. I can't recall how to calculate the bias, but I bet that if you sort it out, you'll find it answers your question (most probably will give a bias of around $5\%$) – Masclins May 18 '15 at 14:45

2 Answers

At the root, the issue here seems to be whether to use the z-statistic or the t-statistic in finding a confidence interval for the population mean $\mu$ or in testing a hypothesis about $\mu.$

Suppose $X_1, X_2, \dots, X_n$ is a random sample from a normal population of which both the mean $\mu$ and the standard deviation $\sigma$ are unknown. We wish to find a 95% confidence interval (CI) for $\mu.$

If we knew $\sigma$ then $$Z = \frac{\bar X - \mu}{\sigma/\sqrt{n}} \sim Norm(0, 1).$$ Thus $$P\left\{-1/96 \le \frac{\bar X - \mu}{\sigma/\sqrt{n}} \le 1.96\right\} = 0.95,$$ in which $\mu$ can be isolated in a few steps of algebra to $$P\{\bar X - 1.96\sigma/\sqrt{n} \le \mu \le \bar X + 1.96\sigma/\sqrt{n}\} = 0.95.$$ And so we say that a 95% CI for $\mu$ is $\bar X \pm 1.96\sigma/\sqrt{n},$ in which all quantities $\bar X, \sigma,$ and $n$ are known. The numbers $\pm 1.96$ are chosen because they cut 2.5% probability from the upper and lower tails of the standard normal distribution, leaving 95% in the center.

In case $\sigma$ is unknown, it is convenient to use the sample standard deviation $S$ instead, claiming that $\bar X \pm 1.96 S/\sqrt{n}$ or perhaps $\bar X \pm 2 S/\sqrt{n},$ is an approximate 95% CI for $\mu.$ If $n \ge 30,$ this approximation is pretty good, for reasons we see just below.

If $\sigma$ is not known, the exact distribution is
$$T = \frac{\bar X - \mu}{S/\sqrt{n}} \sim T(n-1),$$ Student's t distribution with $n-1$ degrees of freedom. Then an exact 95% CI for $\mu$ is $\bar X \pm t^* S/\sqrt{n},$ where $t^*$ cuts 2.5% of probability from the upper tail of $T(n-1)$ and, by symmetry, $-t^*$ cuts 2.5% from the lower tail. Looking at tables of the t distribution we see that for $n \ge 30$ (or $n-1\le 29$), $t^*$ is approximately 2.0. So the approximate procedure with the standard normal distribution and the exact procedure with Student's t distribution amount to about the same thing.

For smaller values of $n$, the values of $t^*$ get noticeably larger. For example if $n = 10$, we have $t^* = 2.262.$ Thus the 95% CI gets longer (less precise). You can think of this loss of precision as a 'penalty' for having to estimate $\sigma$ by $S$ instead of knowing the exact value of $\sigma.$

There are a few good reasons to forget the 'rule of 30' altogether:

First, it 'works' only for 95% CIs. For a 99% CI we need to cut 0.5% of probability from each tail: the normal cut-off value is $z^* = 2.576$ and we need to increase the sample size to about $n = 60$ before $t^* \approx 2.6.$

Second, in using statistical software, either we know the exact value of $\sigma$ or the program will approximate it from data as $S.$ From the start, we have to know whether we are doing a z-interval or a t-interval. Using an unnecessary rule about sample size only confuses issue. The correct rule is: use z-procedures is $\sigma$ is known (and it usually isn't in practice); use t-procedures of not.

Third, some authors of elementary books try to use the 'rule of 30' (without any theoretical justification) for various kinds of limiting procedures, applicability of the Central Limit Theorem, safe use of t-procedures for non-normal data, and so on. In these applications, 30 is seldom an appropriate dividing line.

• Excellent answer! In the classes I missed the 95% CI vs 99% CI sample size reasoning. It makes perfect sense now. – Kocur4d May 9 '18 at 9:22
• Fantastic explanation! I'm just wondering how to handle $X$ being a discrete random variable in combination with a low sample size. Using the sample standard deviation $S$ as an estimator for $\sigma$ does not seem right; say my random variable has two possible outcomes: 0 or 1, with $P[X=0]=P[X=1]=0.5$. If I take only two samples, it may very well come out to be $X_0=X_1=1$ and that will result in $S=0$. That means, using your equation above, that the 99% CI (or even the 99.999% CI) will be $[1,1]$ because $S=0$. How should I estimate $\sigma$ instead? – rem Nov 6 '18 at 8:13
• @rem: Certainly, these t and z methods do not apply to your specific example. // My explanation is for a sample from a normal population. If the dist'n of the $X_i$ is discrete, and $n$ is large enough that $\bar X$ is approximately normal, then some authors might suggest using z or t methods as approximations. // If the type of discrete dist'n is known (e.g., binomial, Poisson, etc.) then I would seek an exact method based on that type of distribution. – BruceET Nov 6 '18 at 18:59

Neither of the two methods of estimating the population standard deviation from the sample produces an unbiased estimate, though the $\frac{1}{n-1}$ method does produce an unbiased estimate of the variance.

If you compare the two estimates of the variance $$s_s^2 = \frac{\sum_i^n (x_i - \bar{x})^2}{n-1}$$ with $$s_p^2 = \frac{\sum_i^n (x_i - \bar{x})^2}{n}$$ then clearly $\frac{s_p^2}{s_s^2} = \frac{n-1}{n}$ and so $$\dfrac{s_p}{s_s} = \sqrt{1-\frac{1}{n}} \approx 1 - \frac{1}{2n}$$ which gets closer to $1$ as $n$ increases (for $n=30$ it is about $0.983$ and for $n=100$ about $0.995$) and this factor is less important than the uncertainty in estimating the population standard deviation from a random sample.

• Moreover: Although sample variance $S^2$ computed using $n-1$ is unbiased for $\sigma^2,$ unbiasedness does not 'survive' nonlinear transformations. For normal data $E(S) = [\sqrt{2/(n-1)}\Gamma(n/2)/\Gamma((n-1)/2)]\sigma.$ Thus for a normal sample of size $n=5,$ we have $E(S) \approx .94\sigma.$ The coefficient in [ ]s converges to 1 with increasing $n$. See Wikipedia on 'Unbiased estimation of standard deviation'. – BruceET Oct 6 '16 at 17:59