3
$\begingroup$

I use Statgraphics for working with statistics.

I generated 3 Binomial samples:

1) N=100, p=0.5

2) N=100, p=0.01

3) N=100, p=0.99

First sample looks like Normal sample:

First sample

And it passes Kolmogorov-Smirnov test:

KS test

So I have theoretical question from probability theory: why first sample acts like Normal and why other two don't?

$\endgroup$
3
  • 1
    $\begingroup$ It's not Bernoulli, it's Binomial (sum of Bernoulli). You may want to read this ("Normal approximation" to the Binomial, on Wikipedia). $\endgroup$
    – Clement C.
    Dec 16 '15 at 22:06
  • $\begingroup$ @ClementC. Oh, a typo. Thanks! Edited my question. $\endgroup$ Dec 16 '15 at 22:07
  • $\begingroup$ For $p=0.5$ the approximation is great in relative terms until you get into the tails where it starts faulting. Also note that $B(100,0.01)$ is Poisson-like not far from the mean. $\endgroup$
    – A.S.
    Dec 16 '15 at 22:53
3
$\begingroup$

Let $X_1, X_2, \dots$ be a sequence of independent, identically distributed random variables with common mean $\mu$ and variance $\sigma^2$. Then the sum $T_n = \sum_{i=1}^n X_i$ of the first $n$ of the $X_i$ has $E(T_n) = n\mu$ and $Var(T_n) = n\sigma^2$. Thus $Z_n = \frac{T_n - n \mu}{\sqrt{n}\sigma}$ has $E(Z_n) = 1$ and $Var(Z_n) = 1.$

The Central Limit Theorem (CLT), says that the sequence $Z_n$ converges in distribution to standard normal. That is, $P(Z_n \le z) \rightarrow \Phi(z)$, for any real $z$, where $\Phi$ is the standard normal CDF. (A similar statement can be made for averages $\bar X_n = (1/n)\sum_{i=1}^n X_i$ and $Z_n = \frac{\bar X_n - \mu}{\sigma/\sqrt{n}}.$)

Whenever you want to put a limit theorem to practical use (such as approximation), your first question should be "how fast is the convergence?" Infinity itself is a 'long way away' Roughly speaking, the convergence in the CLT is more rapid for symmetrical random variables $X_i$ than for skewed ones. Thus, the sum of only ten independent $X_i \sim Unif(0,1)$ (symmetrical) is very nearly normal, while the sum of 25 independent random variables $X_i \sim Exp(1)$ (markedly skewed) is clearly not so well approximated by a normal distribution. The diagram below illustrates this with 10,000 such sums of each type of random variable. The appropriate normal density is shown with each histogram.

enter image description here

The speed of convergence is quite fast in some instances and rather slow in others. A binomial random variable $X \sim Binom(n, \theta)$ is the sum of $n$ independent Bernoulli random variables with success probability $\theta.$ So, for sufficiently large $n$ a Binomial random variable will have a distribution that is 'approximately' normal. In the binomial case, the convergence is much faster if $\theta \approx 1/2$ (symmetrical) than if $\theta$ is near 0 or 1. You have already discovered that $Binom(100, .5)$ is closer to normal than $Binom(100, .01).$

Sometimes a 'rule of thumb' is used. It says that the approximation of normal to binomial is 'reasonably' good if $n\theta$ and $n(1-\theta)$ both exceed 5. Few 'rules of thumb' are always accurate. There are better rules for normal/binomial fit, but this one is often quoted, probably because it is usually pretty good and it is easy to remember.

$\endgroup$
2
  • $\begingroup$ "Whenever you want to put a limit theorem to practical use (such as approximation), your first question should be "how fast is the convergence?" Infinity itself is a 'long way away' Roughly speaking, the convergence in the CLT is more rapid for symmetrical random variables XiXi than for skewed ones. " I don't clearly understand this logical chain. Why do we need skewness and only skewness to understand how fast is the convergence? $\endgroup$ Dec 17 '15 at 4:44
  • 1
    $\begingroup$ Intuitively, the normal distribution is symmetrical, so it is 'easier' for sums of symmetrical random variables to become symmetrical as we include more terms into the sum. Analytically, see the Comment by @A.S. about the proof of the CLT. (Skewness is not quite the whole story, but a big and easily recognized part of the story.) $\endgroup$
    – BruceET
    Dec 17 '15 at 5:37
3
$\begingroup$

Any sum of a large number of identically distributed variables starts to look like a normal distribution; it's just that your first sample reached this approximation more quickly because it started out more "evenly". It's still discrete, though: the "convergence" that we are looking at does not refer to a convergence of density functions.

$\endgroup$
8
  • $\begingroup$ You could add a reference to the central limit theorem in your answer ;) $\endgroup$
    – Tryss
    Dec 16 '15 at 22:18
  • $\begingroup$ @Justpassingby But uniform distribution won't. $\endgroup$ Dec 16 '15 at 22:24
  • $\begingroup$ @Tryss but second and third samples should be approximated by Poisson distribution and not normal. Why doesn't CLT work in these cases? $\endgroup$ Dec 16 '15 at 23:27
  • 1
    $\begingroup$ @Michael $B(10^6,0.01)$ will be well-approximated by Normal-CLT. The problem with $B(100,0.01)$ is that its skew $\approx 1$ and skew of the normal is $0$ - so they can't be close. In general, you need $n\min \{p,1-p\}\gg 1$ for normal approximation/CLT to kick in. $\endgroup$
    – A.S.
    Dec 16 '15 at 23:48
  • 1
    $\begingroup$ @michaeluskov The sum of a large number of uniform variables also approximates a normal variable with the same average. $\endgroup$ Dec 17 '15 at 8:15
3
$\begingroup$

One reason is Stirling's formula.

enter image description here

Instead of central limit theorem, you can use De Moivre–Laplace theorem. See the second of two proofs on Wikipedia which makes use of Stirling's formula. De Moivre-Laplace theorem stated on Wikipedia contains this approximation:

enter image description here

Therefore, the first step of the proof is rewriting $\binom{n}{k}$ as factorials and approximating with Stirling's formula

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.