# Intuition about the Central Limit Theorem

I'm studying statistics, and would like to better understand the Central Limit Theorem. The proof I found on Wikipedia requires some previous knowledge I do not currently possess.

Is there a quick intuitive explanation you can give as to why this theorem is correct?

-
Have you seen this already? –  Ｊ. Ｍ. Dec 4 '10 at 11:50
@J. M., thanks for the link. Although, I didn't find a quick explanation that I understood, sadly. Specifically, I'm interested in why sampling any distribution (even a non-symmetric one) will lead to a normal, symmetric distribution, for large enough samples. –  chaoticdawn Dec 4 '10 at 12:01
en.wikipedia.org/wiki/Illustration_of_the_central_limit_theorem might help provide a little intuition. Great question, by the way! –  Qiaochu Yuan Dec 4 '10 at 12:39
Not to confuse you, but the CLT only applies in the finite second moment case (finite variance). Levy-Stable distributions are also a convergent distribution of sums of random variables. –  user4143 Dec 4 '10 at 16:46

I don't think you should expect any short, snappy answers because I think this is a very deep question. Here is a guess at a conceptual explanation, which I can't quite flesh out.

Our starting point is something called the principle of maximum entropy, which says that in any situation where you're trying to assign a probability distribution to some events, you should choose the distribution with maximum entropy which is consistent with your knowledge. For example, if you don't know anything and there are $n$ events, then the maximum entropy distribution is the uniform one where each event occurs with probability $\frac{1}{n}$. There are lots more examples in this expository paper by Keith Conrad.

Now take a bunch of independent identically distributed random variables $X_i$ with mean $\mu$ and variance $\sigma^2$. You know exactly what the mean of $\frac{X_1 + ... + X_n}{n}$ is; it's $\mu$ by linearity of expectation. Variance is also linear, at least on independent variables (this is a probabilistic form of the Pythagorean theorem), hence

$$\text{Var}(X_1 + ... + X_n) = \text{Var}(X_1) + ... + \text{Var}(X_n) = n \sigma^2$$

but since variance scales quadratically, the variance of $\frac{X_1 + ... + X_n}{n}$ is actually $\frac{\sigma^2}{n}$; in other words, it goes to zero! This is a simple way to convince yourself of the (weak) law of large numbers.

So we can convince ourselves that (under the assumptions of finite mean and variance) the average of a bunch of iid random variables tends to its mean. If we want to study how it tends to its mean, we need to instead consider $\frac{(X_1 - \mu) + ... + (X_n - \mu)}{\sqrt{n}}$, which has mean $0$ and variance $\sigma^2$.

Suppose we suspected, for one reason or another, that this tended to some fixed limiting distribution in terms of $\sigma^2$ alone. We might be led to this conclusion by seeing this behavior for several particular distributions, for example. Given that, it follows that we don't know anything about this limiting distribution except its mean and variance. So we should choose the distribution of maximum entropy with a fixed mean and variance. And this is precisely the corresponding normal distribution! Intuitively, each iid random variable is like a particle moving randomly, and adding up the contributions of all of the random particles adds "heat," or "entropy," to your system. (I think this is why the normal distribution shows up in the description of the heat kernel, but don't quote me on this.) In information-theoretic terms, the more iid random variables you sum, the less information you have about the result.

-

There's an almost-formal argument using cumulants. Given a random variable $X$, define its moment generating function $$M(X) = E[e^{tX}].$$ It's called the moment generating function since opening the Taylor series of the exponential, we get $$M(X) = 1 + E[X]t + \frac{1}{2}E[X]^2t^2 + \cdots.$$ The moment generating function is useful because of its relation to convolution of two independent random variables: $$M(X+Y) = E[e^{t(X+Y)}] = E[e^{tX}e^{tY}] = E[e^{tX}]E[e^{tY}] = M(X)M(Y).$$ One proof of the CLT takes the route of the mgf, but we would like to replace the multiplication by a addition since we only really know how to handle sums. So we define the cumulant generating function $$K(X) = \log M(X).$$ We can calculate the first few coefficients (which are called cumulant) by substituting into the (formal) power series of $\log(1+x) = x - x^2/2 + \cdots$: $$K(X) = \log (1+E[X]t+E[X^2]t^2/2 + \cdots) = E[X]t + E[X^2]t^2/2 - (E[X]^2t^2 + E[X]E[X^2]t^3 + E[X^2]t^4/4)/2 + \cdots = E[X]t + V[X]t^2/2 + \cdots.$$ Also, if $X$ and $Y$ are independent then $$K[X+Y] = K[X]+K[Y].$$ Now suppose $X_1,\ldots,X_n$ are iid variables distributed like $X$ with zero expectation. Then $$K[X_1+\cdots+X_n] = nK[X] = \frac{1}{2}nV[X]t^2 + \frac{1}{6}nK_3(X)t^3 + \cdots,$$ where $K_m(X)$ are just the (normalized) coefficients of the cgf, i.e. the cumulants (they are normalized by $1/m!$). If we scale this sum down by $\sqrt{n}$, then the second cumulant becomes $V[X]$ (i.e. the variance is the same), but the rest of the cumulants $K_m$ for $m \geq 3$ get multiplied by $n^{1-m/2} \rightarrow 0$, so in the limit they disappear, and the cumulant of the limit is just $$K\left[\frac{X_1+\cdots+X_n}{\sqrt{n}}\right] = \frac{1}{2}V[X]t^2.$$ Therefore there is one 'domain of attraction' for distributions, which must be the normal distribution with zero mean and variance $V[X]$; it can be calculated directly from this representation. The same idea can be used to analyze the case where the variables are independent but not identically distributed. The main step missing to make this proof formal is reasoning about the limit distribution from the limit cgf; this is the Levy continuity lemma, which shows that the 'inverse Fourier transform' is continuous.

Had we taken the route of mgf's, we would have had to use the identity $(1+1/n)^n \rightarrow e^n$ somewhere, but otherwise the argument would be much the same.

-