Teasing apart an explanation of the Central Limit Theorem

I'm looking at the central limit theorem, and cannot see in the explanation given to me how the average of identical distributions results in the normal distribution.

I am told to consider a sequence of independent and identically distributed variables, and then suppose they have a mean $\mu$ and variance $\sigma^2$.

Then if we take the linear combination of each evenly weighted by the total count:

$$\bar{X} = \frac{X_1+X_2+\cdots+X_n}{n}$$

...we have $E(\bar{X}) = \mu$ and $\operatorname{Var}(\bar{X})=\frac{\sigma^2}{n}$, which in turn means:

$$\bar{X}\sim N\left(\mu,\frac{\sigma^2}{n}\right)$$

While this appears logical, if I use a computer to generate an array of exponentially distributed random variables (such as stats.expon.rvs() from the Python scipy library), then as $n$ in this array increases, a histogram of this data shows an exponential distribution of $\mu=1$ and $\sigma=1$, and not a normal one.

I know that the collection of points in any subrange of the exponential's domain $x = [0, +\infty)$ approaches a normal distribution as the subrange gets smaller and the $n$ in that subrange increases, since the exponential distribution in that subrange provides a mean around which random independent samples in the subrange follow a normal distribution.

But this only applies to collections of samples in subranges, and not to the entire distribution itself.

Unless I'm missing something, it appears averaging any random set of samples from an entire given distribution and then taking a histogram of that set just results in that initial distribution, and not a normal one as mentioned.

I have good news for you: You are confused.

Firstly, what is a sample? You have $X_1,\ldots,X_n$. That is one sample, not a set of $n$ samples.

If you base a histogram on $X_1,\ldots,X_n$, and $n$ is large, then with high probability, the histogram will look like the density of the "original" distribution.

However, that is only one sample, and it is the whole sample, not the mean of the sample. The sample has a mean, $\bar X=(X_1+\cdots+X_n)/n$. That is the mean of only one sample. One then takes the mean of another sample of $n$ observations. And another. And another. And so on. It is those means that will yield approximately a normal distribution if $n$ is large.

• You, sir, are very correct! :) However, with your and marshall's explanations, I think I have it down. As you say, I was not seeing the situation as a matter of independent samples. I was averaging points on an individual sample's distribution, and not that for separate independent samples. Therefore, the histogram was resulting in the distribution of events dependent on one sample, and not the normal distribution inherently awarded by multiple independent samples shooting for the same target (a la "darts on a dartboard" as is the common illustration for normal distribution). Jun 16 '15 at 5:07
• Another good news/confusion, worthy of note: $$\bar{X}\sim N\left(\mu,\frac{\sigma^2}{n}\right)$$ is not true (except when the i.i.d. random variables are normal).
– Did
Jun 16 '15 at 6:51
• @Did : The "$N$" part is indeed not true, but the $\mu$ and $\sigma^2/n$ are true even without identical distribution and without independence, as long as the expected value is $\mu$ for all of the random variables and the variance is $\sigma^2$ for all of them and they are uncorrelated (the Gauss–Markov assumptions, not to be confused with Gaussian Markov processes). ${}\qquad{}$ Jun 16 '15 at 17:06

That is exactly it! You said,

"it appears averaging any random set of samples from an entire given distribution and then taking a histogram of that set just results in that initial distribution".

The law of large numbers is the assumption that the limit of those sample means and variance converges to the actual mean and variance respectively, CLT is that as they do so, the differences fluctuate normally.

See classical CLT from https://en.wikipedia.org/wiki/Central_limit_theorem

• Lightbulbs have gone on! Thanks! This an Michael Hardy's response outlined I was not using independent samples for the average. For some reason, how it's been explained to me so far made this tough to visualize. I wasn't seeing it as the collective differences being normally distributed, but thats a unique way to conceptualize it. Jun 16 '15 at 5:13