Help understanding the difference between the LLNs and CLT?

(Sorry ahead of time, I don't know LaTeX...)

I've been studying some basic probability theory recently, and I'm having a little trouble understanding why the LLNs and CLT aren't contradictory. Given IID variables {X1,X2,...} each with mean M and variance V, both LLNs seem to say that the average of the first N variables in this sequence approaches M as N approaches infinity. The CLT seems to say that as N approaches infinity, the distribution of this average approaches a normal distribution with mean M and variance V.

The problem I'm having is that it seems like the distribution of the average should converge to something like a discrete variable with a PMF like 1 at M and 0 everywhere else. This is because the strong LLN says the the average MUST be M as N approaches infinity. Instead, the normal distribution given by the CLT seems to say that there's of chance of the average not being M as N approaches infinity, which seems to contradict the strong LLN.

I realize I have a flaw in my logic, since a lot of really smart people came up with these theorems :P but I'd just like to know where I'm going wrong. I'm assuming it has something to do with the differences between convergences of probability, distribution, etc, but I don't know too much about those so if someone could explain it clearly I'd really appreciate it.

Thanks!

-

The problem is that you left out the scaling. The average does converge to a constant: what converges to a normal distribution is $$\left(\text{average} - M\right)\sqrt{n}$$

-
If the (average-M) part is converging to 0, won't the $\sqrt{n}$ part not matter? –  Rotfuchs Jun 21 '13 at 19:37
Of course it will matter. $0 \times \infty$ is an indeterminate form. –  Robert Israel Jun 22 '13 at 21:50
I see, sorry about that. Thanks for the help! –  Rotfuchs Jun 23 '13 at 5:32

Heres my 5 cents.

Central limit theorem states that:

If we have $X_1,X_2,\dots$ iid with mean $\mu$ and variance $\sigma^2$ then $$U_n=\frac{1}{\sqrt{n\sigma ^2}} \sum _{i=1}^n (X_i - \mu) \to N(0,1)$$ in distribution.

Law of large numbers states that:

If $X_1,X_2,\dots$ are i.i.d with mean $\mu$ then $$S_n=\frac{1}{n}\sum_{i=1}^n X_i \to \mu$$ a.s.

Returning to the CLT we can see it also shows that (since $\frac{1}{\sqrt n}=\frac{\sqrt n}{\sqrt n \sqrt n}$): $$\sqrt{n}\cdot (\frac{1}{n} \sum _{i=1}^n (X_i - \mu)) \to N(0,\sigma ^2)$$

So you can see that CLT "works" a bit slower which is where the difference lies.

Convergence a.s. is much stronger than convergence in distribution, so you are right - had the scaling been the same it would imply convergence in distribution to a degenerate, that is almost surely constant random variable. In the case above $S_n \to X$ in distribution where $P(X=\mu)=1$

Maybe you can even now use CLT to say something about how fast the LLN convergence is?

-