4
$\begingroup$

In class, we are always told that for the Central Limit Theorem to be applicable, observations have to be IID (Independent and Identically Distributed). However, we are not always told why this IID condition is so important for the Central Limit Theorem.

This being said, I am trying to create an example where the IID condition is not met and thus show myself why it is required.

Part 1: The first thing that comes to mind is an Autoregressive Process as by definition AR Processes are said not to be IID. For instance, suppose we have an AR(1) Process:

$$y_t = \phi y_{t-1} + \epsilon_t$$

Based on this AR(1) process, I know the following:

  • $E(\epsilon_t) = 0$
  • $E(\epsilon_t^2) = \sigma^2$
  • $E(\epsilon_t\epsilon_s) = 0$
  • $Var(y_t) = \frac{\sigma^2}{1-\phi^2}$
  • $E(y_t) = \phi E(y_{t-1}) = 0$

Part 2: As for the Central Limit Theorem, I know that in when $n$ is large, any Random Variable behaves as:

$$\frac{\bar{x} - E(X)}{\sqrt{\frac{Var(X)}{n}}} \approx N(0,1)$$

Part 3: Putting this all together, I would now show that the above AR(1) Process DOES NOT converge to a Standard Normal Distribution:

$$\frac{Y_t - E(Y_t)}{\sqrt{\frac{Var(Y_t)}{n}}} = \frac{Y_t - E(Y_t)}{\sqrt{\frac{\sigma^2}{1-\phi^2}\frac{1}{n}}} \not\approx N(0,1) $$

However, I am not sure if I am doing this correctly for the AR(1) Process and have in fact shown that in the absence of IID, the Central Limit Theorem is not necessarily valid.

In general, can someone please show me an example where the IID condition is not met and as a result the Central Limit Theorem does not apply?

Thanks!

Note: I am aware that there are versions of the Central Limit Theorem that do not require the IID Condition (e.g. https://en.wikipedia.org/wiki/Central_limit_theorem#Lyapunov_CLT, https://en.wikipedia.org/wiki/Lindeberg%27s_condition) - however, I am specifically interested in constructing an example that shows why the Classic Central Limit Theorem requires the IID condition.

$\endgroup$
8
  • 4
    $\begingroup$ There are a lot of central limit theorems. The fact that the classical Lindeberg–Lévy CLT requires the random variables to be IID doesn't mean that there aren't others that don't have that requirement. If you're just looking for an example where the CLT doesn't follow, let $X_0$ be some fixed random variable with finite mean and variance, and then let $X_i = X_0$ for every $i$. (I hope you can use this comment as a way to refine your question, because I doubt you intend for such a mundane answer) $\endgroup$ Apr 23, 2023 at 3:56
  • $\begingroup$ @ Brian Moehring: thank you so much for your reply! I have added a note to my question. thanks! $\endgroup$ Apr 23, 2023 at 4:22
  • $\begingroup$ The note doesn't really make sense. "constructing an example that shows why the Classic Central Limit Theorem requires the IID condition" So what exactly are you looking for? $\endgroup$
    – Andrew
    Apr 23, 2023 at 4:29
  • 3
    $\begingroup$ If you want an example, beyond what has already been suggested, then you need to look into the literature for central limit theorems, and violate the hypotheses. The IID condition is not necessary, it's just the simplest way to get a central limit theorem. A good place to start might be Wikipedia or Billingsley's Probability and Measure $\endgroup$
    – Andrew
    Apr 23, 2023 at 4:38
  • 1
    $\begingroup$ As already commented above, "the classical CLT" you mentioned actually does not require iid. It is just the simplest case where CLT holds. Refine your question accordingly, if you are still not satisfied with examples provided above. $\endgroup$
    – I H
    May 9, 2023 at 4:49

2 Answers 2

4
$\begingroup$

Lets say Prof Scatterbrain is trying to demonstrate to his large class of 100 students the truth of the central limit theorem.

This theorem states roughly that "If you have a population with mean μ and standard deviation σ and take sufficiently large random samples from the population, then the distribution of the sample means will be approximately normally distributed."

He gets his hands on the results on the 2023 Boston Marathon sorted by bib number going from 1 to 20,000. (Note: I downloaded the actual results for authenticity).

He hands his secretary these results and asks her to type them up. He wants her to take the first 200 finishing times and put them on a single page. Type the next 200 on another page and so on until he has 200 independent finishing times for each student.

The secretary gets bored after 2 pages and just photocopies them 50 times so that she has 100 sheets of paper which she gives to the professor the next morning. The professor hands them out to the class and asks that each student calculate the mean of the results they have and to bring their answer in the next day.

The next day the professor compiles a histogram of the means supplied by the students, which he expects will look like a normal distribution. He is expecting something like this (This is actually from the real results and looks roughly normally distributed): What Professor Scatterbrain expects

But this is what he actually gets (Again, real, but only two different sets. It does not look normally distributed). What Professor Scatterbrain actually gets

The Professor quickly ends the class amid raucous laughter.

And there you have an example of when the data is not IID (specifically not independent) and therefore the Central Limit Theorem does not apply.

$\endgroup$
3
  • $\begingroup$ @ MichaelMcL1960: Thank you so much for your answer! I really liked the pictures! :) $\endgroup$ May 17, 2023 at 4:49
  • $\begingroup$ You're welcome. It was fun thinking up the scenario. $\endgroup$ May 17, 2023 at 5:28
  • $\begingroup$ Also, what do you think of my original example? Does my original example (i.e. the AR process) effectively prove that the Classic Central Limit Theorem requires the IID condition? $\endgroup$ May 17, 2023 at 5:51
3
+50
$\begingroup$

A more mundane (and probably less entertaining) answer than the escapades of Prof Scatterbrain.

Independent but not identical. Suppose $X_k, k \geq 2$ is given by

$$ X_k = \begin{cases} -1 & \text{with probability $\frac{1}{1+k^2}$} \\ 0 & \text{with probability $1-\frac{1}{k^2}$} \\ k^2 & \text{with probability $\frac{1}{k^2+k^4}$} \end{cases} $$

Then each $X_k$ has mean $0$ and variance $1$, but

\begin{align} \lim_{n \to \infty} P\left(\sum_{k=2}^n X_k = 0 \right) & \geq \lim_{n \to \infty} \prod_{k=2}^n P(X_k = 0) \\ & = \lim_{n \to \infty} \prod_{k=2}^n \left( 1-\frac{1}{k^2} \right) \\ & \geq 1 - \lim_{n \to \infty} \sum_{k=2}^n \frac{1}{k^2} \\ & = 2 - \frac{\pi^2}{6} > 0 \end{align}

so the limiting sum

$$ \lim_{n \to \infty} \frac{1}{\sqrt{n}} \sum_{k=2}^n X_k $$

has an impulse at $0$ and is not normally distributed. (Having different distributions allows the probability of each successive random variable equaling $0$ to approach $1$, such that the probability that the partial sums never leave $0$ is strictly positive.)

Identical but not independent. Suppose $X_1$ is given by

$$ X_1 = \begin{cases} -1 & \text{with probability $1/2$} \\ 1 & \text{with probability $1/2$} \end{cases} $$

and $X_k, k \geq 2$ is given by

$$ X_k = \begin{cases} X_{k-1} & \text{if $k$ is a perfect square} \\ -X_{k-1} & \text{otherwise} \end{cases} $$

Then each $X_k$ has mean $0$ and variance $1$, but the limiting sum

$$ \lim_{n \to \infty} \frac{1}{\sqrt{n}} \sum_{k=1}^n X_k $$

has impulses at $-1$ and $1$ (probability $1/2$ each) and is not normally distributed. (Having dependent distributions allows us to force the partial sums into two distinctly different but deterministic directions.)

$\endgroup$
5
  • $\begingroup$ @ Brian Tung: Thank you so much for your answer! In both examples that you listed here (Independent But Not Identical, Identical But Not Independent), do the limiting sums have a distribution at all? Thank you so much! $\endgroup$ May 17, 2023 at 4:49
  • $\begingroup$ Is it possible to somehow plot the results using Python and show how the limiting sums would look like and then see that they don't even look normal? $\endgroup$ May 17, 2023 at 4:51
  • $\begingroup$ Also, what do you think of my original example? Does my original example (i.e. the AR process) effectively prove that the Classic Central Limit Theorem requires the IID condition? $\endgroup$ May 17, 2023 at 5:51
  • $\begingroup$ @stats_noob: I'll have to look over the example you give. As far as my examples go, the second one has a very easy limiting distribution: $\lim_{n \to \infty} \frac{1}{\sqrt{n}} \sum_{k=1}^n X_k$ is either $-1$ or $1$ with probability $1/2$ each. (It equals whatever $X_1$ equals.) The first one is more complicated; I imagine I'd have to write code for it. $\endgroup$
    – Brian Tung
    May 17, 2023 at 6:31
  • $\begingroup$ @stats_noob: Having given it some more thought, I think my first example is not going to have a visually interesting limiting distribution. It'll be an impulse at the origin of maybe a bit more than $1/2$, plus an extremely long and invisible (but non-zero) tail. It will have unit variance, so the impulse at the origin is not indicative of a degenerate normal distribution. $\endgroup$
    – Brian Tung
    May 17, 2023 at 18:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .