2
$\begingroup$

Consider the Central Limit Theorem: Suppose that $X_1, X_2, \ldots, X_n$ are independent and identically distributed random variables with mean $\mu$ and variance $\sigma^2 < \infty$. Let $\bar{X} = (X_1 + X_2 + \cdots + X_n)/n$ be the sample mean. Then, as $n \rightarrow \infty$, the distribution of $\sqrt{n}(\bar{X} - \mu)/\sigma$ converges in distribution to a standard normal distribution, i.e.,

$$\lim_{n \rightarrow \infty} \mathbb{P} \left( \sqrt{n} \frac{\bar{X} - \mu}{\sigma} \leq x \right) = \Phi(x)$$

where $\Phi(x)$ is the cumulative distribution function of the standard normal distribution.

In introductory math classes, we are often told that the Central Limit Theorem is only applicable for when $X_1, X_2, \ldots, X_n$ are independent and identically distributed (iid) - yet the importance for this condition is not really explained. I am trying to understand why this iid condition is so important.

While trying to learn more the importance of this iid condition, I came upon variants of the Central Limit Theorem in which this condition is partly relaxed (e.g. https://en.wikipedia.org/wiki/Lindeberg%27s_condition , https://en.wikipedia.org/wiki/Central_limit_theorem#Lyapunov_CLT) - but I still could not find an explanation as to why the results of the Classic Central Limit Theorem might not be applicable for the non-iid case.

As an example - is it possible to construct an example in which $X_1, X_2, \ldots, X_n$ are created such that they are deliberately non-iid (e.g. auto-correlation), and then demonstrate that in this example, $\sqrt{n}(\bar{X} - \mu)/\sigma$ WILL NOT converge to a Standard Normal Distribution?

Thanks!

$\endgroup$
7
  • $\begingroup$ Have you looked at how the CLT is proven? It becomes pretty clear then where the i.i.d.-part comes in $\endgroup$
    – Lorago
    Apr 1, 2023 at 17:03
  • $\begingroup$ @ Logaro: thank you for your reply! I looked at a proof which involves using Characteristic Functions ... but it was not clear to me where the iid part comes in. maybe you could please walk me through this part if you have time? thank you so much $\endgroup$ Apr 1, 2023 at 17:05
  • $\begingroup$ I don't really have the time right now to write a full answer, but that is the proof I'm thinking of. Basically the fact that the random variables are identically distributed is what guarantees they have the same characteristic function, and the fact that they are independent is what allows you to write the characteristic function of their sum as the product of their characteristic functions. This gets you to a from where you can to a Taylor expansion, take the limit, and get the exponential function you want. However without them being i.i.d. you will not get it to that form $\endgroup$
    – Lorago
    Apr 1, 2023 at 17:09
  • $\begingroup$ @Lorago If they are independent but not identically distributed, you can still take that approach, and in some cases you can get the desired limit but not in others, and it is generally more complicated $\endgroup$
    – Henry
    Apr 1, 2023 at 17:36
  • $\begingroup$ @Henry yeah my point is more so that the standard approach breaks down if you remove these assumptions, and that the result in general doesn't hold without them, but of course you can still get special cases where it does work $\endgroup$
    – Lorago
    Apr 1, 2023 at 19:01

3 Answers 3

3
$\begingroup$

A trivial example for not-independent variables is when $X_1$ is Uniform on $[-1,1]$ and all $X_2, X_3. \cdots$ coincide with $X_1$ ($X_k=X_1)$

An example for independent but not identically distributed (from here). Let $Z_k$ be iid Uniform on $[-1,1]$ and let $X_k = a^k Z_k $ for some $0<a<1$. In this case, $X_k$ are independent, uniform on $[-a^k,a^k]$, so that $E[X_k]=0$ and $\sigma_k^2= \frac13 a^{-2k}$. Because $\sum X_k$ is limited to the range $(−1/(1−a),1/(1-a))$, the sum cannot converge to a Gaussian distribution.

Another example here

$\endgroup$
1
  • $\begingroup$ @ leonbloy: thank you so much for your answer! if you have time, can you please expand on these two examples you provided? e.g. show some intermediate steps ? thank you so much ! $\endgroup$ Apr 10, 2023 at 16:37
1
$\begingroup$

Great question. With respect to an example with autocorrelation that $\sqrt{n}(\bar{X}-\mu)$ does not converge to a standard normal, consider the following MA(1) model and its autocovariances \begin{align} X_t&=\theta Z_{t-1}+Z_t \;\;\; Z_t \sim N(0,\sigma_Z^2)\\ \sigma_X^2&=\gamma(0)=(\theta^2+1)\sigma_Z^2\\ \gamma(1)&=\theta\sigma_Z^2\\ \gamma(k)&=0 \; \forall \; k>1. \end{align}

The non-zero autocorrelation means that in the limit the variance of $\sqrt{n}(\bar{X}-\mu)$ is no longer $\sigma^2_X=(\theta^2+1)\sigma^2_Z$ but is instead $\gamma(0)+2\sum_{j=1}^\infty \gamma(j)= \sigma^2_Z(\theta+1)^2$.

This concept is referred to as long-run variance, and is part of what makes analyzing time-series so difficult. I will also add that there have been extensions to CLT which can handle certain forms of time dependence (see here for example).

$\endgroup$
2
  • $\begingroup$ @ David Veitch: Thank you so much for your answer! If you have time, can you please expand on this? e.g. add some more intermediate steps? thanks! $\endgroup$ Apr 10, 2023 at 16:38
  • $\begingroup$ Hi @stats_noob see stats.stackexchange.com/questions/153444/… for some of the intermediate steps. $\endgroup$ Apr 11, 2023 at 2:49
1
$\begingroup$

Perhaps you can try a simulation-based approach?

  • Using Python you can generate random sequences of data such that each new number is related via some autocorrelation function.
  • Then, you evaluate (and record) the sample mean of this random sequence.
  • Then, you repeat these first two steps many times and plot a histogram of the distribution for these sample means
  • Finally, by studying the EDF (Empirical Distribution Function) of this sample mean (e.g. with Kolmogorov-Smirnov), you can compare this EDF with the expected EDF of theoretical Standard Normal Distribution
  • If these two EDFs are statistically different from one another, you can conclude that empirically, the Central Limit Theorem does not always apply in the absence of the iid condition
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .