If a bunch of random variables $X_i$ are independently and identically distributed with an exponential distribution, their sum apparently follows a Gamma distribution.

But doesn't the central limit theorem imply that (for $X_i$ of any distribution with mean zero and variance $\sigma^2$), the sum $\sum_{i=1}^n X_i$ will become approximately normally distributed $~N(0,n\sigma^2)$ for large enough $n$ ?

Obviously I am missing something basic, but what's going on? How can the sum of i.i.d. exponential random variables have a Gamma distribution, but also be converging to normality?


There are several confusions here (I was also very confused when I started learning about that topic :-).)

  • Exponential random variables have a non zero mean (and are positive). The quantity you should be looking at, which asymptotically converges in distribution to a normal variable is $$\sqrt{n} \left( \frac{\sum_{i = 1}^n X_i}{n} - \mu \right)$$ The $\sqrt{n}$ was essential here, otherwise the distribution of the average will converge to a point mass at $\mu$. That quantity will converge to $N(0,\sigma^2)$. Both $\mu$ and $\sigma$ will be determined by the parameter of the exponential distribution.
  • The central limit theorem is asymptotic. The quantity $\sqrt{n} \left( \frac{\sum_{i = 1}^n X_i}{n} - \mu \right)$ will have a distribution. Let's call it $F_n$. (it is essential to remember that it depends on $n$). $F_n$ in general is not a normal distribution $N(0,\sigma^2)$. The central limit theorem tells us that that distribution gets in a certain sense closer and closer to $N(0,\sigma^2)$ as $n \to \infty$.
  • $\begingroup$ Yes I was mistaken in talking about a mean of zero, which is impossible for exponentials (actually I was originally motivated to ask this question because I wanted to sum Laplaces, but thinking about exponentials was the first step). But I'm confused by the $\sqrt(n)$ being essential there. That expression is giving a distribution for the sample average. Doesn't the CLT also give you an approximate distribution for the sample sum? Is it wrong to say that the sum of random variables with variance $\sigma^2$ will converge to a normal distribution with variance $n \sigma^2$? $\endgroup$ – s4027340 Dec 17 '12 at 2:03
  • 1
    $\begingroup$ @s4027340 The problem with saying that the sum of random variables will converge to a Normal distribution with variance $n\sigma^2$ as $n \rightarrow \infty$ is that $n\sigma^2 \rightarrow \infty$ too, and it doesn't help much to talk about a Normal distribution with infinite variance! It is absolutely correct to say that for any (large-ish) finite $n$, the distribution of the sum is approximately Normal with variance $n\sigma^2$, like you said--it just doesn't make much sense in the limit. $\endgroup$ – Jonathan Christensen Dec 17 '12 at 2:08
  • $\begingroup$ Ah, of course! That makes sense. $\endgroup$ – s4027340 Dec 17 '12 at 2:09

Good question! The Gamma distribution itself converges to a Normal distribution--exactly the Normal distribution that the Central Limit Theorem says the sum of the Exponential random variables converges to.

Of course, for any finite $n$ the distribution of the sum of Exponentials (the Gamma distribution) is not a normal distribution, since it's bounded below by zero, but as $n$ gets larger it gets closer and closer to a Normal distribution.

Here's a page that discusses the approximation, with some graphs that show how the Gamma distribution looks like a Normal when the first parameter (= the number of Exponential random variables you are summing up) gets large.

  • $\begingroup$ Is the difference between gamma and normal basically due to the negative part on the normal's left tail? $\endgroup$ – s4027340 Dec 17 '12 at 1:59
  • 1
    $\begingroup$ The shape of the Gamma will never be exactly a Normal distribution, even if you ignore the left-tail problem. But the CDF will get closer and closer to a Normal CDF. $\endgroup$ – Jonathan Christensen Dec 17 '12 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.