Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Let $X_1, X_2,\dotsc$ be independent and identically distributed with mean $E[X]$ and variance $VAR[X]$. Let $N$ be a non-negative integer-valued random variable independent of the $X_i$'s. Show $$ VAR\left[ \sum_{i=1}^N X_i \right] = E[N]VAR[X]+(E[X])^2VAR[N] $$ I've tried expanding this in a number of different ways but I can't quite seem to get it to work out. I don't really understand how to condition on a random variable like this. Any help would be greatly appreciated.

share|improve this question
2  
Hint: use $ E (E ( Z | N )) = E (Z)$ with $Z = \sum X_i$. Hint 2: you could start with the special case $E(X)=0$, then attack the full problem. –  leonbloy Oct 3 '11 at 16:49
    
Thanks. I knew that formula had to come in somewhere but I was having a hard time setting it up. –  chris Oct 3 '11 at 16:58

4 Answers 4

up vote 7 down vote accepted

The law of total variance says $$ \operatorname{var}(Y) = E(\operatorname{var}(Y\mid X)) + \operatorname{var}(E(Y\mid X)). $$ So $$ \begin{align} \operatorname{var}\left(\sum_{i=1}^N X_i\right) & = E\left(\operatorname{var}\left(\sum_{i=1}^N X_i \mid N\right)\right) + \operatorname{var}\left(E\left(\sum_{i=1}^N X_i \mid N\right)\right) \\ \\ \\ & = E(N\operatorname{var}(X)) + \operatorname{var}(NE(X)) \\ \\ & = \operatorname{var}(X)E(N) + (E(X))^2\operatorname{var}(N). \end{align} $$

share|improve this answer
    
Heck. I submit my answer and see that about everything I just wrote is covered in your answer and your reference. –  robjohn Oct 3 '11 at 20:09
    
At least I learned that the "bayesian decomposition" that I proved for an article on sci.math is called the law of total variance. –  robjohn Oct 3 '11 at 20:15
1  
A catchy way of remembering the law of total variance is that the "total variance equals the mean of the conditional variance plus the variance of the conditional mean." –  Dilip Sarwate Oct 3 '11 at 21:40
    
@Michael: I feel a bit better about my answer since I just saw that you suggested to Sasha that someone should post a proof of the law of total variance. –  robjohn Oct 3 '11 at 23:54
    
@Dilip: alongside with "the variance equals the mean of the squares minus the square of the mean." –  robjohn Oct 3 '11 at 23:59

Performing repeated integration yields $$ \begin{align} \operatorname{E}[X] &=\operatorname{E}_Y[\operatorname{E}_X[X|Y]]\tag{1} \end{align} $$ Applying $(1)$ to $X^2$ and using the fact that $\operatorname{Var}[X]=\operatorname{E}\left[X^2\right]-\operatorname{E}[X]^2$, we get $$ \begin{align} \operatorname{E}\left[X^2\right] &=\operatorname{E}_Y\left[\operatorname{E}_X\left[X^2|Y\right]\right]\\ &=\operatorname{E}_Y\left[\operatorname{Var}_X[X|Y]\right]+\operatorname{E}_Y\left[\operatorname{E}_X[X|Y]^2\right]\tag{2} \end{align} $$ Applying $(1)$ and $(2)$, we get $$ \begin{align} \operatorname{Var}[X] &=\operatorname{E}\left[X^2\right]-\operatorname{E}[X]^2\\ &=\operatorname{E}_Y\left[\operatorname{Var}_X[X|Y]\right]+\operatorname{E}_Y\left[\operatorname{E}_X[X|Y]^2\right]-\operatorname{E}_Y[\operatorname{E}_X[X|Y]]^2\\ &=\operatorname{E}_Y\left[\operatorname{Var}_X[X|Y]\right]+\operatorname{Var}_Y[\operatorname{E}_X[X|Y]]\tag{3} \end{align} $$ Now apply $(3)$ to the problem: $$ \begin{align} \operatorname{Var}\left[\sum_{i=1}^NX_i\right] &=\operatorname{E}_N\left[\left.\operatorname{Var}_X\left[\sum_{i=1}^NX_i\right]\right|N\right]+\operatorname{Var}_N\left[\left.\operatorname{E}_X\left[\sum_{i=1}^NX_i\right]\right|N\right]\\ &=\operatorname{E}_N[N\operatorname{Var}[X]]+\operatorname{Var}_N[N\operatorname{E}[X]]\\ &=\operatorname{E}[N]\operatorname{Var}[X]+\operatorname{Var}[N]\operatorname{E}[X]^2\tag{4} \end{align} $$

share|improve this answer
    
As Michael Hardy points out, $(3)$ is called the Law of Total Variance. –  robjohn Oct 4 '11 at 17:18

Another way to do this: let $Y_i = 1$ if $N \ge i$, $0$ otherwise. Then your sum is $$S = \sum_{i=1}^N X_i = \sum_{i=1}^\infty Y_i X_i$$ (I won't worry about convergence of infinite sums: if you wish you can use a truncated version of $N$ and then take limits). So $$\text{var}(S) = \sum_{i=1}^\infty \text{var}(Y_i X_i) + 2 \sum_{i=1}^\infty \sum_{j=1}^{i-1} \text{cov}(Y_i X_i, Y_j X_j)$$ Now $\text{var}(Y_i X_i) = E[Y_i^2 X_i^2] - E[Y_i X_i]^2 = E[Y_i] \text{var}(X) + \text{var}(Y_i) E[X]^2$, while for $j < i$, $\text{cov}(Y_i X_i, Y_j X_j) = E[Y_i Y_j X_i X_j] - E[Y_i X_i] E[Y_j X_j] = E[Y_i] (1 - E[Y_j]) E[X]^2$, so that $$ \text{var}(S) = \sum_{i=1}^\infty E[Y_i] \text{var}(X) + \sum_{i=1}^\infty \text{var}(Y_i) E[X]^2 + 2 \sum_{i=1}^\infty \sum_{j=1}^{i-1} E[Y_i](1 - E[Y_j]) E[X]^2$$ Now doing the same calculation with $X_i$ replaced by 1 (since $N = \sum_{i=1}^\infty Y_i$), $$ \text{var}(N) = \sum_{i=1}^\infty \text{var}(Y_i) + 2 \sum_{i=1}^\infty \sum_{j=1}^{i-1} E[Y_i](1 - E[Y_j])$$ so that $$ \text{var}(S) = \sum_{i=1}^\infty E[Y_i] \text{var}(X) + \text{var}(N) E[X]^2 = E[N] \text{var}(X) + \text{var}(N) E[X]^2 $$

share|improve this answer

Let $Y = \sum_{i=1}^N X_i$. Notice, that the characteristic function of $Y$ can be expressed as composition of characteristic functions of $X$ $\phi(t)$ and the probability generating function of $N$, $g(s)$: $$ \psi(t) = \mathbb{E}( \exp( i Y t) ) = \mathbb{E}\left( \mathbb{E}( \exp( i Y t) \vert N) \right) = \mathbb{E}\left( \phi(t)^N \right) = \sum_{k=0}^\infty \phi(t)^k \mathbb{P}(N=k) = g(\phi(t)). $$ Notice that the variance of $Y$ is related to its moments $\mathrm{Var}(Y) = m_2(Y) - m_1(Y)^2$, and that $m_r(Y) = (-i)^r \psi^{(r)}(0)$, so that $\mathrm{Var}(Y) = -\psi^{\prime\prime}(0)+\left( \psi^\prime(0)\right)^2$. Using $\psi = g \circ \phi$:

$$ \psi^{\prime}(0)= g^\prime(1) \times \phi^\prime(0) = i \mathbb{E}(N) \mathbb{E}(X) $$ and

$$ \psi^{\prime\prime}(0)= g^{\prime\prime}(1) \phi^\prime(0)^2 + g^\prime(1) \phi^{\prime\prime}(0) = -\left( \mathbb{E}(X)^2 \cdot \mathbb{E}(N(N-1)) + \mathbb{E}(N) \cdot \mathbb{E}(X^2) \right) $$ Combining these together will yield the result you seek to establish. $$ \begin{eqnarray} \mathrm{Var}(Y) &=& \mathbb{E}(X)^2 \cdot \left( \mathbb{E}(N^2) -\mathbb{E}(N) \right) + \mathbb{E}(N) \cdot \left( \mathrm{Var}(X) + \mathbb{E}(X)^2 \right) - \mathbb{E}(N)^2 \cdot \mathbb{E}(X)^2 \\ &=& \mathbb{E}(X)^2 \cdot \left( \mathrm{Var}(N) + \mathbb{E}(N)^2 -\mathbb{E}(N) \right) + \mathbb{E}(N) \cdot \left( \mathrm{Var}(X) + \mathbb{E}(X)^2 \right) - \mathbb{E}(N)^2 \cdot \mathbb{E}(X)^2 \\ &=& \mathbb{E}(X)^2 \cdot \mathrm{Var}(N) + \mathbb{E}(N) \cdot \mathrm{Var}(X) \end{eqnarray} $$

Since I used $g^\prime(1) = \mathbb{E}(N)$ and $g^{\prime\prime}(1) = \mathbb{E}(N(N-1))$ I should note that they follow from the definition of the probability generating function $g(s) = \sum_{k=0}^\infty s^k \mathbb{P}(N=k)$. Indeed $g^\prime(1) = \sum_{k=0}^\infty k \mathbb{P}(N=k) = \mathbb{E}(N)$, and $g^{\prime\prime}(1) = \sum_{k=0}^\infty k (k-1) \mathbb{P}(N=k) = \mathbb{E}(N (N-1))$.

share|improve this answer
    
Certainly a more complicated answer than one gets by routine application of the law of total variance. (But maybe one should also post a proof of that?) –  Michael Hardy Oct 3 '11 at 18:27
    
@MichaelHardy Yes, the machinery is heavy, and a leonboy and yourself gave much neater solutions (+1). I was not aware of the law of total variance, so I'll add it to my bag of tools. –  Sasha Oct 3 '11 at 19:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.