Where is this formula coming from?

Question

I am currently reading this article. On page 301 it says:

About a century and a half ago, it was observed by Poisson (1837) (see also Edwards (I960)) that $$\text{Var}(S_n) = n \bar{p} \, (1-\bar{p}) - n s_p^2,$$ where $s_p^2 = \frac{1}{n}\sum_{i=1}^n (p_i-\bar{p})^2$ is the “variance” within $\{p_1, \ldots ,p_n\}$. Therefore, the variance of $S_n$ increases as the set of probabilities $\{p_1, \ldots, p_n\}$ tends to be more and more homogeneous and attains its maximum as they become identical.

Where $S_n$ is the number of successes after $n$ independent trials, the probability of success at the $k$th trial being $p_k$. I am having trouble seeing where this is coming from: can anyone point me in the right direction?

Answer 1

The formula initially assumes that the probability of success is not the same in each trial.

Hence,

$$\text{Var}(S_n) = \sum_{i=1}^np_i(1-p_i)$$

Now, since $\bar p = \frac 1n \sum_{i=1}^np_i$ for the RHS we have that

$$n \bar{p} \, (1-\bar{p}) - n s_p^2 = \left(\sum_{i=1}^np_i\right)\left(1-\bar p\right) - \sum_{i=1}^n (p_i-\bar{p})^2$$

$$=\left(\sum_{i=1}^np_i\right) - \bar p \left(\sum_{i=1}^np_i\right) - \left(\sum_{i=1}^np_i^2\right) +2\bar p\left(\sum_{i=1}^np_i\right) - n\bar p^2$$

Combine 1st and 3d term, $$=\sum_{i=1}^np_i(1-p_i) - n\bar p^2 + 2n\bar p^2 - n\bar p^2 = \sum_{i=1}^np_i(1-p_i) = \text{Var}(S_n)$$

So it is just an algebraic manipulation in order to bring in the surface how $\text{Var}(S_n)$ is affected by how "homogeneous" are the probabilities of success in each trial.