0
$\begingroup$

I'm not so fit in statistics and I found some controverse answers on the internet so I'm asking here.

I have a set $A$ with 10439 samples. The set is not of unique values and many of the sample values occur more than once. Now I want to calculate the mean and the variance of the set, but there is a limitation. I can't calculate for the whole set at once so I had to split it into $N$ batches. Now lets say I have $N=73$ batches $A_{1}, ... ,A_{73}$ created form the big set $A$. Do those hold?

  1. $mean(A) = \frac{mean(A_{1}) + ... + mean(A_{73})}{N}$
  2. $var(A) = \frac{var(A_{1}) + ... + var(A_{73})}{N}$

If not then how can I achieve my goal using the batches?

$\endgroup$
1
$\begingroup$

Statement 1 and Statement 2 are both untrue.

Statement 1 is not true because the means in the average are not weighted by the size of the set. For example if $A_1 = \{1\}$ and $A_2 = \{2,2,2,2\}$ then $\operatorname{mean}(A_1 \cup A_2) = 1.8$ but $\tfrac{\operatorname{mean}(A_1) + \operatorname{mean}(A_2)}{2} = \frac{1+2}{2} = 1.5$. The correct formula is $$ \operatorname{mean}(A) = \frac{|A_1|\operatorname{mean}(A_1) + |A_2|\operatorname{mean}(A_2) + \cdots + |A_N|\operatorname{mean}(A_N)}{|A_1|+|A_2|+\cdots+|A_N|} $$ where $|A_i|$ gives the number of elements in the set $A$. To see this, expand $$ \operatorname{mean}(A) = \frac{1}{|A|}\sum_{a\in A}a = \frac{1}{|A|}\biggl(\frac{|A_1|}{|A_1|}\sum_{a\in A_1}a + \frac{|A_2|}{|A_2|}\sum_{a\in A_2}a + \cdots + \frac{|A_2|}{|A_2|}\sum_{a\in A_N}a\biggr) $$ and observe that $\tfrac{1}{|A_i|}\sum_{a \in A_i} a$ is exactly $\operatorname{mean}(A_i)$.

Statement 2 is untrue for effectively the same reason. One way to compute the correct $\operatorname{var}(A)$ in batches is to compute $s_i = \sum_{a\in A_i}a^2$ for every batch $i=1,2,\ldots,N$. At this point $$\operatorname{var}(A) = \frac{s_1+s_2+\cdots+s_N}{|A_1| + |A_2| + \cdots + |A_N|} - \operatorname{mean}(A)^2,$$ where $\operatorname{mean}(A)$ was computed as above.

$\endgroup$
  • $\begingroup$ Thank you very much I've calculated the first Statement correctly. I've had to calculate the variance of each batch separately and sequentially for each of the 73 batches with $$var(A_i) = \frac{1}{|A_i|}\cdot\sum_{i=1}^{|A_i|} (a_i - \mu_{A_i})^2$$ Is there a way to add my sequential results to get the whole variance of (A)? $\endgroup$ – TalG Mar 8 at 23:06
  • $\begingroup$ It turns out that $\sigma_{A_i}^2 = \tfrac{s_i}{|A_i|} - \mu_{A_i}^2$ as well, so with $\sigma_{A_i}^2$ already computed you can find $s_i = |A_i|(\sigma_{A_i}^2 + \mu_{A_i}^2)$ and use the formula above. $\endgroup$ – cdipaolo Mar 8 at 23:09
  • 1
    $\begingroup$ Thanks again! The calculation of the second Statement worked with $$var(A)=\frac{|A_1|(\sigma^2_{A_1} + \mu^2_{A_1}) + |A_2|(\sigma^2_{A_2} + \mu^2_{A_2}) + \cdots + |A_N|(\sigma^2_{A_N} + \mu^2_{A_N})}{|A_1| + |A_2| + \cdots + |A_N|}-mean(A)^2$$ $\endgroup$ – TalG Mar 9 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.