1
$\begingroup$

I have a data set of position measurements of an object. However, the data set is split into subsets. The subsets have equal size. I want to find the mean and variance of the whole data set only with access to the subsets, and the not the whole data set at once. How would I go about doing this? Is it correct that the mean of the means of the subsets is equal to the mean of the whole set? What about the variance?

$\endgroup$
4
  • 1
    $\begingroup$ The answer you accepted is incorrect. Does it make any sense? $\endgroup$ Mar 30, 2020 at 21:26
  • $\begingroup$ Yes, I have just realized when I got results that did not make sense. I see your answer below is what I am looking for, but I believe I am concerned with the case that k is very large. In particular, my data set is split into a large number of subsets, or groups, and I believe the expression I am looking for seems to be the pooled variance expression in your answer. $\endgroup$
    – Eoin S
    Mar 31, 2020 at 14:34
  • 1
    $\begingroup$ Sorry I misread your post as saying it is split in two subsets. I don't know of a approximation when $k$ is large, but the general expression of pooled variance gives you the exact answer. $\endgroup$ Mar 31, 2020 at 14:45
  • $\begingroup$ In the context of estimation, one can refer to math.stackexchange.com/a/2971563/321264 where the sample sizes are subtracted by $1$ to get unbiased estimates. This is for $k=2$. $\endgroup$ Apr 30, 2020 at 15:01

2 Answers 2

1
$\begingroup$

Suppose you have $k$ groups of observations on a variable $x$ (say) where the $i$th group consists of $n_i$ observations, $i=1,\ldots,k$. Let the $j$th observation in the $i$th group be $x_{ij}$ for $i=1,\ldots,k$ and $j=1,\ldots,n_i$.

The $i$th group mean is defined as $$\overline{x_i}=\frac1{n_i}\sum_{j=1}^{n_i} x_{ij}\quad,\,i=1,\ldots,k $$

Then the pooled mean or combined mean is given by $$\overline{\overline x}=\frac{\sum_{i=1}^k n_i\overline x_i }{\sum_{i=1}^k n_i} $$

This is a weighted average with weights being the number of observations in the $i$th group.

The $i$th group variance is defined as

$$s_i^2=\frac1{n_i}\sum_{j=1}^{n_i}\left(x_{ij}-\overline{\overline x}\right)^2\quad,\,i=1,\ldots,k $$

And the pooled variance based on all observations from all groups is given by

$$s^2=\frac{\sum_{i=1}^k\sum_{j=1}^{n_i}\left(x_{ij}-\overline{\overline x}\right)^2}{\sum_{i=1}^k n_i}=\frac{\sum_{i=1}^k n_is_i^2}{\sum_{i=1}^k n_i}+\frac{\sum_{i=1}^k n_i\left(\overline x_i-\overline{\overline x}\right)^2}{\sum_{i=1}^k n_i}$$

You are concerned with the case $n_1=n_2=\cdots=n_k$.

$\endgroup$
-1
$\begingroup$

If ${\cal D} = \{ {\cal D}_1, {\cal D}_2, \ldots, {\cal D}_n \}$, each of the same number of points, then the mean of ${\cal D}$ is simply the mean (expectation) of the individual means:

$${\cal E}[{\cal D}] = \frac{1}{n} \sum\limits_{i=1}^n {\cal E}[{\cal D}_i]$$

The variance sum law states is just the sum of the individual variances (for non-zero variances):

$${\rm Var}[{\cal D}] = \sum\limits_{i=1}^n {\rm Var}[{\cal D}_i]$$

$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .