I am trying to derive the expectation $\mathbb E$ of the sample covariance $$\overline{cov}_{X,Y} := \frac{1}{n-1}\cdot \sum_{i=1}^n (X_i-\overline X)(Y_i - \overline Y)$$ where $\overline X = \frac1n \sum_{i=1}^n X_i$ and $\overline Y = \frac1n \sum_{i=1}^n Y_i$ and $X_1,\ldots, X_n$ are some iid random variables, and so are $Y_1,\ldots,Y_n$.

I start as follows:

$\mathbb E[\overline{cov}_{X,Y}] = \frac{1}{n-1}\cdot \mathbb E[\sum_{i=1}^n(X_i Y_i - X_i\overline Y - Y_i\overline X + \overline X\overline Y)]$
$\quad\quad\quad\quad = \frac{1}{n-1}\cdot (n\cdot \mathbb E[X_1 Y_1] - n\mathbb E[\overline X\overline Y] - n\mathbb E[\overline Y\overline X] + n\mathbb E[\overline X\overline Y])$
$\quad\quad\quad\quad = \frac{n}{n-1}\cdot (\mathbb E[X_1 Y_1] - \mathbb E[\overline X\overline Y])$

since $n\overline X = \sum_{i=1}^n X_i$ and $n\overline Y = \sum_{i=1}^n Y_i$.

Then I check

$E[\overline X\overline Y] = \frac{1}{n^2}\cdot \mathbb E[\sum_{i=1}^n\sum_{j=1}^nX_i Y_j]$
$\quad\quad\quad = \frac{1}{n^2}\cdot \sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_i Y_j] \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad\ (1)$
$\quad\quad\quad = \frac{1}{n^2}\cdot \sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_1 Y_1]\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad (2)$
$\quad\quad\quad = \mathbb E [X_1 Y_1]$

But this gives $\mathbb E[\overline{cov}_{X,Y}] = 0$, which can't be right. I feel like I did a mistake by going from line (1) to (2), but I cannot figure out why. I mean, e.g. we should have $\mathbb E[X_1 Y_j] = \mathbb E[X_1 Y_1]$ because all $Y_j$ are iid.

However, ideally I'd want $$\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_i Y_j] = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1, i\ne j}^n \mathbb E [X_i]\mathbb E[Y_j] +\frac{1}{n^2}\sum_{i=1}^n \mathbb E [X_i Y_i]$$ but how?

  • $\begingroup$ Are you sure that your first formula is the covariance? Maybe you should check it. $\endgroup$ – Karl Mar 7 '15 at 18:50
  • $\begingroup$ @Karl If you are referring to the sample covariance formula, then I believe it is correct, see e.g. en.wikipedia.org/wiki/… $\endgroup$ – Phil-ZXX Mar 7 '15 at 20:53

What exactly are the dependence relations between the $X_i$ and the $Y_j$?

If $Y_1$ and $Y_2$ are iid, then you do not necessarily have that $X_1Y_1$ and $X_1Y_2$ have the same distribution. For instance, if $X_1$ is independent of $Y_2$, but dependent on $Y_1$ (which happens e.g. if $X_1=Y_1$, in which case $X_1Y_1=Y_1^2$), the joint distributions of $(X_1,Y_1)$ and $(X_1,Y_2)$ are different, which makes the products $X_1Y_1$ and $X_1Y_2$ behave differently.

I believe that in your case, $X_i$ is independent of $Y_j$ for all $j\neq i$, but that $X_i$ and $Y_i$ are dependent, and that $(X_1,Y_1)$ and $(X_i,Y_i)$ have the same joint distribution for all $i$. This makes sense in regards to the covariance, because you sample your variables pairwise, but otherwise independently, i.e. you get data points $(X_1,Y_1), (X_2,Y_2), \ldots$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.