# Covariance of two expectation estimators that used different numbers of samples

Say I have two estimates of the mean of two functions: $$Q^1_{N_1}=\frac{1}{N_1}\sum_{i=1}^{N_1}f^1(X_i), \quad Q^2_{N_2}=\frac{1}{N_2}\sum_{i=1}^{N_2}f^2(X_i),$$ where each sample $X_i$ is identical when $i \leqslant \min(N_1, N_2)$ and is selected from the probability distribution $X$, and I want to calculate the covariance of the two estimators $\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})$.

Lemna 3.2 of this paper attempts to prove that $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})=\frac{\mathrm{Cov}(f^1(X), f^2(X))}{\max(N_1,N_2)}=\frac{\sqrt{\mathrm{Var(f^1(X)) \mathrm{Var}(f^2(X))}}}{\max(N_1,N_2)}\rho_{12},$$ where $\rho_{12}:=\dfrac{\mathrm{Cov}(f^1(X), f^2(X))}{\sqrt{\mathrm{Var}(f^1(X)) \mathrm{Var}(f^2(X))}}$.

I am pretty sure that the important piece there is $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2}) = \frac{\mathrm{Cov}(f^1(X), f^2(X))}{\max(N_1,N_2)}.$$

This has been a hard proof for me to follow, so I am asking this community for help. I would say my biggest source of confusion is where the $\max(N_1, N_2)$ term comes from.

Here is what I have tried so far. My result is simillar to that in the paper, except I have a minimum instead of a maximum. Can anyone help give me insight as to where the maximum operator comes into play?

The covariance of $Q^1_{N_1}$ and $Q^2_{N_2}$ may be calulated as \begin{align*} &\mathrel{\phantom{=}}{} \mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})\\ &=\frac{1}{\min(N_1,N_2)}\sum_{i=1}^{\min(N_1, N_2)}\left(\left(f^1(x_i) - \frac{1}{N_1}\sum_{j=1}^{N_1}f^1(x_j)\right)\left(f^2(x_i) - \frac{1}{N_2}\sum_{j=1}^{N_2}f^2(x_j)\right)\right), \end{align*} where the minimum of $N_1$ and $N_2$ are the number of samples common to both estimators. Only $\min(N_1,N_2)$ samples are common to both estimators, so we must use at most $\min(N_1,N_2)$ samples to estimate the covariance. Assuming the number of samples is large, we can say $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})= \frac{1}{\min(N_1,N_2)}\mathrm{Cov}(f^1(X), f^2(X))=\frac{\sqrt{\mathrm{Var(f^1(X)) \mathrm{Var}(f^2(X))}}}{\min(N_1,N_2)}\rho_{12}.$$

$\DeclareMathOperator{\cov}{cov}\def\peq{\mathrel{\phantom{=}}{}}$\begin{align*} &\peq \cov(Q_1, Q_2) = E(Q_1 Q_2) - E(Q_1) E(Q_2)\\ &= \frac{1}{N_1 N_2} E\left( \left( \sum_{k = 1}^{N_1} f_1(X_k)\right) \left( \sum_{k = 1}^{N_2} f_2(X_k) \right) \right) - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \Biggl( \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 = k_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) + \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 ≠ k_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) \Biggr)\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \Biggl( E(f_1(X) f_2(X)) \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 = k_2}} 1 + E(f_1(X)) E(f_2(X)) \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 ≠ k_2}} 1 \Biggr)\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} (\min(N_1, N_2) E(f_1(X) f_2(X)) + (N_1 N_2 - \min(N_1, N_2)) E(f_1(X)) E(f_2(X)))\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{\min(N_1, N_2)}{N_1 N_2} (E(f_1(X) f_2(X)) - E(f_1(X)) E(f_2(X)))\\ &= \frac{1}{\max(N_1, N_2)} \cov(f_1(X), f_2(X)). \end{align*}
• Thanks. I believe that your proof is correct. It has been very hard for me to follow. These are only true equalities in the limit of $N_1$ and $N_2$ approaching infinity, right? Would you be willing to add some comments to your proof? – kilojoules May 25 '18 at 15:02
• @kilojoules No, all these are just finite forms not involving limits. This answer only uses linearity of expectation and $E(XY)=E(X)E(Y)$ for independent $X$ and $Y$. – Saad May 25 '18 at 15:07