Say I have two estimates of the mean of two functions: $$Q^1_{N_1}=\frac{1}{N_1}\sum_{i=1}^{N_1}f^1(X_i), \quad Q^2_{N_2}=\frac{1}{N_2}\sum_{i=1}^{N_2}f^2(X_i),$$ where each sample $X_i$ is identical when $i \leqslant \min(N_1, N_2)$ and is selected from the probability distribution $X$, and I want to calculate the covariance of the two estimators $\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})$.

Lemna 3.2 of this paper attempts to prove that $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})=\frac{\mathrm{Cov}(f^1(X), f^2(X))}{\max(N_1,N_2)}=\frac{\sqrt{\mathrm{Var(f^1(X)) \mathrm{Var}(f^2(X))}}}{\max(N_1,N_2)}\rho_{12},$$ where $\rho_{12}:=\dfrac{\mathrm{Cov}(f^1(X), f^2(X))}{\sqrt{\mathrm{Var}(f^1(X)) \mathrm{Var}(f^2(X))}}$.

I am pretty sure that the important piece there is $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2}) = \frac{\mathrm{Cov}(f^1(X), f^2(X))}{\max(N_1,N_2)}.$$

This has been a hard proof for me to follow, so I am asking this community for help. I would say my biggest source of confusion is where the $\max(N_1, N_2)$ term comes from.

Here is what I have tried so far. My result is simillar to that in the paper, except I have a minimum instead of a maximum. Can anyone help give me insight as to where the maximum operator comes into play?

The covariance of $Q^1_{N_1}$ and $Q^2_{N_2}$ may be calulated as \begin{align*} &\mathrel{\phantom{=}}{} \mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})\\ &=\frac{1}{\min(N_1,N_2)}\sum_{i=1}^{\min(N_1, N_2)}\left(\left(f^1(x_i) - \frac{1}{N_1}\sum_{j=1}^{N_1}f^1(x_j)\right)\left(f^2(x_i) - \frac{1}{N_2}\sum_{j=1}^{N_2}f^2(x_j)\right)\right), \end{align*} where the minimum of $N_1$ and $N_2$ are the number of samples common to both estimators. Only $\min(N_1,N_2)$ samples are common to both estimators, so we must use at most $\min(N_1,N_2)$ samples to estimate the covariance. Assuming the number of samples is large, we can say $$\mathrm{Cov}(Q^1_{N_1}, Q^2_{N_2})= \frac{1}{\min(N_1,N_2)}\mathrm{Cov}(f^1(X), f^2(X))=\frac{\sqrt{\mathrm{Var(f^1(X)) \mathrm{Var}(f^2(X))}}}{\min(N_1,N_2)}\rho_{12}.$$


$\DeclareMathOperator{\cov}{cov}\def\peq{\mathrel{\phantom{=}}{}}$\begin{align*} &\peq \cov(Q_1, Q_2) = E(Q_1 Q_2) - E(Q_1) E(Q_2)\\ &= \frac{1}{N_1 N_2} E\left( \left( \sum_{k = 1}^{N_1} f_1(X_k)\right) \left( \sum_{k = 1}^{N_2} f_2(X_k) \right) \right) - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \Biggl( \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 = k_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) + \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 ≠ k_2}} E(f_1(X_{k_1}) f_2(X_{k_2})) \Biggr)\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} \Biggl( E(f_1(X) f_2(X)) \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 = k_2}} 1 + E(f_1(X)) E(f_2(X)) \sum_{\substack{1 \leqslant k_1 \leqslant N_1\\1 \leqslant k_2 \leqslant N_2\\k_1 ≠ k_2}} 1 \Biggr)\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{1}{N_1 N_2} (\min(N_1, N_2) E(f_1(X) f_2(X)) + (N_1 N_2 - \min(N_1, N_2)) E(f_1(X)) E(f_2(X)))\\ &\peq - E(f_1(X)) E(f_2(X))\\ &= \frac{\min(N_1, N_2)}{N_1 N_2} (E(f_1(X) f_2(X)) - E(f_1(X)) E(f_2(X)))\\ &= \frac{1}{\max(N_1, N_2)} \cov(f_1(X), f_2(X)). \end{align*}

  • $\begingroup$ Thanks. I believe that your proof is correct. It has been very hard for me to follow. These are only true equalities in the limit of $N_1$ and $N_2$ approaching infinity, right? Would you be willing to add some comments to your proof? $\endgroup$ – kilojoules May 25 '18 at 15:02
  • $\begingroup$ @kilojoules No, all these are just finite forms not involving limits. This answer only uses linearity of expectation and $E(XY)=E(X)E(Y)$ for independent $X$ and $Y$. $\endgroup$ – Saad May 25 '18 at 15:07
  • $\begingroup$ Thanks for your comment. After realizing the linear property you pointed out, everything makes sense. $\endgroup$ – kilojoules May 25 '18 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.