Interpreting $\mathrm{Var}(\overline{X}) = \mathrm{Cov}(X_i, \overline{X})$

Question

Let $X_i$ be pairwise-uncorrelated random variables, $\forall\,i \in \mathbf{n} \equiv \{0,\dots,n-1\}$, with identical expectation value $\mathbb{E}(X_i)=\mu$, and identical variance $\mathrm{Var}(X_i)=\sigma^2$. Also, let $\overline{X}$ be their average, $\frac{1}{n}\sum_{i\in \mathbf{n}} X_i$.

Then, as shown below,

$$ \mathrm{Var}(\overline{X}) = \mathrm{Cov}(X_i, \overline{X}) \tag{1} $$

What seems uncanny to me about this equality is that each side "arrives" at its value, $\sigma^2/n$, by what look to me like entirely unrelated paths!

On the LHS, this value comes from

$$\mathrm{Var}(\overline{X})=\mathrm{Var}\left( \frac{1}{n}\sum_{i\in \mathbf{n}} X_i \right)=\frac{1}{n^2}\sum_{i\in \mathbf{n}}\mathrm{Var}(X_i)=\frac{1}{n^2}\sum_{i\in \mathbf{n}}\sigma^2=\frac{\sigma^2}{n}$$

IOW, each summand contributes equally to the final value.

On the RHS, it comes from

$$ \begin{align} \mathrm{Cov}(X_i, \overline{X})=\mathrm{Cov}\left(X_i, \frac{1}{n}\sum_{j\in \mathbf{n}} X_j\right) & = \frac{1}{n} \sum_{j\in \mathbf{n}} \mathrm{Cov}(X_i, X_j) \\ & = \frac{1}{n}\left( \mathrm{Var}(X_i) + \sum_{j\in \mathbf{n}\backslash\{i\}} \mathrm{Cov}(X_i, X_j) \right) \\ & = \frac{1}{n} ( \sigma^2 + 0 ) \\ & = \frac{\sigma^2}{n} \\ \end{align} $$

In this case, only the summand for $j = i$ contributes to the final value; the remaining ones are zero, by assumption.

I'm missing an interpretation of the covariance that would help me decide what to make of $(1)$: i.e. is it remarkable? is it uncanny? is it fortuitous? is it trivial?.... Is there an interpretation of the covariance that will put $(1)$ in the right perspective?

PS: I know, of course, that the covariance of two random variables $X$ and $Y$ is the difference between the expected value of their product and the product of their expected values. Alternatively, it is the expected value of the product of the "errors", $(X - \mathbb{E}(X))(Y - \mathbb{E}(Y))$. But neither interpretation tells me what to make of $(1)$ above.

EDIT: If I understand Henry's answer correctly, one can cast each side so that they have a similar-looking form, namely, for the LHS:

$$ \begin{align} \mathrm{Var}(\overline{X}) = \mathrm{Var}\left(\sum_{j\in\mathbf{n}}\frac{X_j}{n}\right) &= \sum_{j\in\mathbf{n}} \sum_{k\in\mathbf{n}}\mathrm{Cov}\left(\frac{X_j}{n}, \frac{X_k}{n}\right) \\ &= \sum_{j\in\mathbf{n}} \mathrm{Var}\left(\frac{X_j}{n}\right) + \sum_{j\in\mathbf{n}}\sum_{k\in\mathbf{n}\backslash\{j\}}\mathrm{Cov}\left(\frac{X_j}{n}, \frac{X_k}{n}\right), \tag{2} \end{align} $$

and on the RHS (using the trick of expressing $X_i$ as $\sum_{k\in\mathbf{n}}X_i/n$):

$$ \begin{align} \mathrm{Cov}(X_i, \overline{X}) = \mathrm{Cov}(\overline{X}, X_i) &= \mathrm{Cov}\left( \sum_{j\in\mathbf{n}}\frac{X_j}{n} , \sum_{k\in\mathbf{n}} \frac{X_i}{n}\right)\\ &= \sum_{j\in\mathbf{n}} \sum_{k\in\mathbf{n}}\mathrm{Cov}\left(\frac{X_j}{n}, \frac{X_i}{n}\right) \\ &= \sum_{k\in\mathbf{n}}\mathrm{Var}\left(\frac{X_i}{n}\right) + \sum_{j\in\mathbf{n}\backslash\{i\}} \sum_{k\in\mathbf{n}}\mathrm{Cov}\left(\frac{X_j}{n}, \frac{X_i}{n}\right) \tag{3}\\ \end{align} $$

Now each side consists indeed of a sum of $n$ terms with value $\sigma^2/n^2$ plus a sum of $n(n-1)$ terms with value $0$.

But I think this maneuver just "smears out" the difference I originally pointed out, and makes it harder to notice. If one pays attention to the indices in each of the final expressions in $(2)$ and $(3)$, one can see that they are semantically different after all.

Then again, at heart, what this manipulation boils down to is to re-express $(1)$ as

$$ \mathrm{Cov}(\overline{X}, \overline{X}) = \mathrm{Cov}(\overline{X}, X_i) \tag{$1^\prime$} $$

or even as

$$ \mathrm{Cov}\left(\sum_{k\in\mathbf{n}}X_k, \sum_{k\in\mathbf{n}}X_k\right) = \mathrm{Cov}\left(\sum_{k\in\mathbf{n}}X_k, \sum_{k\in\mathbf{n}}X_i\right) \tag{$1^{\prime\prime}$}, $$

which, I admit, does make the equality look a little less uncanny.

Answer 1

They are hardly unrelated.

If you rewrite $X_i$ as $\sum_{k=1}^n X_i/n $ while $\bar{X}= \sum_{i=1}^n X_i/n$ with $\text{Var}(X_i/n)= \dfrac{\sigma^2}{n^2}$ then both the variance and the covariance in your $(1)$ become each the sum of $n(n-1)$ copies of $0$ plus $n$ copies of $\dfrac{\sigma^2}{n^2}$.

Answer 2

There is a small positive correlation---exactly $1/\sqrt{n}$, between $X_i$ and $\bar X$. One expects that simply because of the way in which $X_i$ goes into $\bar X$. Make $X_i$ bigger and, all other things being equal, $\bar X$ also gets bigger. That would be perfect correlation if all other things were equal, but "all other things" means specifically all but one of $X_1,\ldots,X_n$, and they're not generally equal.

A quite interesting corollary is that $\operatorname{cov}(X_i-\bar X, \bar X)=0$. That's interesting because with suitable additional hypotheses, that leads us to conclude that the sample mean and the sample variance are independent, which in turn leads to such things as the basics of the theory of Student's t-distribution.