Finding a more direct way to reach $\mathbb{E} \left( \sum (X_i - \mu)^2 \right) - \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right) = \sigma^2$

Let $X_i$ be independent random variables, $\forall\,i \in \mathbf{n} \equiv \{0,\dots,n-1\}$, with identical expectation value $\mathbb{E}(X_i)=\mu$, and identical variance $\mathrm{Var}(X_i)=\sigma^2$.1 Also, let $\overline{X}$ be their average, $\frac{1}{n}\sum X_i$ (where the summation for all $i\in \mathbf{n}$ is implicit, a convention I'll use throughout).

It is not hard to show, by direct (if slightly tedious) calculation, that

$$\textstyle \mathbb{E} \left( \sum (X_i - \mu)^2 \right) - \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right) = \sigma^2$$

Whenever I arrive at a very "simple" result through a "tedious" derivation, as in this case, I get the strong suspicion that there has to be a more direct, and yet entirely rigorous, reasoning to reach it. Or rather, a way to view the problem that makes the result immediately "obvious".2

In this case, the best I have found goes something like this: the above result follows from the fact that, first,

$$\textstyle \mathbb{E} \left( \sum (X_i - \mu)^2 \right) = n \sigma^2$$

and, second, if we subtract $\overline{X}$ instead of $\mu$, we have "lost one degree of freedom", and "therefore"

$$\textstyle \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right) = (n - 1) \sigma^2$$

I find this hand-wavy argument thoroughly unconvincing. (I doubt that those who propose it would believe it if they didn't already know the result from a more rigorous derivation.)

Is there something better?

EDIT:

Here's an example of the kind of argument I'm looking for. It still has too many gaps to be satisfactory, but at least it shows a reasoning that does not require pencil and paper: it could be delivered orally, or with crude "marks in the sand" (no algebra), and be readily understood.

First we can see that

$$\textstyle \mathbb{E} \left( \sum (X_i - \mu)^2 \right) \geq \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right)$$

Why? Because, the value of $c$ that minimizes $\sum (X_i - c)^2$ is $c = \overline{X}$ (a fact for which I could give a similarly hand-wavy, not-entirely-watertight argument, though easy to prove by tedious computation), which means that whenever $\overline{X} \neq \mu$, we would have $\sum (X_i - \mu)^2 > \sum (X_i - \overline{X})^2$.

Now, what "drives" $\overline{X}$ away from $\mu$ (so to speak) is $\sigma^2$. So we can conclude that the difference

$$\textstyle \mathbb{E} \left( \sum (X_i - \mu)^2 \right) - \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right) \geq 0$$

should increase monotonically as $\sigma^2$ increases...

Fair enough, but why is the difference exactly $\sigma^2$, and not, say, $\sigma^2/n$, or *gasp* $\pi \sigma^2/n$? Here my hand-waving begins to run out of steam... It is suggestive that $\mathbb{E}(\overline{X}) = \mu$ and

$$\mathrm{Var}(\overline{X}) = \frac{\sigma^2}{n} = \mathbb{E} \left( ( \overline{X} - \mathbb{E}(\overline{X}))^2 \right) = \mathbb{E} \left( ( \overline{X} - \mu )^2 \right)$$

Therefore it is tempting to surmise that, since, for each $i$, $(X_i - \mu) - (X_i - \overline{X}) = \overline{X} - \mu$, then each term $(X_i - \mu)^2 - (X_i - \overline{X})^2$ would contribute, "on average", $\mathbb{E}\left( ( \overline{X} - \mu )^2 \right) = \sigma^2/n$ to the total difference. This would require justifying the tantalizingly Pythagorean-looking equality:

$$\mathbb{E} \left( ( X_i - \mu )^2 \right) = \mathbb{E} \left( ( X_i - \overline{X})^2 + ( \overline{X} - \mu )^2 \right)$$

...though I readily concede that this is beginning to look as tedious as any algebraic computation.

(I note that in this argument I did not use the fact that in this case $\mathrm{Var}(\sum X_i)=\sum\mathrm{Var}(X_i)$, which is surely the way forward.)

1 The typically given condition is to say that the $X_i$ are independent and identically distributed, but, AFAICT, the last condition is stronger than necessary. For that matter, as Dilip Sarwate pointed out, the independence condition is also stronger than needed. It is sufficient that $\mathrm{Var}(\sum X_i)=\sum\mathrm{Var}(X_i)$.

2 The "scare quotes" around "simple", "tedious", and "obvious" aim to convey the concession that these terms are all, of course, in the eye of the beholder. So "simplicity" is shorthand for subjective simplicity or perceived simplicity, etc. Also in the eye of the beholder is how much (perceived) tedium seems too much relative to the (perceived) simplicity. If the difference $\mathbb{E} \left( \sum (X_i - \mu)^2 \right) - \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right)$ had been, say, $\sigma^2/\sqrt{\pi}$, I would not have perceived the standard algebraic derivation as particularly tedious, because $\sigma^2/\sqrt{\pi}$ does not seem to me particularly simple.

-
You are correct when you say that the random variables need not be identically distributed (as long as they have the same mean and same variance without which the result to proved would not make much sense anyway). Now consider whether independence is really necessary or whether the random variables being uncorrelated (which is a pairwise property) is all that is needed. –  Dilip Sarwate Apr 5 '13 at 11:43
So, how could I improve my answer? It seems to show precisely why that difference is $\sigma^2$. It's because the best you can do is have all the $X$'s be uncorrelated except for each $X$ to itself, thus the one $\sigma^2$. –  trb456 Apr 5 '13 at 13:13
Also, your variance inequality is backwards. It should be $\mathrm{Var}(\sum X_i) \geq \sum\mathrm{Var}(X_i)$, because covariance matrices have to be positive semidefinite. I think this may be your issue, as this fact reinforces that uncorrelated is the lowest possible difference in variance. –  trb456 Apr 5 '13 at 13:24

An intuitive and possibly "completely rigorous" derivation of the result

The result in question follows from an application of the Pythagorean theorem of plane geometry.

Without loss of generality, assume that $\mu=0$ and consider a fixed point $\mathbf{x} = (x_1,x_2,\ldots, x_n)$ in $\mathbb R^n$. Define $\bar{x}$ as $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$. The set of all points $(y_1,y_2,\ldots, y_n) \in \mathbb R^n$ that satisfy $$y_1+y_2+\cdots + y_n = n\bar{x}$$ is a hyperplane $H$ in $\mathbb R^n$ that contains $\mathbf{x}$, and it is easily shown that the point in $H$ that is closest to the origin $\mathbf 0$ is $\bar{\mathbf{x}} = (\bar{x},\bar{x}, \ldots, \bar{x})$ and that the straight line through $\mathbf 0$ and $\bar{\mathbf{x}}$ is perpendicular to $H$. Now, the three points $\mathbf 0$, $\bar{\mathbf{x}}$, and $\mathbf x$ define a plane, and in this plane, they are the vertices of a right triangle (with right angle at $\bar{\mathbf{x}}$). The Pythagorean theorem of plane geometry tells us that $$\sum_{i=1}^n x_i^2 = \sum_{i=1}^n \left(\bar{x}\right)^2 + \sum_{i=1}^n \left(x_i-\bar{x}\right)^2 = n\left(\bar{x}\right)^2 + \sum_{i=1}^n \left(x_i-\bar{x}\right)^2$$ or, equivalently, $$\sum_{i=1}^n x_i^2 - \sum_{i=1}^n \left(x_i-\bar{x}\right)^2 = n\left(\bar{x}\right)^2.$$ Now, the above identity holds for all choices of the $x_i$, and in particular, it holds for all realizations $(x_1, x_2, \ldots, x_n)$ of the random vector $(X_1, X_2, \ldots, X_n)$, that is, $$\sum_{i=1}^n X_i^2 - \sum_{i=1}^n \left(X_i-\bar{X}\right)^2 = n\left(\bar{X}\right)^2 ~~\text{with probability } 1.$$ Therefore, assuming all the expectations exist, we have that $$E\left[\sum_{i=1}^n X_i^2\right] - E\left[\sum_{i=1}^n \left(X_i-\bar{X}\right)^2\right] = nE\left[\left(\bar{X}\right)^2\right].$$ Introducing a common mean $\mu$ for the $X_i$'s merely translates the origin to $(\mu,\mu, \ldots, \mu)$ giving

$$E\left[\sum_{i=1}^n (X_i-\mu)^2\right] - E\left[\sum_{i=1}^n \left(X_i-\bar{X}\right)^2\right] = nE\left[\left(\bar{X}-\mu\right)^2\right] = n\cdot\operatorname{var}\left(\bar{X}\right).$$

Notice that the result holds for all random variables with common mean $\mu$: we have not made any assumptions about independence or zero correlation or even about common variance. Now, for the special case of uncorrelated random variables with common variance $\sigma^2$, the right side of the above equality is just $$n\cdot\operatorname{var}\left(\bar{X}\right) = n\cdot\operatorname{var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = n \cdot \frac{1}{n^2}\sum_{i=1}^n \operatorname{var}(X_i) = \sigma^2$$ giving

$$E\left[\sum_{i=1}^n (X_i-\mu)^2\right] - E\left[\sum_{i=1}^n \left(X_i-\bar{X}\right)^2\right] = \sigma^2.$$

(Previous answer: no intuition or geometry, just a simple derivation)

You already have noted that $$E\left[\sum_{i=1}^n (X_i-\mu)^2\right] = \sum_{i=1}^n E[(X_i-\mu)^2] = n\sigma^2.$$ Since $E\left[\bar{X}\right] = \mu = E[X_i]$, we have that $Y_i = X-\bar{X}$ is a zero-mean random variable, and so $E\left[\left(X_i-\bar{X}\right)^2\right]$ is the variance of $Y_i$ which gives \begin{align} E\left[\left(X_i-\bar{X}\right)^2\right] &= \operatorname{var}(Y_i) = \operatorname{var}\left(X_i-\bar{X}\right)\\ &= \operatorname{var}\left(\frac{n-1}{n}X_i-\sum_{j\neq i}\frac{X_j}{n}\right) &\scriptstyle{\text{write as a weighted sum of the uncorrelated variables }X_i}\\ &= \left(\frac{(n-1)^2}{n^2}+ (n-1)\frac{1}{n^2}\right)\sigma^2 &\scriptstyle{\text{so that we can use the formula}\operatorname{var}\left(\sum_i a_iX_i\right) = \sum_i a_i^2\operatorname{var}(X_i)}\\ &= \frac{n-1}{n}\sigma^2&\scriptstyle{\text{for the variance of a sum of uncorrelated random variables}} \end{align} leading to $$E\left[\sum_{i=1}^n\left(X_i-\bar{X}\right)^2\right] = (n-1)\sigma^2.$$

-
It would be nice to explain that the step taking $\mathrm{var}\left(X_i-\bar{X}\right)$ to $\mathrm{var}\left(\frac{n-1}{n}X_i-\sum\limits_{j\ne i}\frac{X_j}{n}\right)$ is there to separate the independent variables (since $X_i$ and $\bar{X}$ are not independent). Otherwise, this is a nice approach. (+1) –  robjohn Apr 5 '13 at 17:02
@robjohn I added the explanations that you suggested, but later thought of a different way of approaching the result which seemed more intuitive, and included this in my answer too. Comments on the latter would be most appreciated. –  Dilip Sarwate Apr 6 '13 at 15:06
I'm not sure I see the relation to $\mathbb{R}^n$ in more than a pro forma sense. The use of the $y_i$ in a hyperplane seems to remove the independence from the $y_i$. However, the identity achieved there is still true: \begin{align} \sum_{i=1}^nx_i^2-\sum_{i=1}^n\left(x_i-\bar{x}\right)^2 &=\sum_{i=1}^nx_i^2-\sum_{i=1}^n\left(x_i^2-2x_i\bar{x}+\bar{x}^2\right)\\ &=2\bar{x}\sum_{i=1}^nx_i-n\bar{x}^2\\ &=2n\bar{x}^2-n\bar{x}^2\\ &=n\bar{x}^2 \end{align} In any case, it is nice to see another approach. –  robjohn Apr 6 '13 at 16:45

Note that if the $X_i$ are independent, \begin{align} \mathrm{Var}\left(\bar{X}\right) &=\mathrm{Var}\left(\frac1n\sum X_i\right)\\ &=\frac1{n^2}\sum\mathrm{Var}(X_i)\\ &=\frac1{n^2}n\sigma^2\\ &=\frac{\sigma^2}{n}\tag{1} \end{align} Simply expand and simplify to get \begin{align} &\mathbb{E}\left(\sum\left(X_i-\mu\right)^2\right)-\mathbb{E}\left(\sum\left(X_i-\bar{X}\right)^2\right)\\ &=\mathbb{E}\left(\sum\left(X_i^2-2\mu X_i+\mu^2\right)-\sum\left(X_i^2-2\bar{X}X_i+\bar{X}^2\right)\right)\\ &=\mathbb{E}\left(\sum\left(\mu^2+2\left(\bar{X}-\mu\right)X_i-\bar{X}^2\right)\right)\\ &=\mathbb{E}\left(n\mu^2+2n\left(\bar{X}-\mu\right)\bar{X}-n\bar{X}^2\right)\\ &=n\mathbb{E}\left(\mu^2-2\mu\bar{X}+\bar{X}^2\right)\\ &=n\mathbb{E}\left(\left(\mu-\bar{X}\right)^2\right)\\ &=n\frac{\sigma^2}{n}\tag{2}\\ &=\sigma^2 \end{align} $(2)$ is just $n$ times $(1)$.

-
This is a reordered version of this argument. –  robjohn Apr 5 '13 at 12:30

I think you want to do the tedious calculations and then extract the key insight. And for me, the key insight is that, for each $X_i$:

$$\textstyle \mathbb{E} \left( X_i \overline{X} \right) = \frac{\sigma^2}{n} +\mu^2$$ which does require that $Cov(X_i,X_j)=0$ if $i\ne j$.

Once you have this, then you can see that:

$$\textstyle \mathbb{E} \left( \sum (X_i - \overline{X})^2 \right) = \mathbb{E} \left( \sum [(X_i -\mu)+(\mu- \overline{X})]^2 \right)$$

$$\textstyle = \mathbb{E} \left( \sum (X_i -\mu)^2+\sum2(X_i -\mu)(\mu- \overline{X})+\sum(\mu- \overline{X})^2 \right)$$

$$\textstyle = n\sigma^2-2\sigma^2+\sigma^2=(n-1)\sigma^2$$

which is indeed tedious to compute, but this is where your "degree of freedom" shows up.

-