Expected Squared Error, why do I get a different result I am not sure if the context matters, but this piece of mathematical equivalence transformation is about showing that Bagged models in Machine Learning require to be uncorrelated in order to reduce the total variance of the ensemble.
Regardless, let me show you the transformation I have as part of the solution:
let $E[\epsilon_k^2]=v$ and $E[\epsilon_k \epsilon_l]=c$
computing the total variance of $K$ models:
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2] = 
\frac{1}{K^2}E[\sum_{k=1}^K (\epsilon_k^2+ \sum_{k \neq l} \epsilon_k \epsilon_l)] = \frac{1}{K}v + \frac{K-1}{K}c$
Now, here is how I tried the transformation:
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2] = 
\frac{1}{K^2}E[\sum_{k=1}^K \epsilon_k^2+ 2\sum_{k \neq l} \epsilon_k \epsilon_l)] = \frac{1}{K}v + \frac{2}{K}c$
which leads to a different result. I basically entangled the expression
$\sum_{k=1}^K (\epsilon_k^2+ \sum_{k \neq l} \epsilon_k \epsilon_l)$
to
$\sum_{k=1}^K \epsilon_k^2+ 2\sum_{k \neq l}^K \epsilon_k \epsilon_l$
Are both expressions not equivalent?
 A: The term $\sum_{k=1}^K\sum_{k \neq l}^K \epsilon_k \epsilon_l$ can be represented in a $ K\times K$ matrix:
$$ \begin{pmatrix}
 0 & \mathbb E(\epsilon_1 \epsilon_2)& \cdots & \mathbb E(\epsilon_1 \epsilon_K) \\ \\
 \mathbb E(\epsilon_2 \epsilon_1) &  0& \cdots &  \mathbb E(\epsilon_2 \epsilon_k)  \\ \\
 \vdots & \vdots & \ddots & \vdots \\ \\
\mathbb E(\epsilon_1 \epsilon_K)  &  \mathbb E(\epsilon_K \epsilon_2) & \cdots & 0
\end{pmatrix}$$
Here $\epsilon_i \epsilon_j=\epsilon_j \epsilon_i\ \ \forall \ i,j=\{1,...,K\}$ and $\mathbb E(\epsilon_k \epsilon_l)=c$.
For $k=1$ and $\sum_{k \neq l}^K \epsilon_k \epsilon_l$ you sum up the values of the first row. This is $c\cdot (K-1)$
In total we have $K$ rows. Thus the sum of all elements in the matrix is $c\cdot (K-1)\cdot K$
Finally you divide the result by $K^2: \qquad c\cdot \frac{(K-1)\cdot K}{K^2}=c\cdot \frac{(K-1)}{K}$
A: We are given :
$E[\epsilon_k^2]=v$
$E[\epsilon_k \epsilon_l]=c$
We want to Evaluate :
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2]$
The Question Post has got Contradictory Inconsistent Total. Where is it going wrong ?
Issue 1 :
When expanding the Square , we are introducing 2 Index Variables with Summation Criteria $ k ≠ l $ , hence we have to consider that in the total.
Issue 2 :
When expanding the Square , we will get $ \epsilon_1 \epsilon_2 $ & $ \epsilon_2 \epsilon_1 $ which will give $ 2 \epsilon_1 \epsilon_2 $ , hence we must not Explicitly include the 2. Alternately , we might make the Summation Criteria $ k < l $ , to skip  $ \epsilon_2 \epsilon_1 $ , in which case we must Explicitly include the 2.
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2] = 
\frac{1}{K^2}E[\sum_{k=1}^K (\epsilon_k^2+ \sum_{k,l : k ≠ l} \epsilon_k \epsilon_l)] = \frac{1}{K}v + \frac{(K-1)(K-1+1)}{K^2}c$
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2] = 
\frac{1}{K^2}E[\sum_{k=1}^K (\epsilon_k^2+ 2 \sum_{k,l : k < l} \epsilon_k \epsilon_l)] = \frac{1}{K}v + 2\frac{(K-1)(K-1+1)/2}{K^2}c$
These two are Correct & Equivalent.
We eventually will get :
$E[(\frac{1}{K} \sum_{k=1}^K \epsilon_k)^2] = \frac{1}{K}v + \frac{(K-1)}{K}c$
Intuitively : We see that every Model contributes Equally to the total , $1/k$ for 1 self Model { $v$ } & $1/k$ for every other $(K-1)$ Model { $c$ } , hence giving the Intuitive Weighted Average.
