Tricky proof that the weighted average is a better estimate than the un-weighted average:

Question

The following is a word for word copy of a tough question and the solution to it. I have marked $\color{red}{\mathrm{red}}$ the parts of the solution for which I do not understand and the parts marked with $\color{#180}{\text{green underbraces}}$ are not part of the solution and represent what I think the author is saying:

Start of Question:

Show that the simple average: $\bar{q}=\sum_i\frac{q_i}{N}$ is not as good an estimate as the weighted average. Using propagation of errors, find the uncertainty $\sigma_{\bar{q}}$ on the simple average. Is $\sigma_{\bar{q}}$ smaller than $\sigma_j$ for all $j$ in this case? By taking the ratio: $$\frac{\sigma_{\bar{q}}^2}{\sigma_{\widehat{e}}^2}$$ show that $\sigma_{\bar{q}}\ge\sigma_{\widehat{e}}$, which means the weighted average is better. Under what condition does $\sigma_{\bar{q}} = \sigma_{\widehat{e}}$. What do the general expressions for $\widehat{e}$ and $\sigma_{\widehat{e}}$ simplify to in this special case?

End of Question.

---

Start of Solution:

With $\bar{q}=\sum_i\frac{q_i}{N}$, then $\frac{\partial{\bar{q}}}{\partial{q_i}}=\frac{1}{N}$. Hence, $$\sigma_{\bar{q}}^2=\sum_i\left(\cfrac{\partial{\bar{q}}}{\partial{q_i}}\right)^2\sigma_i^2 \tag{Error propagation formula}$$ $$=\sum_i\frac{1}{N^2}\sigma_i^2=\frac{1}{N^2}\sum_i\sigma_i^2$$ In this case $\sigma_{\bar{q}}$ is not necessarily smaller than all the $\sigma_j$. A specific example; with $N=2$ measurements with uncertainties $\sigma_1=1.0$ and $\sigma_2=0.1$, then $\sigma_{\bar{q}}^2=\frac{1.01}{4}\approx 0.25$ so $\sigma_{\bar{q}}=0.5$ and hence is bigger than $\sigma_2$.

Taking the ratio with $\sigma_{\bar{q}}^2$ gives $$\frac{\sigma_{\bar{q}}^2}{\sigma_{\widehat{e}}^2}=\frac{1}{N^2}\left(\sum_{i}\sigma_i^2\right)\left(\sum_{j}\frac{1}{\sigma_j^2}\right)$$ Note that two different indices are required on the two running sums. The double sum can be divided into the terms with $i=j$ and $i\ne j$: $$\frac{\sigma_{\bar{q}}^2}{\sigma_{\widehat{e}}^2}=\frac{1} {N^2}\sum_{i,j}\frac{\sigma_i^2}{\sigma_j^2}=\frac{1}{N^2}\left(\sum_{\color{#180}{\underbrace{\color{red}{i}}_{\color{#180}{i: i= j}}}}\frac{\sigma_i^2}{\sigma_i^2}+\sum_{\color{#180}{\underbrace{\color{red}{i,j\ne i}}_{\color{#180}{i: i \ne j}}}} \frac{\sigma_i^2}{\sigma_j^2}\right)=\frac{1}{N^2}\left(N+\sum_{\color{#180}{\underbrace{\color{red}{i,j\ne i}}_{\color{#180}{i: i \ne j}}}}\frac{\sigma_i^2}{\sigma_j^2}\right)$$ $\color{red}{\text{There are}} \space\color{red}{N^{2}} \color{red}{-N} \space \color{red}{\text{terms in the second sum. For every}\space}\color{red}{\frac{\sigma_i^2}{\sigma_j^2}}\space\color{red}{\text{term, there is also a}\space}\color{red}{\frac{\sigma_j^2}{\sigma_i^2}}\space\color{red}{\text{term.}}$

Let $r_{ij}=\frac{\sigma_i^2}{\sigma_j^2}$; then the sum of the two terms is then $r_{ij}+\frac{1}{r_{ij}}$. This has a minimum value when $$\frac{\mathrm{d}}{\mathrm{d}r_{ij}}\left(r_{ij}+\frac{1}{r_{ij}}\right)=1-\frac{1}{r_{ij}^2}=0$$ which holds for $r_{ij}=1$ (where $r_{ij}=-1$ is dropped as $r_{ij}$ must be positive). Hence, the minimum possible value for each pair of terms is $1+\frac{1}{1}=2$. Hence, the total of all the terms must be $\ge N^2 -N$. Therefore, $$\frac{\sigma_{\bar{q}}^2}{\sigma_{\widehat{e}}^2}\ge \frac{1}{N^2}\left(N+N^2-N\right)\ge 1$$ Hence $\sigma_{\bar{q}}\ge\sigma_{\widehat{e}}$ as required. The equality only holds if all the $r_{ij}=1$. This means that all the uncertainties must have the same value; let this be $\sigma$. The weighted average then becomes $$\widehat{e}=\frac{\left(\sum_i q_i\right)/\sigma^2}{\left(\sum_i 1\right)/\sigma^2}=\sum_i\frac{q_i}{N}=\bar{q}$$ i.e the usual un-weighted average. The uncertainty on the weighted average becomes $$\frac{1}{\sigma_{\widehat{e}}^2}=\sum_i\frac{1}{\sigma^2}=\frac{N}{\sigma^2}$$ and so $$\sigma_{\widehat{e}}=\frac{\sigma}{\sqrt{N}}$$ which is the uncertainty on the usual average.

End of Solution.

---

For the $\color{red}{\mathrm{red}}$ sentence I have no idea why $\color{red}{\text{there are}} \space\color{red}{N^{2}} \color{red}{-N} \space \color{red}{\text{terms in the second sum}}$. Since $$\frac{1}{N^2}\left(N+\sum_{\color{#180}{\underbrace{\color{red}{i,j\ne i}}_{\color{#180}{i: i \ne j}}}}\frac{\sigma_i^2}{\sigma_j^2}\right)=\frac{1}{N}+{\frac{1}{N^2}\sum_{\color{#180}{\underbrace{\color{red}{i,j\ne i}}_{\color{#180}{i: i \ne j}}}}\frac{\sigma_i^2}{\sigma_j^2}}$$ I assume that the author means $\color{red}{\text{there are}} \space\color{red}{N^{2}} \color{red}{-N} \space \color{red}{\text{terms}}$ in $${\frac{1}{N^2}\sum_{\color{#180}{\underbrace{\color{red}{i,j\ne i}}_{\color{#180}{i: i \ne j}}}}\frac{\sigma_i^2}{\sigma_j^2}}$$ If this is the case; Could someone please explain to me how this is true and why $\color{red}{\text{for every}\space}\color{red}{\frac{\sigma_i^2}{\sigma_j^2}}\space\color{red}{\text{term, there is also a}\space}\color{red}{\frac{\sigma_j^2}{\sigma_i^2}}\space\color{red}{\text{term}}$?

Also; Can I please get confirmation that the text written in the $\color{#180}{\text{green underbraces}}$ is correct or not?

Thank you,

BLAZE.

Answer 1

Note that $$\sum_{i,j}a_{i,j}=\sum_i^N\sum_j^N a_{i,j},$$ so there is $N^2$ terms in this sum.

Similarly $$\sum_{i,j\ne i} a_{i,j}=\sum_i^N\sum_{j:j\ne i}^N a_{i,j} = \color{red}{\sum_i^N\left(\sum_{j=1}^{i-1} a_{i,j} + \sum_{j=i+1}^{N} a_{i,j}\right)}= \sum_i(a_{i,1}+a_{i,2}+\dots+a_{i,i-1}+a_{i,i+1}+a_{i,i+2}+\dots +a_{i,N}),$$ there is $N$ terms of the form $a_{i,i}$ which omitted in this sum, so the sum has $N^2-N$ terms.

---

Thus, $i:i\ne j$ is not the same as $i,j\ne i$. The former means all $i$' such that (':' = 'such that') $i\ne j$, i.e. all $i$'s except $i=j$ (i.e. $j$ is something known in this case). The latter says all $i$'s and $j$'s except the case of $i=j$.

For every $\frac{\sigma_i^2}{\sigma_j^2}$ t‌​erm, there is also a $\frac{\sigma_j^2}{\sigma_i^2}$ term comes from the commutativity of the addition, which gives the identity $$\sum_i^N\sum_{j:j\ne i}^N a_{i,j} = \sum_j^N\sum_{i:i\ne j}^N a_{i,j}$$ This is just taking $$(0\; +a_{1,2}+a_{1,3}+a_{1,4}+\dots +a_{1,N})+\\ (a_{2,1}+0+a_{2,3}+a_{2,4}+\dots +a_{2,N})+\\ (a_{3,1}+a_{3,2}+0+a_{3,4}+\dots +a_{3,N})+\\ \vdots\\ (a_{N,1}+a_{N,2}+a_{N,3}+\dots +a_{N,N-1}+0) $$ and changing columns with rows $$(0+a_{2,1}+a_{3,1}+a_{4,1}+\dots +a_{N,1})+\\ (a_{1,2}+0+a_{3,2}+a_{4,2}+\dots +a_{N,2})+\\ (a_{1,3}+a_{2,3}+0+a_{4,3}+\dots +a_{N,3})+\\ \vdots\\ (a_{1,N}+a_{2,N}+a_{3,N}+\dots +a_{N-1,N}+0) $$

Try to convince your self these are similar sums.

Please let me know if you need more details.