# Statistics: Higher confidence, higher weight

So we have an experiment going on where a group of colleagues evaluate each other's skill levels, ranging from 1 to 5. We have $$n$$ colleagues, so in the end we should get $$n^2$$ evaluations. However, nobody evaluates himself, so that leaves us with $$n^2 - n$$ evaluations.

The evaluations are weighted using the following rationale: Let's say we have two colleagues, Alice and Bob. When Alice evaluates the skill level of Bob, then Alice needs to set a confidence factor, ranging from 0 to 100%. Alice may think: "I can rate the skill level of Bob very confidently, because we work in the same team". In that case, Alice would choose a rather high confidence factor, such as 0.8. Bob can then give his own estimation of how high Alice's confidence may be. Bob may think: "Alice knows about my skills, but not that much." He could then give her rating a confidence of 60%.

So now, for the single evaluation dubbed "Alice evaluates Bob", we have Alice's confidence level $$c_1 = 0.8$$ and Bob's confidence in Alice's rating of $$c_2 = 0.6$$. From this we calculate the mean confidence $$c_3$$ of that rating using the arithmetic average:

$$c_3 = \frac{(c_1 + c_2)}{2}$$

Let's say Alice has evaluated Bob's skill level to be at 5. This rating then receives the weight of $$c_3 = 0.7$$. At the end of the evaluation process, Bob will have 11 evaluations with a (potentially) different weight, and Bob's final rating will be the weighted sum over all evaluations he received.

Now there's a problem: A lot of people had no clue about Bob's skill level and rated him in the middle, although with a very low confidence. The mass of low-confidence evaluations shifted Bob's skill level towards the middle, because it overruled the few ratings with high confidence.

Now comes the question: What would be a proper method to give even more weight to high confidence ratings, and even less weight to low confidence ratings? Is this something we could fix using a different calculation for $$c_3$$? I've searched the web and I found the "quadratic mean" which may help. Any ideas?