So we have an experiment going on where a group of colleagues evaluate each other's skill levels, ranging from 1 to 5. We have $n$ colleagues, so in the end we should get $n^2$ evaluations. However, nobody evaluates himself, so that leaves us with $n^2 - n$ evaluations.

The evaluations are weighted using the following rationale: Let's say we have two colleagues, Alice and Bob. When Alice evaluates the skill level of Bob, then Alice needs to set a confidence factor, ranging from 0 to 100%. Alice may think: "I can rate the skill level of Bob very confidently, because we work in the same team". In that case, Alice would choose a rather high confidence factor, such as 0.8. Bob can then give his own estimation of how high Alice's confidence may be. Bob may think: "Alice knows about my skills, but not that much." He could then give her rating a confidence of 60%.

So now, for the single evaluation dubbed "Alice evaluates Bob", we have Alice's confidence level $c_1 = 0.8$ and Bob's confidence in Alice's rating of $c_2 = 0.6$. From this we calculate the mean confidence $c_3$ of that rating using the arithmetic average:

$$ c_3 = \frac{(c_1 + c_2)}{2} $$

Let's say Alice has evaluated Bob's skill level to be at 5. This rating then receives the weight of $c_3 = 0.7$. At the end of the evaluation process, Bob will have 11 evaluations with a (potentially) different weight, and Bob's final rating will be the weighted sum over all evaluations he received.

Now there's a problem: A lot of people had no clue about Bob's skill level and rated him in the middle, although with a very low confidence. The mass of low-confidence evaluations shifted Bob's skill level towards the middle, because it overruled the few ratings with high confidence.

Now comes the question: What would be a proper method to give even more weight to high confidence ratings, and even less weight to low confidence ratings? Is this something we could fix using a different calculation for $c_3$? I've searched the web and I found the "quadratic mean" which may help. Any ideas?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.