0
$\begingroup$

So we have an experiment going on where a group of colleagues evaluate each other's skill levels, ranging from 1 to 5. We have $n$ colleagues, so in the end we should get $n^2$ evaluations. However, nobody evaluates himself, so that leaves us with $n^2 - n$ evaluations.

The evaluations are weighted using the following rationale: Let's say we have two colleagues, Alice and Bob. When Alice evaluates the skill level of Bob, then Alice needs to set a confidence factor, ranging from 0 to 100%. Alice may think: "I can rate the skill level of Bob very confidently, because we work in the same team". In that case, Alice would choose a rather high confidence factor, such as 0.8. Bob can then give his own estimation of how high Alice's confidence may be. Bob may think: "Alice knows about my skills, but not that much." He could then give her rating a confidence of 60%.

So now, for the single evaluation dubbed "Alice evaluates Bob", we have Alice's confidence level $c_1 = 0.8$ and Bob's confidence in Alice's rating of $c_2 = 0.6$. From this we calculate the mean confidence $c_3$ of that rating using the arithmetic average:

$$ c_3 = \frac{(c_1 + c_2)}{2} $$

Let's say Alice has evaluated Bob's skill level to be at 5. This rating then receives the weight of $c_3 = 0.7$. At the end of the evaluation process, Bob will have 11 evaluations with a (potentially) different weight, and Bob's final rating will be the weighted sum over all evaluations he received.

Now there's a problem: A lot of people had no clue about Bob's skill level and rated him in the middle, although with a very low confidence. The mass of low-confidence evaluations shifted Bob's skill level towards the middle, because it overruled the few ratings with high confidence.

Now comes the question: What would be a proper method to give even more weight to high confidence ratings, and even less weight to low confidence ratings? Is this something we could fix using a different calculation for $c_3$? I've searched the web and I found the "quadratic mean" which may help. Any ideas?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.