Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I have a form where users can rate presentations and they can say how knowledgeable they are on the subject. The range for both sets is 1-5. (1 being lousy and 5 being great) So for example, a user can rate a presentation with score 1 and their knowledge 5, which means they are very sure that the presentation was bad.

A presentation can be rated by two distinct people who don't know what the other person rated. If these scores are far apart, a third rater should come into play who acts as a tiebreaker.

What I need is a way to calculate the difference between the two distinct ratings on which I can decide whether or not I should ask the tiebreaker to rate. Obviously it should be some sort of weighted difference. If we go down this path, it could be implemented as follows:

(score person A)(knowledge person A) - (score person B)(knowledge person B)

However this doesn't have the desired result, because for example 3*2 - 1*5 = 1 is a very small difference whereas person B is really sure about his rating so a tiebreaker should probably come into play here. On the other hand 5*5 - 4*5 = 5 is a big difference but both raters are very confident that they know what they are talking about so a tiebreaker should NOT come into play.

What I think would be of help is if somehow the knowledge factor is not linear but progresses along a sort of bell curve. Any ideas on how to come with a better algorithm would be appreciated.

share|improve this question

2 Answers 2

up vote 1 down vote accepted

Elaborating on jpalecek's suggestion, I would fit a best average rating $x$ in the least-squares sense, weighted by experience; if $w_i$ are the experiences and $x_i$ are the ratings, this amounts to minimizing the energy $$\frac{w_1 (x-x_1)^2 + w_2(x-x_2)^2}{w_1+w_2}$$ which has solution $$x=\frac{w_1 x_1 + w_2 x_2}{w_1+w_2}.$$ Plugging this minimum back into the energy (and normalizing to get a number between 0 and 1) gives you a measure of how bad the fit is, $$E=\frac{w_1 w_2 (x_1-x_2)^2}{4(w_1+w_2)^2}.$$ $E$ has some of the common-sense properties you want: $E$ is higher if two people of equal experience disagree than if two people of disparate experience disagree, and for fixed values of experience, increases as the disagreement in ratings increases.

share|improve this answer
    
Hm, did you just change the formula? I remember seeing a '25' in there? –  yossarian Nov 9 '11 at 12:23
    
Yes, there was an error originally; I thought the ratings were 0-5 instead of 1-5 which changes the scaling a bit. –  user7530 Nov 9 '11 at 12:33
    
Ok, sorry to be of trouble again, but doesn't w1 = 5, x1 = 5, w2 = 5, x2 = 1 give E = 0 now? (which would be odd) –  yossarian Nov 9 '11 at 12:35
    
The only way for $E=0$ is when $x_1 = x_2$. For your numbers I get $E=1$, the maximum possible. –  user7530 Nov 9 '11 at 12:37
    
My bad :) You are right –  yossarian Nov 9 '11 at 12:40

I'd go for a weighted mean square error or maybe weighted mean absolute error. YMMV.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.