I'm working on a computer program but I need help coming up with a formula. This program has list of customers that wrote reviews for products. For each review written by this customers there is a number of votes and also a number indicating how many of these votes marked the review as helpful (think of Amazon).

And I would like to come up with a shorter list of expert reviewers. The way I envisioned my solution was to assign weights to each customers based on the votes given to the reviews, so I thought a ratio between the number of votes and number of those votes marked as "helpful" could be meaningful.

However, I ran into instances in which one reviewer had 1 vote and that one vote was marked as helpful, thus the ratio = 1/1 =1

And there are some other instances in which a customer had 19 votes and all of those votes were marked as helpful. that is 19/19 = 1

It's evident that mathematically they are similar; however I would like to assign a bigger weight to the customer who got 19 votes versus only 1. Similarly, I believe a review that got 24 helpful votes out of 25 (24/25 = .96) should have a bigger weight than the customer who got 1 vote which was marked as helpful.

Do you have any thoughts on how I could come up with a formula that takes into consideration the ratio and also the number of reviews?

Thank you!

-
I appreciate everybody's help. The answer is indeed subjective and the solutions proposed are a great start. Another consideration is that records with few reviews (1/1) should probably be filtered out. – Ulises May 6 '11 at 4:02

You need to define carefully what you want, then you can certainly make a formula to do that. For example, maybe unhelpful votes are rare (as they are on this site), and you feel they should give a big penalty. You could have a score of helpful-10*unhelpful. This will make 19/19 beat 1/1 badly, and even 24/25 will beat 1/1, but 8/9 will lose to 1/1. Is that what you want?

Maybe a good way to think about it is what (n-1)/n score should rank even with 1/1? With 2/2? What (n-2)/n score should be even with 1/1? If you get a number of these, a formula can be constructed.

-

The simplest way is to add some numbers to the numerator and denominator: for example if you added $5$ and $10$ then you would get the reviewers ranked as follows:

Original Adjusted Decimal
24/25     29/35    0.829
19/19     24/29    0.828
1/1       6/10    0.600


or you could choose other adjustments.

-

Since this is pretty subjective, what you are looking for is a way to quantify your subjective opinion about what qualifies as a good reviewer and what qualifies as a bad one. In that case, I suggest you try to visualize it by drawing a heat map.

Make a graphic with on the horizontal axis the number of reviews and on the vertical axis the number of helpful reviews. Since there can't be more helpful reviews than there are reviews, only the half of the quadrant below the line $y=x$ will be colored. Then, you colour all the points on the graphic for which you want to attribute the same weight. For instance, if all the people with the same number of helpful reviews have to receive the same weight, you give them the same color. If all the people with the same ratio of helpful to total reviews need to get the same weight, you give them the same color. If not, you find another color scheme. The colors is just to make it more pictorial, it will in the end of course correspond to the weights.

Once you have done that, you can either directly work with this heat map to associate weights, or you can look for the function of reviews and helpful reviews that best approximates your heat map.

-