I'm working on a review site and I need to create a scoring algorithm to give each product a score, similar to trustpilots trustscore, out of 10 based on a number of factors. The factors I need to take into account are:

  • The star ratings between 1 and 5 left on the product
  • The age of the star ratings (ratings should hold less weight the older they are)
  • The type of rating (verified reviews should have higher scoring)
  • A bayesian average should be used to prevent bias for low vote counts

I'm by no means a mathematician so I really don't know where to start with this, so if anyone could offer any guidance or examples, that would be great.

Update with answers to some questions

  1. A verified review should score twice as high as an unverified review
  2. A product with a single 5 star review on it's own would have a score of 5, however as per the last constraint, an initial average should be applied to all reviews to prevent this bias for new products. I would do this by adding the score of 7 additional reviews to the calculation with an average rating (so on the scale of 1 to 5, would be a rating of 3) so everyone essentially starts "average".
  3. For the age factor, I see the worth of a score diminishing by say 10% per year (would be handy to be pluggable, ie change to 15% if we found this was too slow).
  4. The score will get recalculated when a new review is left so based on that reviews date (but if this could be plug-able to change to a recalculate per day system, that would be handy)
  5. Old verified reviews should have more worth than a new unverified review, but should probably be affected by the age sliding scale so eventually there will come a tipping point where a new unverified review will weigh more than the old verified review because it's so old.
  • $\begingroup$ Can you clarify your last constraint? In what sense does a straight average show "bias for low vote counts"? $\endgroup$ – lulu Jul 25 '17 at 13:50
  • $\begingroup$ In that, a product with a single 5 star review should not out rank a product consisting of multiple 4/5 star reviews. The way trustpilot do this is by adding 7 additional reviews with a 1 to 10 score of 7 to the score to balance this. $\endgroup$ – Matt Brailsford Jul 25 '17 at 13:55
  • $\begingroup$ Well, ok. I'd first ignore that one (as it is the only inter-product constraint) and focus on the other. Toward that end, I'd describe some benchmarks. How much do you want the aged votes to diminish over time? Some geometric decay maybe? How much should verification matter (probably a straight multiplier). Then do the same for your inter-product issue. $\endgroup$ – lulu Jul 25 '17 at 14:02
  • $\begingroup$ I do think the later restraint still has relevance at this stage, but I've added some notes to the original question to hopefully help set some initial benchmarking values. $\endgroup$ – Matt Brailsford Jul 25 '17 at 14:15
  • $\begingroup$ I don't understand your seven extra votes. Why switch from $1-5$ to $1-10$ for the extra votes? When you say you add them at the average rating do you mean the average rating of all the other products out there or do you mean the flat number $5$? In either case, why do you say this yields a score of $7$? $\endgroup$ – lulu Jul 25 '17 at 14:23

I think this might be business logic more than mathematics/stats. You can absolutely give everything a number and weight and then average it, but I doubt you would get what you need straight away.

I'd start by thinking about the edge cases - what score should a product with only one 5 star review get? what about a single 1 star review?

Then think about your weights - is the age of the review only important in comparison to the other reviews of the same product, or in comparison to [today]? As in, should a review automatically age, or only age as new reviews are added?

Also, how much more important are verified reviews compared to unverified ones?

After all that, we would then need to work out the comparitive importance of each factor (does an old verified review trump a new unverified one, what about a v.old one, etc), and then we can start making a fun algorithm :)

  • $\begingroup$ Would it be best to answer you questions in the comments? or an amendment to the initial q? $\endgroup$ – Matt Brailsford Jul 25 '17 at 14:01
  • $\begingroup$ Added some notes to the original question. $\endgroup$ – Matt Brailsford Jul 25 '17 at 14:14
  • $\begingroup$ Because I don't have enough rep to comment above: (2) - a product with a single 5 star review would score 5 (out of 10), but then we add additional (fake) reviews with a score of 7 - we ae classed 7 as "average" here? $\endgroup$ – GeeksAreForLife Jul 25 '17 at 14:19
  • $\begingroup$ Sorry, I explained that badly, I've updated the example. It should be equivalent of adding 7 additional reviews with a star rating of 3, ie, average. $\endgroup$ – Matt Brailsford Jul 25 '17 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.