Let's say I have a website where people can rate movies on a scale of 0-10. We say people vote honestly when the rating they give to a movie is what they actually think. People vote dishonestly when they try to get the website's rating to be closer to their own rating: for example, if a movie's current average rating is 7.2, but I believe it deserves an 8, I will rate it a 10 to bump the average higher. (Everyone who votes dishonestly always gives a 0 or a 10.)

It's clear that if everyone votes honestly, the website's rating will reflect the true average viewer rating of the movie. My question is: if everyone votes dishonestly, will the website's rating still reflect the true average viewer rating? In other words, does it matter?

Hypothesis: It doesn't matter. Presumably the order of who votes when affects things (when everyone votes dishonestly), but when the number of voters $N$ gets large enough, it should converge.

Follow up questions:

  • If it does make a difference, why?

  • If it doesn't make a difference, how large of an $N$ do we need before we start to see convergence with the true average? Is that only true if the true ratings follow a certain distribution (uniform vs. normal vs. two-humped, etc.)?

  • What if it's actually a mix: some people vote honestly, and some vote dishonestly?

  • 2
    $\begingroup$ If you set up your website so that people could (and did) change their votes, then an all-honest score would give the mean rating and an all-dishonest score would give the median rating. I don't know how not being able to change your vote would affect things, but I would look for the same affect. $\endgroup$ – Matthew Daly Jan 4 at 14:36
  • $\begingroup$ If there is an $\infty$ of dishonest votes, and each voter votes $10$ if $rating < whattheywant$ and $0$ otherwise, the limit should indeed $\to$ the rating of all people voting honestly. $\endgroup$ – Ring Ø Jan 4 at 14:41
  • $\begingroup$ Do users know the current number of votes? Otherwise, they cannot know which strategy will get the score closer to their opinion. $\endgroup$ – cangrejo Jan 4 at 15:07
  • $\begingroup$ @broncoAbierto I would think for a large enough number of votes the dishonest strategy is always more efficient, and we're talking about asymptotics here so we don't need to worry about that so much. $\endgroup$ – Jack M Jan 4 at 15:18

When the mean is closer to one end of the scale, one side of the voting has more to gain than the other from being dishonest, so it's going to bias the voting.

Say that half the voters think that the movie is a 7 out of 10, and half the voters think that the movie is a 9 out of 10. The honest average rating would be 8.

Voters that want the rating to be 7 have great power to influence the rating when it's above 7: they can vote 0 instead of 7. Voters that want the rating to be 9 have very little power to influence the rating when it's below 9: they can vote 10 instead of 9. Because of this, even if the voters are randomly ordered, the long-term average is going to stabilize around 7. (I've tried this out in simulations.)

Here's why. There are three situations the average can be in: it can be in the ranges $[0,7]$, $[7,9]$, and $[9,10]$.

  • In the first range $[0,7]$, all voters vote $10$. The average vote is $10$, so the vote trends up to $7$, possibly moving to the second range.
  • In the second range $[7,9]$, half the voters vote $0$ and half the voters vote $10$. The average vote is $5$, so the vote trends down to $7$, possibly moving to the first range.
  • In the third range $[9,10]$, all the voters vote $0$. The average vote is $0$, so the vote trends down to $9$, moving to the second range.

Eventually, we're going to be alternating between the first and second ranges, and the vote will hover around $7$.

In general, this argument tells us that under dishonest voting, the long-term average rating occurs where the average value of a dishonest vote equals the current average. This happens at the intersection of the blue and orange curves in the graphs below; the blue curve (the average value of a dishonest vote) is derived from the ratings of the users, and the orange curve is just the line $y=x$.

(In these graphs, I'm switching to a $1$-to-$10$ rating system rather than $0$-to-$10$ to match real data.)

enter image description hereenter image description here

The first image is the example I gave above. The blue and orange curves intersect when $x=7$.

The second image is based on data taken from https://www.imdb.com/title/tt7286456/ratings. The blue and orange curves intersect when $x=8$. Note that the actual mean is $8.8$ and median is $9$, but under dishonest voting, the few voters that think that the rating should be $8$ or less have more of an impact.

  • 1
    $\begingroup$ I think your result is the same as the somewhat more intuitive - if we rescale ratings to $[0,1]$ - "A dishonest rating of $p$ means that the proportion of voters who believe the film deserves a rating of at least $p$ is at least $p$ and the proportion who believe the film deserves a rating of at most $p$ is at least $1-p$" - so it's some weird quantile where the quantile-like metric where the percentile depends on the end result. $\endgroup$ – Milo Brandt Jan 4 at 23:50
  • $\begingroup$ @MiloBrandt That sounds equivalent. I was going for something visual. $\endgroup$ – Misha Lavrov Jan 5 at 0:41
  • $\begingroup$ @Milo: Something a bit like the h-index, then: "the largest $p$ such that at least $p$ proportion of voters believe the film deserves a rating of at least $p$." $\endgroup$ – Rahul Jan 5 at 5:42
  • $\begingroup$ This is a great answer. In addition to this I think the standard deviation of the voting responses in case of the dishonest voters will be higher than the sd of the preferences. This is intuitive from the explanation provided from the above answer I think. For a small simulation thingy: github.com/HariharanJayashankar/strategic_voting/blob/master/… $\endgroup$ – Hariharan Jan 8 at 7:07

Here's an example that dishonest voting can lead to a result that departs from the true average.

Suppose half the people rate a (bad) movie $0$ and the other half rate it $2$, so the true average rating is $1$. Let's say there are $100$ voters in each camp.

Suppose the $0$-rating camp votes first. Then the first $25$ $2$-raters will give it a dishonest $10$, which brings its average rating up to $(100\cdot0+25\cdot10)/(100+25)=250/125=2$, and after that the rest of the $2$-raters will give it an honest $2$, leaving the rating at $2$, instead of the true average $1$.

This example, of course, is rather artificial, especially in its assumption regarding the order in which votes are cast. But it suggests, to me at least, that dishonest voting is unlikely to come with any guarantee of arriving at the true average. It might be worth asking how close it comes, on some average (such as averaging over the different orderings of the voters), to the true average.

  • 1
    $\begingroup$ This is extrême (and in the dishonest scenario after the 25 vote 10, the others should only vote 0 or 10 to finally reach $\approx 2$), but that shows order matters $^{+1}$. Statistically, if voting is more evenly distributed, the result should not be far from $1$. $\endgroup$ – Ring Ø Jan 4 at 15:38
  • $\begingroup$ This is a good example, buy I think for the sake of generality let's assume voters vote in a random order. $\endgroup$ – jamaicanworm Jan 4 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.