Consider a story ranking website in which the ranking is crowd sourced from the number of up votes and down votes received by a story.

The score is computed as the lower bound Wilson's algorithm.

lower bound = $ (p+\frac{z^2_{\alpha/2}}{2}\pm z_{\alpha/2}\sqrt{[p(1-p)+(z^2_{\alpha/2}/4n)]/n}) / (1+\frac{z^2_{\alpha/2}}{n})$

where $p$ is the observed fraction of positive ratings, $n$ is the total ratings, and $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of standard normal distribution.

Now consider two stories. One story receives votes from all the voters and the other story receives $X$ times (0<$X$<1) the votes as the first story. Wilsons algo. calculates the score $S$ the second story such that "IF" the second story had as many votes as the first story then you could say with a 95% confidence that the score is $S$.

What is the minimum $X$ for which the score remains within a $\pm i$ interval?

ref. http://evanmiller.org/how-not-to-sort-by-average-rating.html

This question is kind of similar to https://mathoverflow.net/questions/6019/calculating-the-most-helpful-review

  • $\begingroup$ You are being contradictory in saying that every person votes on every story yet latter saying stories may not have the same number of votes. Which is it? $\endgroup$
    – Dale M
    Mar 25, 2013 at 0:38
  • $\begingroup$ For the second vote used to calculate ranking $r_2$ the stories may not have the same number of votes. $\endgroup$
    – Manas
    Mar 25, 2013 at 16:15
  • $\begingroup$ Are you asking how many votes to change a given stories score or how many to swap the positions of 2 stories? Or n stories? $\endgroup$
    – Dale M
    Mar 25, 2013 at 19:45
  • $\begingroup$ How many to swap the positions. Let me rephrase the question if it helps. Lets say I have a story with 20 up and 5 down votes from a total of 25 people. Using Wilsons algo. I can calculate its "score" as the lower interval bound with a 95% confidence using the above formula. Let the score be 50. What this means is that given the votes the idea received, you can be 95% sure that the real score is 50. Now my question is, what is the minimum number of votes (from 25) required so that I can be 95% sure that the score is still 50. Doing this to all ideas will ensure ranking remains unchanged. $\endgroup$
    – Manas
    Mar 25, 2013 at 20:49
  • $\begingroup$ Please check edited question. Thanks. $\endgroup$
    – Manas
    Mar 25, 2013 at 22:28

1 Answer 1


OK @Manas, before we start, you seem to have a misguided understanding of what Wilson's algorithm does.

If you consider everybody in the world (all 7 billion of us) there is a number of people who would rate this story positively and a number who would rate it negatively. This is (at any given time) an exact number, call it the population rating $r_p$.

What we have is a sample of that population who have actually rated the story and we have another exact number which is how this sample rated the story, call this the sample rating $r_s$. This is our best available information as to $r_p$.

Wilson's algorithm allows us to build a confidence interval (to any level of precision 90%, 95% and 99% being typical) of where the value $r_p$ actually lies. Naturally, the larger the sample size, the tighter the bounds on this interval become.

For the purposes of ranking stories, using the lower bound of Wilson's algorithm allows you to say "With $(1-\frac{\alpha}{2})$% confidence I believe that the value of $r_p$ for this story is at least $ (p+\frac{z^2_{\alpha/2}}{2}\pm z_{\alpha/2}\sqrt{[p(1-p)+(z^2_{\alpha/2}/4n)]/n}) / (1+\frac{z^2_{\alpha/2}}{n})$"

This rating is solely dependent on the side of the sample, the up vs down votes and the confidence interval you select. In particular it has nothing to do with the ratings of other storys.

To the specific question you have asked.

You are interested in knowing for any given $p$, $n$ and $z_{\alpha/2}$ (which I will call $z$ to simplify the appearance of the equations), how future votes will affect the rating. Clearly, this depends not only on the number of additional votes cast $\Delta n$ but also on what proportion of those votes are positive, $\Delta p$.

Please note that the new value of $p$ is given by:

$$p_1=\frac{pn+\Delta p\Delta n}{n + \Delta n}$$

I suggest you consider 4 cases:

$$\Delta p = 0\qquad\ldots(1)$$ $$\Delta p = 0.5\qquad\ldots(2)$$ $$\Delta p = p\qquad\ldots(3)$$ $$\Delta p = 1\qquad\ldots(4)$$

For each of the cases, you simply need to solve the following for $\Delta n$

$$\Bigg |{p_1+\frac{{z^2}}{2}- z\sqrt{p_1(1-p_1)+\frac{z^2}{4(n+\Delta n)}\over n+\Delta n} \over {1+\frac{z^2}{n+\Delta n}}}-{p+\frac{{z^2}}{2}- z\sqrt{p(1-p)+\frac{z^2}{4n}\over n} \over {1+\frac{z^2}{n}}}\Bigg |= i$$

Note that for cases (3) & (4) the term on the left is always positive so you only need to consider $+i$; this is also true for case (2) iff $0.5>p$.

Good luck.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .