A colleague is trying to determine whether there's a statistically significant difference between the % turnout values for two separate elections. The elections have entirely different candidates and entirely different voters. None of the voters in election A can vote for candidates in election B, and vice versa. The assumption is that each voter decides whether to vote, and votes, independently of the others.

Election A
Eligible voters: 21499
Number of individuals who voted: 2121
Turnout: 9.9%

Election B
Eligible voters: 1373
Number of individuals who voted: 278
Turnout: 20.2%

I would be grateful if someone could say a) what statistical test should be used, b) whether there is a statistically significant difference and c) the 𝑝-value, where the null hypothesis is that the two turnout percentages are the same and 𝑝 is the probability that the null hypothesis is true.

My colleague also wants to compare the turnout % in Election A with that in Election C:
Eligible voters: 2245
Number of individuals who voted: 483
Turnout: 21.5%

I have been asked to provide some context to the question. It is important because the voters in the three elections are in fact all members of the same organisation, but can only vote for candidates in their own countries. The elections A, B and C pertain to different countries. My colleague raised the question as to whether the differences between the countries in turnout % are statistically significant. Whilst it seems evident to me that there is a clear difference, this may not be sufficient for some. The question is important because it potentially reflects different engagement levels of voters. The reasons for that may be complex, but it may help if the question of statistical significance can be addressed to the extent that it need not form a material part of the discussions.

  • $\begingroup$ You know how many votes were given among all voters, so they are indeed different. Also having the whole populations as the sample makes several tests wrong to use because they assume that the sample is much smaller than the population size. For statistical methods to apply you should only know whether or not some random subset of each population voted. $\endgroup$ – Coolwater May 19 at 18:38
  • 2
    $\begingroup$ You omitted the most important part: what is your mathematical model of voting? Are you assuming that each voter comes to the election independently of others with probability $p$ depending on the election and are you asking whether the hypothesis that $p$ is the same for both elections should be rejected. Or do you have a completely different setup in mind? Statistics starts spitting out reasonable answers only if you have precise competing mathematical models of what is going on. Otherwise such questions are simply meaningless. $\endgroup$ – fedja May 19 at 18:51
  • $\begingroup$ Thank you fedja, have added detail to the question. In short, the model you described is the one I had in mind. $\endgroup$ – whispersan May 19 at 18:56
  • $\begingroup$ Welcome to Mathematics Stack Exchange! A quick tour will enhance your experience. Here are helpful tips to write a good question and write a good answer. For equations, use MathJax. $\endgroup$ – dantopa May 20 at 0:55
  • 1
    $\begingroup$ The findings be pretty much the same unless you have some reason to believe that the standard deviations are much larger than the Bernoulli model predicts (you have no way to directly estimate them empirically from a single observation). So, unless you meet some well-grounded criticism of the Bernoulli model, I would stick to it. It may look difficult to you not because of computations (we had next to none) but just because the general ideas may be hard to absorb. However, they don't change with the model, so it makes sense to try to understand them anyway. Feel free to ask questions, BTW! $\endgroup$ – fedja May 20 at 11:50

I'll start with an overview of all relevant statistical ideas.

Suppose that you have two independent elections with $M$ and $N$ eligible voters. Suppose also that there are associated probabilities $p$ and $q$ such that each eligible voter attends the first election with probability $p$ and the second election with probability $q$ if somebody is eligible to vote in both, the corresponding choices are also independent). Note that it is not very realistic model for many reasons and the normal distribution model for the turnout is better and much easier to handle as long as the turnout is neither too low, nor too high, but let's stick to Bernoulli.

One thing to understand is that the test is merely a partition of the set of all possible outcomes $(m,n)\in[M]\times[N]$ into two parts $E$ and $E^c$. This partition should be chosen before you get the real election attendance data, so telling us the actual turnovers was a mistake. Let's pretend you didn't do it. You reject the hypothesis $p=q$ if the observed $(m,n)\notin E$ and accept it (or, rather, grudgingly admit that you do not have enough evidence to reject it = "the difference in observations is not statistically significant") if $(m,n)\in E$.

You can declare that level of confidence of your rejection is at least $1-\alpha$ (usually $\alpha=0.05=5\%$, but you can choose any value you want) if for every Bernoulli model with $p=q\in[0,1]$, the probability that $(m,n)\in E$ is at least $1-\alpha$, i.e. that $E^c$ is an universal event of small probability ($P(E^c)\le\alpha$). Note that some choices of $E$ are better than others but usually there is no choice that is the best in all respects unless you test a simple hypothesis against a simple alternative (in your case neither the hypothesis, nor the alternative is simple). Also note that different tests mistake on different samples, so the term "statistically significant" is in general test-dependent.

To convince somebody that you can reject the hypothesis with confidence level $1-\alpha$, you have to

a) present him with a set $E$ that is chosen without any reference to the actual results (the best thing is to declare it before the elections, but if it is too late, explaining why the choice is "natural" may work to a certain extent, though when multiple standard tests are available, the common way of cheating is to try them all and to present only the one in your favor)

b) present a proof that $P(E^c)<\alpha$ for every Bernoulli law with $p=q$.

c) show that the observed $(m,n)\in E^c$ (the set $E$ can be rather complicated, so it may also require an argument or a computation).

Quite often, the set $E$ is chosen as a level set of some non-negative function $f(m,n)$ ($E=\{(m,n):f(m,n)\le T\}$). Then we are interested in the tail probabilities $\sup_{p=q\in[0,1]}P(f>T)\le \alpha(T)$ which makes the application of the technique trivial: you just compute $T=f(m,n)$ and compare $\alpha(T)$ with your prescribed $\alpha$. The big work is getting $\alpha(T)$ (or the inverse function $T(\alpha)$. That is what you see in various tables.

Let's now design a statistical test from scratch. It won't be the most powerful one, but it will give you an idea of what is going on. We shall choose the most obvious function $f(m,n)=|\frac mM-\frac nN|$. We need to estimate the probability that $f>T$. To this end, split it into the probability that $\frac mM-\frac nN> T$ and the probability that $\frac mM-\frac nN< -T$. We'll get the same bound for each.

Recall the Hoeffding Lemma It gives the estimate $$ P(\frac mM-\frac nN\ge T)\le e^{-xT}Ee^{x(\frac mM-\frac nN)}\le e^{-xT}e^{\frac{x^2}8(\frac 1M+\frac 1N)} $$ for any $x>0$ regardless of the particular probability $p=q$. Also note that is it a clean estimate without any approximation, etc., so you may be sure that the result it gives is completely reliable, albeit suboptimal. Now the best choice of $x$, given $T$ is $x=\frac {4T}{M^{-1}+N^{-1}}$ yielding the bound $$ P(f(m,n)>T)\le 2\exp\left[-\frac {2T^2}{M^{-1}+N^{-1}}\right]=\alpha(T). $$ (The factor of $2$ appears because we have exactly the same bound for the second case).

Now back to elections. When comparing the first two using this test, we get $T=0.103$, $M=21499$, $N=1373$ whence, after plugging everything into the calculator, $\alpha(T)\approx 2.56\cdot 10^{-12}$, which is way below the customary $5\cdot 10^{-2}$, so our test allows you to declare the difference in turnouts between A and B statistically significant and blah-blah-blah.

Comparing A and C gives pretty much the same result. If you compare B and C, however, you get $T\approx 0.0126$ and $\alpha(T)\approx 1.52$ (remember that $\alpha(T)$ is just an upper bound for the actual probability, not the probability itself, so its being greater than $1$ is harmless but, of course, with such value you have no grounds for the claim that B and C are different in your sense. That is how it works. If you cannot find a standard test in the literature or by asking your statistician friends, you can always design your own. Remember that the design of good tests (i.e., the choice of $E$) is half-science and half-art, but if your homemade concoction works fairly well (like it was in our case), you do not need to look any further. Any criticism of the kind that "it doesn't look professional to use homemade tools" can be met with "But I know what I'm doing and am getting the results I can be confident about and can explain to anybody".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.