# What are good ways to score an ordering?

For context, a friend hosts a pub trivia night and would like to know a good way to score ranking questions. For example, put these five movies from the 70's in order of release: Jaws, Star Wars, Alien, Blazing Saddles, and Superman. There is a correct ordering, but how should partial credit be allotted to incorrect orderings?

Somewhat more formally, suppose that you put out a question asking participants to order $N$ things. What are good ways to grade their responses with partial credit?

For example, suppose $N=5$ and participants are asked to put B, C, E, A, and D in order. Full credit (whatever that means) is given for ABCDE. But what's a good way to give partial credit for other permutations?

For example, I think EDCBA should get no credit at all, since it's completely backwards (despite having one letter in the right position). I think ACBDE should get the highest possible partial credit (whatever that would be), since it is only one transposition away from correct. (Personally I'd also factor in that the transposition is in the middle, so at least the participant got the extremal letters correct. However, that turns this into a non-symmetric problem, and I'm happy to throw out that consideration.) I think BCDEA should get a fair amount of partial credit, since the relative ordering of 4 out of 5 items is correct, despite having no letters in the correct position.

I'm aware of a several reasonable-sounding methods to assign partial credit. Each method measures how "far away" the submitted answer is to the correct answer, and then uses some function to turn that distance into a score in a negative manner (the more the distance the less the score.)

Possibilities for measuring distance:

$\bullet$ Count the minimal number of adjacent transpositions that turn the submitted answer into the correct answer.

$\bullet$ Create an $N$-vector measuring the distance from each letter's submitted position to its correct position, and put a notion of size on that vector. For example DCABE would have vector $(3,1,2,2,0)$. Size could be the standard Euclidean distance, the square of Euclidean distance, the taxicab metric...

$\bullet$ Other ideas?

Then there are lots of reasonable functions that turn distance into a score.

$\bullet$ $\mbox{score(distance)}=100\%\cdot r^{\mbox{distance}}$, for some $r<1$.

$\bullet$ $\mbox{score(distance)}=100\%-k\%\cdot\mbox{distance}$, where $k\%\cdot\mbox{(maximal distance)}=100\%$.

$\bullet$ $\mbox{score(distance)}=\displaystyle\frac{100\%}{1+\mbox{distance}}$

$\bullet$ Other ideas?

What I am looking for is more than just new and different proposals for scoring (although I'd love to hear more of those too). I am more interested in reading a convincing argument for preferring a particular method of scoring.

-
"minimal number of adjacent transpositions" is occasionally called the $\it disorder$ of a permutation. –  Gerry Myerson May 23 '11 at 7:39

It depends on your point of view and the questions:

Regard the permutation BCDEA

For example, if the question is about historical events, and someone decides that the pyramids where built after the Empire State Building, then you might not want to give much credit, because it displays complete ignorance. The distance of each letter to each correct place punishes this error accordingly.

But the ordering BCDEA for the movies above could just mean that the participant did not know the movie A while being perfectly aware of the four other movies which is quite a lot of knowledge about movies released before their birth. So, you might want to base your score on the question: How many letters have to be removed to arrive at a correct ordering?

-

If I was scoring this by computer, then I would look at the $N(N-1)/2$ pairs (10 when $N=5$) and give a point (or $\frac{2}{N(N-1)}$ of a full credit) for each correct pair. So ABCDE gets 100%; ABCED gets 90%; BCDEA gets 60%; and EDCBA gets 0%.

In a pub quiz, this is too complicated. You could just give bonus points if the full order is correct and nothing if there is a single error.

-

You should take a look at the Kendall Tau Distance and the Kendall Tau Rank Correlation Coefficient.

-