# Objective metric to describe skill vs luck in games that include randomness

There are many games that even though they include some random component (for example dice rolls or dealing of cards) they are skill games. In other words, there is definite skill in how one can play the game taking into account the random component. Think of backgammon or poker as good examples of such games.

Moreover, novice players or outsiders might fail to recognise the skill involved and attribute wins purely to luck. As someone gains experience, they usually start to appreciate the skill involved, and concede that there is more to the game than just luck. They might even concede that there is "more" skill than luck. How do we quantify this? How "much" luck vs skill? People can have very subjective feelings about this balance. Recently, I was reading someone arguing that backgammon is $9/10$ luck, while another one was saying it's $6/10$. These numbers mean very little other than expressing gut feelings.

Can we do better? Can we have an objective metric to give us a good sense of the skill vs luck component of a game.

I was thinking along these lines: Given a game and a lot of empirical data on matches between players with different skills, a metric could be:

How many games on the average do we need to have an overall win (positive win-loss balance) with probability $P$ (let's use $P=0.95$) between a top level player and a novice?

For the game of chess this metric would be $1$ (or very close to $1$). For the game of scissors-paper-rock it would be $\infty$.

This is an objective measure (we can calculate it based on empirical data) and it is intuitive. There is however an ambiguity in what top-level and novice players mean. Empirical data alone does not suffice to classify the players as novices or experts. For example, imagine that we have the results from 10,000 chess games between 20 chess grandmasters. Some will be better, some will be worse, but analysing the data with the criterion I defined, we will conclude that chess has a certain (significant) element of luck. Can we make this more robust? Also, given a set of empirical data (match outcomes) how do we know we have enough data?

What other properties do we want to include? Maybe a rating between $[0, 1]$, zero meaning no luck, and one meaning all luck, would be easier to talk about.

I am happy to hear completely different approaches too.

• A boring simple answer: the more money that you walk away with, the better you are at poker. Similarly, if you win all matches of a reasonable length (e.g. 9 points) then you are the better backgammon player. Jul 12, 2017 at 14:39
• @badjohn I am not asking for a way to rank players. An Elo ranking system can do the job there. I am interested in a metric that shows how "much" skill vs luck does a game have. My title might be misleading, I'll update it. Jul 12, 2017 at 14:48
• Yes but if one player consistently beats another then that is evidence that there is an element of skill. You should be able to find a statistical test to compare the result against the null hypothesis of there is no skill. Jul 12, 2017 at 17:15
• Here's a video that explain the basic analysis of the book "The Success Equation". youtube.com/watch?v=HNlgISa9Giw From what I can tell, it suffers from the same problem I describe in the question. It only considers top-level players and thus the luck vs skill metric is not really indicative of the game as a whole. Feb 20, 2018 at 0:26

## 3 Answers

A natural place to start, depending upon your mathematical background might be the modern cardinal generalizations of the voting literature. Consider the following toy model, and feel free to adapt any parts which don't seem appropriate to the problem at hand.

Data: Say we have a fixed, finite population of players $N$ who we observe play. A single observation consists of the result of a single match between two players $\{i \succ j\}$ for some $i,j \in N$. Let's the space of datasets $\mathcal{D}$ consists, for fixed $N$, of all finite collections of such observations.

Solution Concept: Our goal here is twofold: mathematically we're going to try to get a vector in $\mathbb{R}^N$ whose $i^\textrm{th}$ component is the 'measure of player $i$'s caliber relative to all the others.' A player $i$ is better than player $j$ by our rule if the $i^\textrm{th}$ component of this score vector is higher than the $j^\textrm{th}$. We might also like the magnitude of the differences in these components to carry be increasing in some measure of pairwise skill difference (the score vector is cardinal, to use economics lingo).

Edit: This score vector should not be seen as a measure of luck versus skill. But what we will be able to do is use this to 'clean' our data of the component of it generated by differences in skill, leaving behind a residual which we may then interpret as 'the component of the results attributable to luck.' In particular, this 'luck component' lives in a normed space, and hence comes with a natural means of measuring its magnitude, which seems to be what you are after.

Our approach is going to use a cardinal generalization of something known as the Borda count from voting theory.

Aggregation: Our first step is to aggregate our dataset. Given a dataset, consider the following $N\times N$ matrix 'encoding' it. For all $i,j \in N$ define:

$$D_{ij} = \frac{\textrm{Number of times i has won over j}- \textrm{Number of times j has won over i}}{\textrm{Number of times i and j have played}}$$

if the denominator is non-zero, and $0$ else. The ${ij}^\textrm{th}$ element of this matrix encodes the relative frequency with which $i$ has beaten $j$. Moreover, $D_{ij} = -D_{ij}$. Thus we have identified this dataset with a skew-symmetric, real-valued $N\times N$ matrix.

An equivalent way of viewing this data is as a flow on a graph whose vertices correspond to the $N$ players (every flow on a graph has a representation as a skew-symmetric matrix and vice-versa). In this language, a natural candidate for a score vector is a potential function: a function from the vertices to $\mathbb{R}$ (i.e. a vector in $\mathbb{R}^N$) such that the value of the flow on any edge is given by the its gradient. In other words, we ask if there exists some vector $s \in \mathbb{R}^N$ such that, for all $i,j \in N$:

$$D_{ij} = s_j - s_i.$$

This would very naturally be able to be perceived as a metric of 'talent' given the dataset. If such a vector existed, it would denote that differences in skill could 'explain away all variation in the data.' Generally, however, for a given aggregate data matrix, such a potential function generally does not exist (as we would hope, in line with our interpretation).

Edit: It should be noted that the way we are aggregating the data (counting relative wins) will generally preclude such a function from existing, even when the game is 'totally determined by skill.' In such cases our $D$ matrix will take values exclusively in $\{-1, 0 ,1\}$. Following the approach outlined below, one will get out a score function which rationalizes this ordering but the residual will not necessarily be zero (but will generally take a specific form, of a $N$-cycle conservative flow that goes through each vertex). If one were to, say, make use of scores of games, the aggregated data would have a cardinal interpretation.

Construction of a score: The good news is that the mathematical tools exist to construct a scoring function that is, in a rigorous sense, the 'best fit for the data,' even if it is not a perfect match (think about how a data point cloud rarely falls perfectly on a line, but we nonetheless can find it instructive to find the line that is the best fit for it). Since this has been tagged a soft question, I'll just give a sketch of how and why such a score can be constructed. The space of all such aggregate data matrices actually admits a decomposition into the linear subspace of flows that admit a potential function, and its orthogonal complement. Formally, this is a combinatorial version of the Hodge decomposition from de Rham cohomology (but if those words mean nothing, don't worry about looking them up). Then, loosely speaking, in the spirit of classical regression, we can solve a least-squares minimization problem to orthogonally project our aggregate data matrix onto the linear subspace of flows that admit a potential function:

$$\min_{s \in \mathbb{R}^N} \|D - (-\textrm{grad}(s))\|^2$$ where $-\textrm{grad}(s)$ is an $N\times N$ matrix whose $ij^\textrm{th}$ element is $s_i - s_j$.

If you're interested in seeing this approach used to construct a ranking for college football teams, see:

http://www.ams.org/samplings/feature-column/fc-2012-12

If you'd like to read a bit more about the machinery underlying this including some brief reading about connections to the mathematical voting literature, see:

https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/ref/jiang-lim-yao-ye.pdf

• Thank you for your answer. I do not understand your proposal, partly because you use some terms that I do not know, partly because you probably need more words to explain it. In particular it would be great if you could explain how we go from flows to a single number. Also you seem to want to consider all possible flows, when intuitively, we should only be considering the flows of people that have the greatest expertise difference between them. Jul 12, 2017 at 5:08
• A small thing: You seem to be using $N$ both as a set of players, and the cardinality of the set. I think it's better to have different symbols. $N$ is usually the cardinality of a set. Jul 12, 2017 at 5:09
• @Thanassis I am, that's an unfortunate abuse of notation from economics. So critically, we never get from a flow to a single number, we get from a flow to a vector of numbers, one number per player in our dataset. These numbers have the followng interpretation: if the score if $i$ is higher than the score of $j$, our metric says $i$ is better than $j$ given the data it's been fed Jul 12, 2017 at 15:03
• I see. I thought you were comparing players, but was not sure. This is not what I was asking. I can see how I could be unclear (someone else in the comments made a similar remark), and hopefully the revised title is somewhat clearer. I am not interested in a ranking of the players. I am interested in a scalar metric that gives us a sense of how important is skill vs luck in the game. Jul 12, 2017 at 15:12
• @Thanassis The idea is to use an estimated ranking of players, given your dataset, as a means of cleaning your data to look at the 'residual' that can be attributed to luck. In a game of pure skill, there will be no random component, and the data matrix $D$ will take a very specific form. Moreover, so too will the residual of the 'regression' we are doing. The more room for luck, the 'noisier' and larger generally this residual will be. I agree, this is most certainly not a numeric criterion applicable for any game out-of-the-box, but it is one way of assessing the relative import of skill. Jul 12, 2017 at 15:16

Found this: How to Measure Luck vs Skill in Games? at
boardgames.stackexchange.com

The following links might also be helpful and/or of general interest:

The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing
$$\;$$ by Michael J. Mauboussin
NOTES ON THE SUCCESS EQUATION

Estimated true skill = Grand average + shrinkage factor (observed average − grand average)

Bayesian inference

I hope the OP can forgive me for not focusing on his questions. Ten years ago I might have been more enthusiastic, but now all I could muster up was to provide links that anyone interested in games should find interesting.

So what is my problem? Two letters:

# AI

Ten years ago it was thought that it would take AI 'bots' several decades to become #1 over humanity in two games:

Go (a game of perfect information)

OR

Poker (a game of imperfect information/randomness)

Today, it is a fait accompli.

The researchers matching their bots against human players in poker can claim victory whenever the out-performance is statistically significant.

GO is a different animal. Any win against a 10 dan professional player is 'off the charts' significant (throw that statistics book out the window).

• Thank you for the links. It would be great if you could expand the text to turn it into an answer. Why do you think these links are relevant? What are the main points? From a quick look it seems to me that only the Success Equation paper might contain some useful ideas (although it is not referring to games). The formula that you give does not make much sense without context, and the blogpost that includes it does not help much. Jul 12, 2017 at 5:19

In a similar vein to @Caradonna, one way would be an empirical study of a large, round-robin tournament. In primarily luck-based games, you would expect to see the wins relatively equally distributed. In more skill-based games, you would expect 'upset wins' to be infrequent.

For a tournament of n teams [A1, A2, A3 .... An], which each have a certain number of wins [W1, W2, W3 ... Wn] your 'skill score' would simply be the standard deviation of the number of wins. The maximum score would occur when A1 beats all teams (W1 = n-1), A2 beats all teams other than A1 (W2 = n-2), and so on to An, which loses all its games. In this situation, the score would be:

sqrt( ((n-1)/2 ^2 + (n-3)/2 ^2 + (n-5)/2 ^2 + ... + (n-(2n-1))/2 ^2)/(n-1) )

For large n, this boils down to approximately n/sqrt(12). For instance, with 100 teams, you'd score about 29.

On the other hand, even if the game was pure luck, you wouldn't expect each team to have exactly equal wins. You'd score about 4.8 for 100 teams.

If you wanted, you could scale this appropriately to get a score between 0 and 1.

One problem with this system would be some games that are basically pure skill (eg Chess) do still have upset wins, because human 'form' - their level of performance in any game or sequence of games - varies. Computer programs would be best to run this tournament, because they don't make mistakes.

A different way to tackle the problem is how hard is it for a human player to learn the game well - how many games does it take to become an 'expert'. The problem here is the subjectivity of expertise, and the variability of human learning curves. An alternative metric would be how many lines of code it takes to create the perfect computer program for a game. This seems more objective, but does depend on the type of programming language, etc.

Perhaps a more objective measure, which may become feasible in the future, would involve the use of general artificial intelligence. A computer is programmed not to play any specific game, but to learn games of strategy. The more it plays any game, the better it becomes at it (as it collects more data about what works and what doesn't). Your metric of how skill-based the game is, then, could be the probability of a computer which has played the game 10 times before, beating the same general artificial intelligence which has played the game 100 times before. (The 10 and 100 numbers are arbitrary, of course). This would be a score between 0 and 0.5, with closer to 0 indicating more skill.