# How many chess games of a suspect does one have to analyze to have a reliable answer to the question whether the suspect cheats?

This is a practical question that was asked on a correspondence chess site. The site struggles with chess engine users, who are considered cheaters. The method people use to determine whether someone is a cheater is this. Take a large enough number $x$ of games the person has played; analyze them with an engine (starting from the point where the game leaves online chess databases, whose use is allowed) and find the average match-up rates, for example:

• 1st choice - 50% match-up rate,
• 2nd choice - 65% match-up rate,
• 3rd choice - 80% match-up rate,

where the first number is the ratio $$\frac{\text{the number of analyzed moves that are considered best by the engine}}{\text{the number of analyzed moves}},$$

the second number is the ratio $$\frac{\text{the number of analyzed moves that are considered best or second best by the engine}}{\text{the number of analyzed moves}},$$

and the third number is found in a similar way.

The question is how large $x$ must be in order for these numbers to approximate the overall match-up rates well.

I think the right way to ask this question more precisely is to fix a small error $\varepsilon$ and a small probability $p$, and ask how large $x$ must be for the probability of the actual error exceeding $\varepsilon$ to be less than $p$.

I did do a course in statistics, but I remember almost nothing of it. Could you help me with this?

-
Your question is not entirely clear. Is it correct that "analyzed moves" refers only to moves that the suspected cheater has actually played, while "considered best by the engine" means the unique move found best by the engine among the options at that point in play? So as an extreme case, if the cheater just was using the engine blindly, the first number would be 100% (and the other a fortiori also). The idea is that if a player is doing very well (by the engine's standards), then it must be a cheater. At least in cycling this appears to be fairly sound reasoning. – Marc van Leeuwen Nov 16 '12 at 14:32
Don't you get into trouble when there is a long sequence of exchanges? Then every move is obvious to a human and computer: would they be flagged as cheaters? – Fixee Nov 16 '12 at 15:09
@Marc Yes, that's what I meant. – Bartek Nov 17 '12 at 13:03
@Fixee Well, the idea is to analyze many games, not just one. The match-up rates are hoped to become more and more meaningful as the number of analyzed games grows. – Bartek Nov 17 '12 at 13:06