# Can regression to the mean explain the unexpected underperformance (per player-rating) of girls against boys on chess?

I'm somewhat mathematically illiterate, I'd appreciate some help in order to satisfy a curiosity.

Concise summary attempt: chess ratings are derived from past performance and therefore thought to be good predictors of future performance, odds of the match according to the ratings of the opponent. Men are "on average", better chess players than women, but it would still be "predicted" that odds of a match between a man and a woman would be "dictated" only by their ratings. But a psychological finding was that girls underperformed in matches against boys. The hypothesized psychological explanation is that girls, being stereotypically worse than boys (and, in fact, worse on average), are excessively stressed, intimidated in such matches, and perform worse than when competing against other girls. A commenter on a blog suggested that this explanation isn't needed, that the finding can be explained just by regression towards the mean. This explanation doesn't seem to make sense to me, just as (intuitively, and I could be wrong) I wouldn't expect a similar finding in a random small group of chess players which also had a smaller average score -- the individual ratings would still be more predictive than "belonging" to this group, I'd bet. I also think I've read once that regression towards the mean could the explanation in the opposite direction, that women aren't "really" worse than men on average, that it's just a mathematical artifact of there being way fewer female chess players. It couldn't work both ways, could it?

Not so concise. :-/

Following there's the post I wrote before, with perhaps some still relevant points, additional explanation of my "reasoning", and hopefully not terrible to read.

Chess has a fairly interesting rating system, that is roughly (perhaps "roughly" underscores it) an estimate of performance, and that "predicts" odds of next performance, according to the rating of the opponent. Chess players gain or lose points according to the rating difference of the adversary, and the result. The higher are the odds against the winner (per higher rating of the adversary), the higher the winner's ratings are increased. One can even score points by drawing with a significantly stronger opponent (perhaps with "any" stronger opponent, but only a negligible increase the closer the ratings are). And likewise, the higher were the odds favoring someone who loses, the more the loser's rating is decreased.

A psychological study has found that girls, in gender-mixed tournaments (which I guess are the majority[*]) underperform when playing against boys, according to this system. The theoretical psychological explanation is "stereotype threat", that psychological stress makes people perform significantly worse in situations that evoke stereotypes about the inferior performance of their group, like "girls aren't good at chess", or even mildly improve, in positive stereotypes.

http://bps-research-digest.blogspot.com/2014/01/girls-underperform-when-they-play-chess.html

One of the commenters on the blog suggested that it's just the expected from regression towards the mean, given that women, in general, have lower ratings than men (that's a fact, not questioned here).

But can it really explain this finding? Would it also be expected that by sorting random groups of players (can be all men, compared only against men, or computers/chess engines), only adding up to a similar a lower average rating of that group (or perhaps preferably a similar rating distribution), and looking at their matches, find that individuals on this group are "more likely" to have underperformed, as expected from their ratings?

It strikes me just as odd as saying that somehow Kasparov has a "higher chance" of underperforming if I, a weak chess non-player, enter the same tournament, and we're wearing the same shirt color. And at the same time I'd not play any better, even if everybody with other shirt colors are bad enough to have together a lower average than the embarrassing "Kasparov and I" average. That is, a random, smaller group, only with a lower average rating wouldn't obviously affect the performance of any individual player (good or bad). But would it somehow be saying anything about odds of individuals within that group? Perhaps something like boys, heavy-metal fans only, would be expected to still often have shorter hair length when compared to girls? While that doesn't seem that unlikely, I'm not sure it's quite analogous with the expectations of the rating mechanism.

[*] the proportion can have a role explaining the disparity, as the ratings of female-only tournaments could be inflated. But again I'm not sure it's that much significative, I think it would be similar for any proportionally weak "beginner/weaker player tournaments". And there would be many more weaker boys than girls in absolute numbers, if that levels things up. I'm not sure of anything really. I just think that such think would have an effect at national levels, that is, weak countries with inflated endogenous ratings, but I've never heard of that. It's also perhaps worth mentioning that the full range of potential explanations isn't in question here (it even sort of goes against the site's rules), the scope should be whether just random below-average groupings would allow us to predict a lower average performance for individuals within that group, even when that wouldn't be expected without such grouping, somehow. Or whether somehow these are mathematically different propositions, an invalid analogy.

For me it's just utterly counter intuitive, seems plain wrong, but in the other hand, like many other people, I also thought that it wouldn't make a difference to chose the other door or not on the Monty Hall dilemma.

I'm sorry for the length of the post.

-
Your question seems to boil down to whether significance tests can be constructed for the sampling of pairs in these competitions, to detect if the apparent bias is due to random sampling. Of course these can be constructed, though the Stats.SE site would probably give a better discussion of appropriate tests. Despite the length of the post (or perhaps because of it), it's not quite clear what is "utterly counter intuitive", the findings or the explanations offered (in part by yourself, it seems). –  hardmath Jul 20 '14 at 15:45
What I think is counter-intuitive (perhaps because it's simply wrong) is the attempt of refutation from the blog commenter, the idea that it's all due to regression to the mean (of lower averages), not a psychological effect. I think it would imply that the same would likely be found with any smaller "random" group with a lower average score. That seems weird to me. Unless perhaps it's just the witnessing of distorted chess ratings being "naturally adjusted". –  patzer Jul 21 '14 at 18:25
Okay, then I take your point, though I'd probably frame it as "weak refutation" rather than a counter-intuitive argument. As you say, it would imply at least a likelihood of the same effect with a small subgroup that (perhaps for systematic reasons, perhaps by accident) has a lower "stratification" in the overall population. Because chess ratings have been "gamed" quite intentionally as the sport evolved to cope with matches between unequal players, I admit to not having much "intuition" about how it ought to predict those outcomes, where upsets would be something of a "tail" distribution. –  hardmath Jul 21 '14 at 18:36

I'm somewhat mathematically illiterate... Then I'll try to keep this in mind.

Firstly, may I clarify a point:

• ...girls underperformed in matches against boys. This is not a matter of boys are better than girls on average. This asserts that a girl with rating $X$ will tend to perform better against a girl with rating $Y$ than a boy with rating $Y$.

I don't see the relevance of "regression towards the mean" comments; I think it's just used as a fancy-sounding, tangentially related buzzword.

The real question is: is it statistically significant?

Overall, they performed at 83 per cent of their expected success rate when playing boys. -- blog post.

Here are some questions: What if we repeated the experiment: would the girls still perform at 83 per cent? Or is it possible that boys performed at 83 per cent (instead of girls)? Imagine if the first experiment instead gave the result "boys performed at 83 per cent"; would we instead be finding explanations as to why this is "true"?

The research was conducted by actual scientists and published in a peer review journal (link to paper), and are not just some random comments on a web blog. So (for meta reasons) we can be safe in assuming it's not going to be explained away so simply (if it were, the reviewers would have done so). I.e., we can assume they answered the above questions via hypothesis testing: they would have shown that it's unlikely that this occurred merely by chance. (Note: I don't have access to the full text, so I can't confirm this.)

-
The point you've mentioned is perhaps the key thing that makes the "refutation" of the study seem so obviously/intuitively nonsensical to me. If my analogy is valid (of equivalency with any almost-random lower-average grouping, and no expectancy of reproducing the effect), the only way I can see it begin to make sense (and including the use of "regression to the mean", I guess) would be considering additionally an effect of inflated beginner female ratings (deviations from the mean - but wouldn't be in both directions, unbiased? If that's relevant) –  patzer Jul 20 '14 at 18:50