Chess psychometrics – Female privilege II

In the last post we have seen that women on average bring fewer Elo points to the table if they draw tournament chess games against men. Fewer than if they would play other women and fewer than men would require to draw against men.

One explanation for this would be that men are more inclined to make disfavorable draws against women. In this scenario men would willingly donate Elo points to women instead of pushing for a win. Maybe because they try to specifically show goodwill to female opponents, maybe because they just can’t muster the same aggression as against male players.

However, this statistically robust phenomenon could also be due to confounding. The most realistic confounder is age or speed of improvement. Young players improve quickly. Therefore they will generally be stronger than their rating implies. This of course results in the effect that they will on average be lower rated than their opponent in drawn games.

To make sure that we are not chasing a chimera we bin our Elo differences by age for both male and female players against male opposition. Then we can look for each age group separately whether women have „drawing privilege“ compared to males of the same age. This takes care of the improvement issue, in fact it probably overcorrects, because boys probably improve a little faster.

The following plot shows the number of games for each 5 year age span.

Here we have to confront an ugly truth: Our unique identification of players is not very unique. There aren’t actually any player of the age 0-5 and not many more at the age of 90+. These are noise, due to misidentification. However, the age groups 15-35 rise far above the noise, indicated in the figure by the black line, so this is the area where our results may be reasonably accurate. Additionally, we have to take the possibility into account, that the Elo difference in drawn games is an underestimate diluted by misidentified and possibly misgendered players.

Here are the corresponding Elo difference between male and female drawing players. Inside the black box are the observations where the sample number rose above the noise line in the earlier plot. The age groups 10-15 and 45-50 should maybe also be removed, but they fit the overall trend even if 10-15 is a major outlier.

So what do we observe? The female drawing privilege drops steadily with age. At age forty it is only half of what it used to be at age twenty. The effect size is slightly smaller, so maybe having many young players did make a difference. Overall we see that our results are not confounded by age. In fact they fit the theory very well that men are unwilling to try hard to beat young women.

Chess psychometrics – Female privilege

Privilege is a hot topic currently. For some reason, one of the most obviously privileged demographic is rarely mentioned: Young women.

Young women have better educational outcomes. Earn more than their male counterparts in almost all bigger US cities. And are set to enjoy a significantly longer life.

On top of that, young women, especially if they are somewhat attractive, can count on support, leniency and favoritism from a significant number of men.

Of course many people would dispute that. A few years ago there was an interesting paper [1], showing that in chess men play more aggressive opening lines against female opponents.

The implication is that female chess players do not compete under the same conditions as male players do. An elevated aggressiveness could very well discourage women from competing in chess. A similar pattern could be present in the labor market as for instance at a job interview. If male recruiters treat female job applicants more aggressively than male applicants then women could be discouraged from applying for or accepting such jobs.

Patrik Gränsmark

The method used in this paper is quite crude and I hope to present a much more elegant way to determine risk taking and aggressiveness in chess in a later blogpost. But I do not doubt the result that men play more aggressive opening lines against women. However, I am less sure about the motivation behind this choice.

The authors of the paper seem to assume that the objective of this opening strategy is to beat the female opponent. They then show that the strategy backfires and men actually score slightly worse if they play more aggressively. But it could just as well be the case that men choose risky lines trying to impress their female opponents. In that case the choice would not be irrational despite the lower probability of winning. The optimal course of such a game might be a daring and creative gambit later followed by a generous draw offer.

These are two opposing interpretations of the same result. Either men go out of their way to beat women, or men are trying to impress the rare potential chess playing mate and don’t mind losing an Elo point or two on the way.

To get to the bottom of this puzzle we will analyze the draws, offered and accepted, between male and female players.

In chess, you can offer a draw after making a move, before you press the clock, and the opponent may accept the offer or make a move to continue the game. So in almost all cases the player to last make a move in a drawn game has been the player to offer the draw. If you want to keep your Elo at its current level, you are well advised to make draws against opponents that average the same Elo as you. But if you are favorably inclined towards a certain group, say young women, you might accept and make draw offers against players averaging a significantly lower Elo. Conversely, if you very much want to beat them, you might only accept draw offers by significantly stronger players.

We analyze two million chess games, trying to uniquely connect the players names to the fide player database. This allows us to determine sex, age and other interesting attributes. As we will see, it isn’t at all easy to uniquely identify the correct player and we only manage to do this for 30% of the players. Typos, varying initials and different transcriptions make this difficult.

Initially, I looked at games with both players above 1900 Elo and I distinguished between games longer than 20 moves or shorter, and between white offering the draw and black offering the draw, and between the women playing white and the man playing white.

However, I found that the result is always pretty much the same. In a drawn game between male players, or between female players only, black will be slightly higher rated (4 to 12 points). This makes sense, because it is compensation for the first move advantage that white enjoys.

If a male player makes a draw against a female player, compared to the baseline, the female player will be 40-50 points lower rated. So a women has to bring 40-50 fewer points to the table to make a draw (if playing a man).

Given that we sliced and diced the dataset into several parts, we can be sure that this result is very robust. Here are the values for the overall dataset, the first player is white, the number is the average Elo advantage of white:

Women against men: -62.7 points
Men against men: -10.0 points
Women against women: -8.8 points
Men against women: 43.96

Note the symmetry: It doesn’t matter who is white and who is black. The Elo loss for the men compared to the baseline of intra sex games is the same 52-54 points.

These numbers are based on more than 10,000 games between a man and a women and significantly more games by men against men.

This does not look like men are trying harder to beat women. In fact, it looks more like support, leniency and favoritism.

But it is not a slam dunk yet. In the following blogposts we will dig a little deeper, make sure we are not being confounded in any way and try to unveil the entire story.

[1] Men play more aggressively against women

Chess psychometrics – Women in Chess

After looking at the rating gaps between ethnic groups, we will today turn our inquiring gaze towards the relative performance of women in chess. More than a decade ago, there was a paper that purported to explain the relative absence of female players at the very top by the low participation rate of women [1].

This never sounded particularly convincing to me, because if underrepresentation were the only reason, we would expect a women among the top twenty (at a 1:19 underrepresentation) and 5 women among the top 100. Instead there are 0 and 1 women in the top 20 and the top 100. I also knew that women average roughly 200 points lower in the German rating system. So a lower performance across the board based on differential ability or interest in the game seemed more likely.

Initially, my investigation supported this suspicion. For 150,000 men and 16,000 women among the active chess players in my database, the men have on average roughly a 200 Elo advantage (1679 vs 1474). A closer look at the distributions, however, made it unlikely that this gap is due to innate differences.

As we can see, the female curve is not even remotely gaussian. The true bell curve seems to be hidden under an avalanche of beginners. By restricting the dataset to those players born between 1960 and 1990, we eliminate most of the influence of age and almost all beginners.

Now, the avalanche of very weak players is almost completely purged and the bell curves become very similar. There is actually a female advantage between 2100 and 2400. Maybe this is the area where you are very good as a woman but nothing special as a man (the rating necessary for women titles fall into the space: 2100 for women fide master, 2200 for women international master and 2300 for women grandmaster).

After 2500 there is a distinct male advantage, which reproduces the observation that underrepresentation cannot explain the under representation at the very top. However, given that overall these bell curves are very similar and given that there was actually a women in the top ten only a decade ago, this doesn’t seem to point to innate ability differences.

Of course, underrepresentation can hide innate ability differences, if the female players are just sampled from a higher percentile of innate talent. But this is difficult to investigate and certainly beyond this blogpost.

[1] Why are the best women so good at chess?

Chess psychometrics – Ethnic Elo Gaps in the US

In the last post, we showed that US-American Ashkenazim have a higher average Elo rating than the average US chess player. In this post we extend the analysis to the ethnic and racial minorities African Americans, Asians and Hispanics.

Originally, I was just curious whether Asians had overtaken the Ashkenazim. This seems to be happening in several other measures of academic excellence. I assumed that to be the case, because chess rating first and foremost reflects how much work you have put into chess. So it likely responds strongly to the Asian work ethic.

When comparing Ashkenazim to the general US player I was methodologically quite lazy. I didn’t bother correcting for age because I assumed that the age structure would be similar enough. When looking at the Asian Elo this assumption no longer holds: Most current Asian chess players in the US are likely the kids of relatively recent immigrants.

Indeed, the average birth year of Asian chess players in the US seems to be 2001 vs 1986 for all players. Instead of using Elo directly, we look at the deviation from the average global Elo for the given birth year. This allows us to eliminate the effect of age.

Global average Elo rating by age

We use the 2010 US Census [1] to create samples of Asian, Black, Hispanic and White chess players. These are all players whose names belong with a >90% probability to the respective ethnic group. (For blacks we choose >80% probability, because otherwise the sample is almost nonexistent.) I restrict the samples to male players and birth years >1950.

I also create an additional Ashkenazi sample using my list of Ashkenazi names that occur among Israeli chess players and are not obviously mostly non-Ashkenazi (I excluded Perez, Miller, Brown).

This results in 420 Asians, 60 Hispanics, 66 Ashkenazim, 391 Whites, 14 Blacks. The low number of Whites and Blacks is a result of the difficulty of clearly distinguishing these two groups by surname. The high number of Asians is a result of uniquely Asian names. Ashkenazim are probably much more over represented than they are in my samples.

These are the age controlled deviations from the global average Elo for each group:
Asians 339
Ashkenazim 250
Whites 189
Hispanics 140
Blacks 125

With a White standard deviation of 215 we transform these numbers into IQ for a more intuitive comparison:

Asians 110.4
Ashkenazim 104.2
Whites 100.0
Hispanics 96.6
Blacks 95.5

The differences are probably distorted by weaker groups being more predominantly sampled from the right tail. But the ranking is quite unsurprising. After all, the 2019 US chess champions are called Hikaru Nakamura and Jennifer Yu [2]. With the much younger age structure US chess is bound to become much more Asian dominated in the future, with the ex-Soviets fading into the background.

[1] US census 2010

[2] US chess championship 2019

Chess psychometrics – The Ashkenazi advantage

In this post we are going to take a look at the chess performance of Ashkenazim relative to other Europeans. Ashkenazim are massively overrepresented in the higher echelons of chess history, with almost half of all World Chess Champions having at least partial Ashkenazi ancestry. But of course, it is not a priori clear that this is the result of stronger chess playing ability on average. 

Our data set has some clear sampling issues. Generally only relatively strong players are going to play rated games and the less developed the chess infrastructure of a country the more that is going to be the case. For example, the highest average rating of all countries is exhibited by Cuba. Cuba is a legit strong chess playing country with a World Champion (Capablanca) and some current very strong players (Lazaro Bruzon, Lenier Dominguez) to its name. But if we compare the rating distribution to the distribution of the USA, we see that the higher average is due to left part of the distribution missing, while there is still a gap at the right side.

In this figure we normalized the distributions by height. That’s not perfect, but it is probably better than normalizing by number of players, because that would over emphasize the right tail if the left tail is thinner due to under sampling. 

For the Ashkenazim, we can partly circumvent that problem by comparing US Ashkenazim to all other US players. Then at least the chess infrastructure is the same, although a group with a lower mean might still be under sampled at the left tail possibly reducing the difference. This is also a high bar to clear for the Ashkenazim because the US has stronger rated players than comparable Western European countries.

We look at US players with a typical Ashkenazi name. We circumvent the problem discussed in my post „Counting Names“, by only considering names that also occur among Israeli players. This makes it unlikely that we pick up German, English or Spanish names that are also (but rarely) found among Ashkenazim. 

The figure shows an Ashkenazi advantage both at the left tail and the right tail. Contrary to my assumptions it is bigger at the left tail. It might be the case that less Ashkenazim commit to high level chess due to a comparative advantage in other fields. That is for example very noticeable in Germany, where there is a strong chess infrastructure, a general high level, but almost no (native) chess professionals and consequently no international contenders. 

With these caveats in mind, the average US Ashkenazi rating is 2014 while the average US non-Ashkenazi rating is 1924. A difference of 90 points or 0.40 standard deviations. In terms of IQ this translates to 106 which is not completely out of line with other cognitive measures, especially if one takes into account that the g-loading of chess is likely not very high. 

Chess psychometrics – The gender equality paradox

Chess databases contain millions of games, whose players can largely be identified by the Fide players database [1] which contains age, sex, nationality and ratings. These games are interactions providing information about behavior in a competitive context. They are a goldmine for psychological or sociological research into a wide range of topics. The datasets derivable from chess databases are much larger than what can be realistically achieved in typical psychological research. As a researcher you are really only limited by your imagination and the number of your grad students.

While the typical university professor is severely limited in the former, I am unfortunately limited in the later. So, we will see how many of my chess psychometrics projects I’ll be able to bring to completion. For now we’ll start with something simple and not very original: We will check whether the gender equality paradox holds in chess.

The gender equality paradox is the observation that women in more gender equal societies tend to choose more stereotypical female occupations and are less likely for example to go into STEM. The gender disbalance in chess is very comparable to the disbalance in STEM research [2]. In fact, it is usually even more extreme, with women in many countries only representing less than 5% of the pool of rated players.

Outside of developed countries, the number of rated players is often quite small and not very representative when it comes to age, rating or possibly sex. So it is not surprising that on a global level we find no correlation between the global gender gap index and the fraction of female players.

In European countries however, there is a significant negative correlation between gender equality and the fraction of female players. (Yes, Turkey is for some reason in my list of European countries.)

Pearson correlation: -0.43603692297301305, p-value: 0.01119225533794084

This looks like a straightforward result. However, I am generally skeptical about the significance of these kind of correlations, because I suspect that often the significance is a result of countries falling into a small number of similar behaving clusters. If these clusters are then arranged linearly by chance, we get a significant correlation by virtue of decomposing these clusters into many countries.

So it might be the case that Northern countries all have high gender equality and low female chess player fractions by chance. While Eastern Europeans have low gender equality and high female chess participation for historical reasons. Because these clusters and the rest of the countries in between constitute a lot of observations the results looks a lot more robust than it really is.

Sure enough, there is no such correlation in Eastern Europe and restricted to Western Europe the correlation looses all significance.

On the other hand, the loss of statistical significance is due to just two outliers: Iceland and France. So is the gender equality paradox a thing in chess or not?

To detect even a rather weak tendency, we average over all countries that fall into the same section of the GGG-index. This time we look at all countries in our dataset.

If we ignore the four countries with the lowest gender equality which average very low, we actually see a nice downward trend in female chess playing the higher the gender equality. I tentatively conclude that the gender equality paradox does actually exist in chess.

[1] Fide player database

[2] The Gender-Equality Paradox in STEM Education

Demographic Change in France: Discussion

The typical rightwing theory would be that the immigrant population is outbreeding the natives due to much higher birthrates. Is this the story behind the sickle cell data? If we translate the percentages into absolute numbers based on the number of births in each relevant year, we get absolute births:
2000: 778900,
2007: 785985
2010: 802224
2012: 790290
2013: 781621
2015: 760421
Sickle cell tested newborns:
2000: 147991
2007: 223613
2010: 252701
2012: 272176
2013: 279039
2015: 295804
Not sickle cell tested newborns:
2000: 630909
2007: 562372
2010: 549523
2012: 518114
2013: 502582
2015: 464617

This is a growth of 4.72% per year and a shrinking of -2.02% per year respectively. Let’s imagine there was a French population and an immigrant population established maybe in the 60ies. And both populations were breeding merrily away with a perfectly steady fertility rate. In this closed system, what kind of fertility rates would account for the growth rates we see?

Well, with a generation length of 30 years (and human generation lengths almost always fall close to 30 years even with very different fertility rates), a fertility rate of 7.97 kids per woman for the immigrant population and 1.08 kids per woman for the French population would lead exactly to the growth rates we calculated.

That’s of course insane. According to the studies I have seen on this topic, immigrant birth rates have never been close to 8 kids per woman and these days they are certainly far lower. The French fertility rate of 1.08 kids per woman is also crazy low, because we made the assumption of a steady rate over many generations. If the rate was higher in the past, today’s birth rate would have to be even lower to account for the decline. Or conversely, if the birthrate today was actually higher, it must have been below 1.0 in the past.

So how do we square the circle? The sickle cell birth rate increase has to be predominantly driven by recent immigration. That fits the numbers. Birthrates among immigrant usually drop relatively quickly towards the birth rate typical of the country. Legal immigration to France has been massive in recent years. And there is also illegal immigration, estimated by Wikipedia to lie between 80,000 and 100,000 per year [1]. The story is probably one of young immigrants coming to France and then having kids, which also implies that a total immigration stop might lead to a reduction of the percentage of sickle cell babies.

But how can the native French birth rate be this low? The answer is probably that there is a certain amount of intermarriage, which get’s counted for the sickle cell numbers, and the overall fertility rate of ethnic French women is pretty close to that in neighbouring European countries, maybe in the vicinity of 1.4.

By the way, the current rates of growth and decline predict parity in sickle cell births and non-sickle cell births in 2022 and a 66% majority for the former in 2032. Of course, for reliable predictions a more sophisticated model is needed than just extrapolating growth rates.

I am not somebody to grieve for the French genepool, but I think this rapid change is dangerous for a variety of reasons.

It seems probably that within a decade or two, most French people will wake up to a reality were France is still 70% white, but the future is very noticeably 70% black. How will they react?

If the relative growth rates hold and at some point the political power changes hands, how will that effect the ethnic French? Even in a very peaceful best case scenario the new government will have been brought into power by a electorate much younger than the opposition, and with wages and pensions much lower than those of the opposition. In this situation a massive cut in pensions is the logical result in a democracy.

Or maybe the percentage stabilizes somewhere and French and Africans just have to live side by side. Well, the Basques, the Northern Irish, the Ukrainians and all inhabitants of Balkan states will tell you that even in Europe, having different ethnic groups in one country is not a recipe for peace. What does the trouble in the banlieues look like if scaled up five-fold?

I am not sure how big the achievement gap between second generation immigrant and the ethnic French is. But if there is a significant gap, the massive influx of lower qualified workers into the labour market will retard economic growth. That’s not gonna work wonders for ethnic relations. All in all a very worrying development, with France not the only European country in which rapid demographic change might lead to major upheaval in the next decades.

[1] Illegal Immigration to France