Eluned Smith

Particle physicist proves that YOU are cleverer than 20% of people at CERN (or how to spot dodgy data analysis when its staring you in the face)

Advertisements

Half way through my conference this week I received a set of slides from a colleague of mine, which claimed to show I was probably only at this conference, or in a job, because of tokenism. What a bummer.

The slides in question were from theoretical particle physicist Alessandro Strumia, who, having spent clearly a lot of time doing extraordinarily high quality research into how men have a much harder time getting a job in high energy physics,  decided not to submit it to a peer-reviewed journal, but invited himself to a  gender equality workshop to share his results with the mainly female audience. Lucky them.

The slides have since been taken down by CERN because they contained personal attacks on members of the academic community. Strumia has also been suspended.

If, like me, you read the slides and saw nothing more than a great case study for someone fudging data to fit a predefined offensive narrative, you’ve probably not given the study’s conclusions a second thought. Needless to say I didn’t immediately  boycott my plenary talk at said conference to make a political point that I was clearly there at the expense of a poor, overlooked, but much more capable, male.

However, my worry is that, when you have someone who people see as “clever”, who throws some data into eye catching plots with much flourish and “maths”, as much as people may protest that the analysis is sexist (believe me, it is) people may secretly think – maybe he has a point – it is data after all. Data is like proof. It can’t lie.

Or as one twitter user put it “I don’t understand why they are hiding this very thorough detailed analysis into gender differences. Another case of cultural Marxism”. Yes, it is just the thought police covering up what we don’t want you to know. FEMALES ARE ACTUALLY REALLY BAD AT SCIENCE!.

So, for all you non stats-enthusiasts  (or non particle physicists confused by the jargony slides) here is a quick as possible break down on why not just the premise of the slides is wrong [*] but why the data analysis itself is clearly very very dodge.

[*] he equates number of citations with ability throughout which is problematic as there are many studies showing citation bias against non -westerners and women – but this has already been well pointed-out by others – and actually – as we will see – by his own data.

The information source for his data analysis is the InSpire data-base, which is a data-base used to track paper citations (i.e. how often your paper is quoted by another paper, a big deal in science ) in high energy physics. In addition, there is information provided on each author in the database, whose depth depends on how much an author has bothered to update his/her profile. Gender is not given so presumably he used a list of standard names or something to sort this.

He shows a number plots, all of which are just variations of the following two plots here:

Fig 1a. Distribution of the number of citations per author per reference  for each gender. The numbers written on the plot show the average or mode I assume. The black curve is the ratio of the red curve over the blue curve. Strumia assures us he has accounted for the fact that there are proportionally many more males at the top in physics, due to the steadily increasing female participation, but doesn’t detail how he’s done this.

Fig 1b. Number of citations per author against scientific age at which one was ‘hired’. Scientific age is defined as time since PhD or first post doc I assume. He seems to suggest that hiring is defined as being longer than a certain amount of time at an institution, where he says that the results are unchanged when defining hiring as being 5 years or 10 years at the same place.

A cross check with any data set is to ask ‘does it makes sense?’. Not only in what the author is trying to show, but other secondary implications given by the data. The data in Fig 1b clearly doesn’t make sense for a number of reasons.

First, literally meet no one who, 30 years after their PhD,  is still in a job where they are moving less then every 5 years (unless I have misunderstood what exactly hiring is defined as).  The sharp spikes in the data also don’t have any obvious explanation, although maybe it’s because they are single data points – without error bars – we can’t know.

(My two cents would be that it’s simply because the database used gives no reliable indication of how long anybody has been around. I checked some of my colleagues on the InSpire data set profile and, of ten, one had bothered to change their affilation on their profile since they started their PhD).

Secondly, it is important to ask if a metric being used is easily susceptible to bias.

The metric here is always some variant on number of citations/number of authors. There is perhaps a reason he never shows absolute citations. Experimentalists tend to have more absolute citations,  being part of large collaborations. Dividing through by number of authors  however (there are often around 3000 authors on a paper) drastically reduces the number of citations for experimentalists.

This means that if there were proportionally more males than females in theory compared to experiment, this would favour males, or vice versa.

Thus, checking gender variation across theory and experiment (which I suspect there is from my experience, and which is very easy to do with the information in this data base) would be a paramount check before ever using this metric. The fact that he hasn’t checked this should ring alarm bells.

And then we come to his money plot. He claims to be able to explain the black curve in Fig 1a (the number of females over males for a given number of citations per author per reference) assuming the following

This is the theory that the male IQ has the same average as the female IQ but more deviation (more spread). This is not necessarily wrong, although he doesn’t cite where he takes the 15% deviation from.  I googled for a while and I couldn’t find an obvious reason for this choice of 15%.

That in itself is very worrying, as he’s about to take this equation and claim how his model is ‘proven’ because it ‘fits’ the citation data.  If someone comes up with an arbitrary model, and then tells you its closeness to data proves that its right, without telling you previously why they picked this model, one might worry the model was picked only because it looks like the data. (This is essentially having a scienitifc hypothesis, and then changing said hypothesis once you’ve seen the data.)

But this is not the only thing that is clearly very wrong about this ‘theory’.

He shows this plot

Fig 2. Curve from individual citations ( number of citations per author per reference), which has already been shown in Fig 1a. You’ll notice he doesn’t go below 10^{-1} ( = 0,1) on the bottom axis. Projected against this is the blue curve he gets from Equation 1 (‘C theory’), and above  the plot is the number of sigma above the average IQ an author with this citation level apparently has

He says that he’s assuming one person in 10^{9} (= 1 000,000,000) people has a 6 sigma better-than-average IQ, where a sigma (the funny symbol above the plot in Fig. 2) here can be seen as a measure of distance from the mean value of a distribution. So 6 sigma above the mean is a far-above-the-average IQ.

He therefore says that those at 10^{3} (= 1000) individual citations have a 6 sigma better-than-average IQ and extrapolates backwards.

As can be seen from Fig. 2, this means that people with individual citations of 10^{-1} correspond to having an average IQ and those with an individual citation less than this have a less than average IQ.  That corresponds to, I don’t know, 20% of high energy physicists [**] with a less than average IQ

Just think about that for a second. I am the first one to say there are some right incompetent idiots among us in physics, but I don’t think I ever met anyone who has a significantly lower than average IQ. Let alone 20% of us.

[**] and implies that women in this low IQ region are over represented, which immediately goes against his assumption that IQ = ability = citations, where both gender IQs are gaussianly distributed 

Of course what has really happened here is that he has picked a mapping between IQ level and citation level that suits his narrative, as well as picking a relationship between male and female IQ to suit his narrative. 

Lets assume the more ‘reasonable[***]’ assumption that people with the lowest citations per author per reference have an average IQ. Then you have to shift that black line on the plot in Fig 2 to the right by a lot – and what do you see – women at a given IQ out perform men – I’ve proved that women are actually better scientists!

[***] although nothing is reasonable about any of this exercise, but to demonstrate how you to can prove any theory you want to…

Of course, I haven’t. I have proved nothing because the only thing he’s shown here is how obviously citations and IQ (or ability) don’t match[****].  Which is what plenty of peer-reviewed research has already told us.

[****] This mismatch between IQ and citations is also something one see from just looking at how non-gaussian (non-bell curve like) the citation distributions are in the first place (note that the x axis in Fig 1a is log)

My after thoughts in foot notes

For anybody who is further interested, his reference to an ‘S_[2]’ symmetry in the first slide is an attempt to compare his theory to a particle physics like theory.

Particle physics is built on the idea that if you can find something that is symmetrical under some group theory symmetry, you will have a corresponding conserved quantity (like energy or momentum). The irony here is that this founding principal of particle physics was deduced by the great mathematician Emmy Nöther. A women who was never allowed to lecture under her own name due to her sex.

Advertisements

Advertisements