Against fact-checking

What does it mean to check a fact? It’s a more complicated question than it may seem.

Let us take a simple example.

Primus: The sky is green.

Secunda: Let me check that fact.

The sky, as we know, is blue and not green. But what is Secunda assuming, by taking Primus at his word and checking whether the sky is in fact green? Nothing controversial: Secunda assumes that there is a sky, and that it can be attributed properties of colour. Secunda goes outside, looks up, notes that the sky is blue and not green, and strikes Primus across the forehead for being a prat.

Let’s take a more complicated example, such as the one Channel 4 FactCheck dealt with today.

Primus: Men are more likely to be raped than be falsely accused of rape.

Secunda: Hang on, let’s check that fact.

Secunda’s presuppositions are partly quite uncontroversial. There is such a thing as men, and rape, and false accusations of rape. We can all accept this, and we can broadly agree on definitions of each of those things.

The tricky presupposition comes when Primus talks about probability. Secunda may note that rape and false accusations are both, at least in theory, capable of being recorded and studied, so she might look for some data on their prevalence. However, by ‘checking the fact’ of whether men are indeed more likely to be raped than be falsely accused of rape, here are the premises Secunda must accept:

  1. It is possible to perform a valid statistical comparison on these two probabilities.
  2. The sources from which we might acquire statistics for such a comparison are capable of supporting such a comparison.

To assess the prevalence of male rape, Channel 4 uses the figures from the Crime Survey of England and Wales, which estimate a yearly figure of around 8,000 victims in the entire adult male population (about 0.05%, it adds helpfully).

To assess the prevalence of false accusations, Channel 4 has a more complex method. In its cheery mid-Atlantic dialect it notes:

First up, let’s acknowledge the elephant in the room: there is no reliable data that exists to tell us how many people are accused of rape in day-to-day life.

There are all sorts of environments — at work, at home, in public — where someone could make an allegation of rape, truthfully or otherwise, that may go unrecorded. If false, these accusations could have far-reaching and profound effects on the accused’s life, even if they never result in police or judicial action.

But we can only analyse the data we have. So, for the purposes of this article, we will look at rape allegations that make it to the courts.

This is quite a caveat. Already we have hurtled past assessing a comparable statistical population. We are no longer considering what proportion of men in the whole population are falsely accused; rather, it is what proportion of men who are prosecuted for rape are falsely accused.

Overall, in the CPS’ view, false allegations of rape are “serious but rare”. They estimate that there is one prosecution for a false rape claim out of every 161 rape cases prosecuted. That means — according to the best available data — false allegations make up 0.62 per cent of all rape cases.

I have written before about how it is possible for an instance of rape to fail to meet the criminal standard while still being a true allegation. In this instance, Channel 4 outlines the statistics – noting with considerable understatement that “the data we have is not always directly comparable” – and concludes as follows:

But from what we can tell: yes, men are more likely to be raped than be falsely accused of rape.

According to the most reliable data we have, the average adult man in England and Wales aged 16 to 59 has a 0.03 per cent chance of being raped over the course of a year (based on 2016-17 figures).

The best data we have — the number of people prosecuted for making false allegations — suggests that the average adult man in England and Wales has a 0.00021281 per cent chance of being falsely accused of rape in a year. (That’s based on 35 prosecutions for false rape allegations in 2011 compared to 16.5 million men aged 16 to 59 living in England and Wales at the time).

By this measure, a man is 230 times more likely to be raped than to be falsely accused of rape.

This is tricky, but look closely. We are being asked to compare:

  1. figures from the Crime Survey of England and Wales on the prevalence of male rape in the whole population to
  2. the proportion of prosecutions for perverting the course of justice which succeed, which are a proportion of the whole number of prosecutions for perverting the course of justice, which are a proportion of rape prosecutions which fail in court, which are a proportion of all rape prosecutions, which were themselves selected from the whole number of rapes and alleged rapes which exist in society.

Then, that proportion is taken to be exhaustive of the entire population in the way that the Crime Survey data is (statistically) a sound sample of the entire male adult population.

It is rather as if we used the number of watches successfully repaired in a given year as exhaustive of the number of watches that break down, and then compared it to the number of people who reported in a survey that their computer crashed. Many watches are not discovered when they break down. Many are not repaired at all for reasons of expense or laziness. Many are simply disposed of and replaced. Either way, the figure is simply not reflective of the underlying reality in the way that the survey on computers is.

This is why the Crime Survey of England and Wales is taken to be a better picture of the number of sexual assaults, as many go unreported. To compare like with like, we would need the Crime Survey of England and Wales to ask a question along the lines of ‘Have you ever been falsely accused of rape?’ The Crime Survey of England and Wales does not do this, for many reasons – but unless and until they do, a comparison reconstructed from prosecutorial data is inherently flawed and of virtually no use whatsoever.

Channel 4 does, to a degree, recognise this and makes an effort to steelman the case against it.

…let’s put the stats through an even stronger test.

Imagine for a second that you believe that every single one of the men prosecuted for rape in England and Wales in 2016-17 was falsely accused.

Even if that unlikely scenario were true, there would still have been more adult male victims of rape (8,000) than men prosecuted for those rapes they “didn’t commit” (5,190).

Here we are not so much comparing apples and oranges as comparing apples and lamb chops. We are still comparing crime survey data to prosecutorial data, but now we have tossed away the notion that we are comparing likelihood at all. Logically, those rape cases which succeed at court are (one imagines) the most promising cases of all and have the best evidence behind them – consequently, those are the least likely to be false accusations. A more testing statistical steelman would be not that all prosecutions are false accusations, but that all reports are false, as many more cases are reported than prosecuted. This would be closer to – though still very far away from – the statistical standard we see in the crime survey data. Instead, in purporting to use a stronger test, it actually uses a weaker and sillier one.

There is an even more serious concern. A slogan like “men are more likely to be raped than falsely accused” is not used neutrally. It is by its very nature only used when the prosecution of a rape case has inflamed passions, typically on social media. In that scenario, the relevant population for the question of false accusations is not ‘all men’, but ‘all men who are accused of rape’. It would sound ridiculous to ask “what proportion of men who are accused of rape are themselves victims of rape?”, of course. But the statistical comparison only works if the same controls are embedded in the data. Otherwise we are confounding ourselves with irrelevant statistics.

The trouble with fact checking is that by checking a ‘fact’ it endorses the epistemic framework in which the ‘fact’ is produced, e.g. that a putative comparison between the Crime Survey and prosecution records is a statistically valid idea in the first place. Fact-checking therefore is in the unenviable position of reproducing the questionable premises of the ‘fact’ it is checking. Channel 4 is unlikely to conclude by saying ‘we cannot know and any comparison purportedly based on data is meaningless’, but that is the only honest response. By assessing whether the data backs up the claim, however, it implicitly accepts that the data can back up the claim – and that apples may speak to oranges on equal terms. They cannot, and nor should we spread the idea that they can.

Advertisements

Leave a comment

Filed under Law

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.