Monthly Archives: October 2018

Against fact-checking

What does it mean to check a fact? It’s a more complicated question than it may seem.

Let us take a simple example.

The sky is green.

Here the speaker is asserting: there is such a thing as a sky; it can be attributed properties of colour; and we can know what that colour is. One can go outside, look up, and check the fact.

Let’s take a more complicated example, such as the one Channel 4 FactCheck dealt with today.

Men are more likely to be raped than be falsely accused of rape.

Some of the suppositions contained within this statement are quite uncontroversial. There is such a thing as ‘men’, and ‘rape’, and ‘false accusations of rape’. We can all accept this, and we can broadly agree on definitions of each of those things, even if we might disagree firmly about the relevance and prominence of each.

The tricky supposition is the probability function contained within the expression “more likely than”. The reader may note that rape and false accusations are both, at least in theory, capable of being recorded and studied, so she might look for some data on their prevalence. However, by ‘checking the fact’ of whether men are indeed more likely to be raped than be falsely accused of rape, here are the premises the checker must accept:

  1. It is possible to perform a valid statistical comparison on these two probabilities.
  2. The sources from which we might acquire statistics for such a comparison are capable of supporting such a comparison.

To assess the prevalence of male rape, Channel 4 uses the figures from the Crime Survey of England and Wales, which estimate a yearly figure of around 8,000 victims in the entire adult male population (about 0.05%, it adds helpfully).

To assess the prevalence of false accusations, Channel 4 has a more complex method. In its cheery mid-Atlantic dialect it notes:

“First up, let’s acknowledge the elephant in the room: there is no reliable data that exists to tell us how many people are accused of rape in day-to-day life.

“There are all sorts of environments — at work, at home, in public — where someone could make an allegation of rape, truthfully or otherwise, that may go unrecorded. If false, these accusations could have far-reaching and profound effects on the accused’s life, even if they never result in police or judicial action.

“But we can only analyse the data we have. So, for the purposes of this article, we will look at rape allegations that make it to the courts.”

This is quite a caveat. Already we have hurtled past assessing a comparable statistical population. We are no longer considering what proportion of men in the whole population are falsely accused; rather, it is what proportion of men who are prosecuted for rape are falsely accused.

Overall, in the CPS’ view, false allegations of rape are “serious but rare”. They estimate that there is one prosecution for a false rape claim out of every 161 rape cases prosecuted. That means — according to the best available data — false allegations make up 0.62 per cent of all rape cases.

I have written before about how it is possible for an instance of rape to fail to meet the criminal standard while still being a true allegation. In this instance, Channel 4 outlines the statistics – noting with considerable understatement that “the data we have is not always directly comparable” – and concludes as follows:

“But from what we can tell: yes, men are more likely to be raped than be falsely accused of rape.

“According to the most reliable data we have, the average adult man in England and Wales aged 16 to 59 has a 0.03 per cent chance of being raped over the course of a year (based on 2016-17 figures).

“The best data we have — the number of people prosecuted for making false allegations — suggests that the average adult man in England and Wales has a 0.00021281 per cent chance of being falsely accused of rape in a year. (That’s based on 35 prosecutions for false rape allegations in 2011 compared to 16.5 million men aged 16 to 59 living in England and Wales at the time).

“By this measure, a man is 230 times more likely to be raped than to be falsely accused of rape.”

This is tricky, but look closely. We are being asked to compare:

  1. figures from the Crime Survey of England and Wales on the prevalence of male rape in the whole population to
  2. the proportion of prosecutions for perverting the course of justice which succeed, which are a proportion of the whole number of prosecutions for perverting the course of justice, which are a proportion of rape prosecutions which fail in court, which are a proportion of all rape prosecutions, which were themselves selected from the whole number of rapes and alleged rapes which exist in society.

Then, that proportion is taken to be exhaustive of the entire population in the way that the Crime Survey data is (statistically) a sound sample of the entire male adult population. By making the comparison, then, we inevitably compare one definition of “false accusation”  (successful prosecution for perverting the course of justice) to a definition of “rape” (anonymous response to a crime survey) which is not easily compared.

It is rather as if we used the number of watches successfully repaired in a given year as exhaustive of the number of watches that break down, and then compared it to the number of people who reported in a survey that their computer crashed. Many watches are not discovered when they break down. Many are not repaired at all for reasons of expense or laziness. Many are simply disposed of and replaced. Either way, the figure is simply not reflective of the underlying reality in the way that the survey on computers is.

This is why the Crime Survey of England and Wales is taken to be a better picture of the number of sexual assaults, as many go unreported. To compare like with like, we would need the Crime Survey of England and Wales to ask a question along the lines of ‘Have you ever been falsely accused of rape?’ The Crime Survey of England and Wales does not do this, for many reasons – but unless and until they do, a comparison reconstructed from prosecutorial data is inherently flawed and of virtually no use whatsoever. The very act of fact-checking itself alters the question being assessed.

Channel 4 does, to a degree, recognise this and makes an effort to steelman the case against it.

“…let’s put the stats through an even stronger test.

“Imagine for a second that you believe that every single one of the men prosecuted for rape in England and Wales in 2016-17 was falsely accused.

“Even if that unlikely scenario were true, there would still have been more adult male victims of rape (8,000) than men prosecuted for those rapes they “didn’t commit” (5,190).”

Here we are not so much comparing apples and oranges as comparing apples and lamb chops. We are still comparing crime survey data to prosecutorial data, but now we have tossed away the notion that we are comparing likelihood at all. Logically, those rape cases which succeed at court are (one imagines) the most promising cases of all and have the best evidence behind them – consequently, those are the least likely to be false accusations. A more testing statistical steelman would be not that all prosecutions are false accusations, but that all reports are false, as many more cases are reported than prosecuted. This would be closer to – though still very far away from – the statistical standard we see in the crime survey data. Instead, in purporting to use a stronger test, it actually uses a weaker  one.

There is an even more serious concern. A slogan like “men are more likely to be raped than falsely accused” is not used neutrally. It is by its very nature only used when the prosecution of a rape case has inflamed passions, typically on social media. In that scenario, the relevant population for the question of false accusations is not ‘all men’, but ‘all men who are accused of rape’. It would sound ridiculous to ask “what proportion of men who are accused of rape are themselves victims of rape?”, of course. But the statistical comparison only works if the same controls are embedded in the data. Otherwise we are confounding ourselves with irrelevant statistics.

The trouble with fact checking is that by checking a ‘fact’ it endorses the epistemic framework in which the ‘fact’ is produced, e.g. that a putative comparison between the Crime Survey and prosecution records is a statistically valid idea in the first place. Fact-checking therefore is in the unenviable position of reproducing the questionable premises of the ‘fact’ it is checking. Channel 4 is unlikely to conclude by saying ‘we cannot know and any comparison purportedly based on data is meaningless’, but that is the only honest response. By assessing whether the data backs up the claim, however, it implicitly accepts that the data can back up the claim – and that apples may speak to oranges on equal terms. They cannot.

Advertisements

Leave a comment

Filed under Culture, Law