Don’t run away.
This post is going to be about maths and probabilit
I SAID DON’T RUN AWAY
There was a scientific paper recently published, in a respected academic journal, which purported to demonstrate evidence of human precognition.
Yep, science says people can tell the future.
Except, not really. Not yet, anyway. As the study’s author, psychology research Daryl Bem, said himself in the published paper, it was important for other scientists to repeat the experiment, and see if they got the same results. Richard Wiseman has been among those involved in such attempted replications, which so far have failed to support Bem’s original conclusion.
There’s a big moan I’m not quite in the mood to make, about how science generally gets publicised in the media, and the tabloids’ tendency to make a massive fuss over preliminary results, without concerning themselves with facts which later emerge and completely undermine their sensationalist headlines.
But I want to talk about the maths.
Replication is always important in science, particularly where the results look unlikely, or demonstrate something completely new. This is partly because, for all we know, Bem’s original research could have been dishonest or deeply flawed. Most people seem to consider both of these unlikely, though, and I’m certainly not suggesting that he’s faked his results.
But people often seem to assume that these are the only two options: that positive results must mean either an important and revolutionary breakthrough, or very bad science. The idea that something could just happen “by chance” now and then never seems to get much credibility.
Almost every time someone in a TV show or a movie proclaims something to be “just a coincidence”, or that there’s a “perfectly rational explanation”, we’re meant to take it as an ultra-rationalist denial of the obvious – usually supernatural – facts. Remarkable coincidences just don’t happen in the way that ghosts and werewolves obviously do. In fictional drama, there are good reasons for this. In the real world, this is a severe misunderstanding of probability.
When deciding whether or not to get excited about a result, scientists often look for significance “at the 5% level”. Bem’s results, supporting his precognition hypothesis, were significant at this level. But this does not mean, as you might think, that there’s only a 5% chance of the hypothesis being wrong.
What it means is: there would be a 5% chance of getting results this good, just by chance, if people aren’t really psychic.
So, getting results like this – statistically significant at the 5% level – is actually slightly less impressive than rolling a double-six. (If you have two regular six-sided dice, the odds of both landing on 6 on a single roll is 1 in 36, which is slightly less than 3%.)
I’ve rolled plenty of double-sixes. If you’ve rolled a lot of dice, so have you. And if you do a lot of science, you’d expect just as many random chance results to look significant.
So, if you’re thinking that we should probably ask for something a bit more conclusive than a double-six roll before accepting hitherto unconfirmed magic powers, you’re probably right.
This is the essence of Bayesian probability. Imagine having one of the following two conversations with a friend who has two dice:
“These are loaded dice, weighted to always land on a double-six. Watch.”
“Huh, so they are. Neat.”
“I’m going to use my psychic powers to make these dice land on double-six. Watch.”
“…Okay, that’s a little spooky, but you could’ve just got lucky. Do it again.”
You see why you might not believe it right away when your friend claims something really outlandish? But when it was something pretty normal, you’d be more likely to buy it?
In either case, the odds of rolling sixes by chance were exactly the same, 1 in 36, independent of what was allegedly influencing the outcome. But that doesn’t mean you should be equally convinced in either case when the same result comes up.
Both claims become more likely when the double-six is thrown. After all, if the dice really are loaded (or psychically influenced), then what you’ve just seen is exactly what you’d expect to see. But they’re not both getting more likely from the same starting point. One started out as a much more plausible claim than the other, and it’s still more plausible now.
Loaded dice? Sure, they have those. Telekinesis? Well, you have my attention, but let’s see you do it again. And again. And a dozen more times with a fresh set of dice.
This is part of my recurring, occasional project to convince the world that Bayesian probability is both important and intuitive, when it’s expressed right.
Ben Goldacre wrote about Bem’s research, the New Scientist also discussed it, there are some details of the replication attempts at The Psychologist, and I was prodded into thinking about all this in some more depth by a recent episode of the Righteous Indignation podcast.