Feeds:
Posts
Comments

Posts Tagged ‘math’

Ben Goldacre’s got a fab example of misleading statistics, and the ways in which you can learn to think about things to avoid jumping to a wrong conclusion.

Look at his first nerdy table of data on that article. All they’ve done is take a bunch of people who drink alcohol, and a bunch who don’t, and counted how many from each group ended up with lung cancer. It turns out that the drinkers are more likely to get lung cancer than the non-drinkers.

The obvious conclusion – and (spoiler alert) the wrong one – is that drinking alcohol somehow puts you at greater risk of developing lung cancer. You might conclude, from that table, that if you currently drink alcohol, you can reduce your risk of developing cancer by no longer drinking alcohol, thus moving yourself to the safer “non-drinkers” group.

This is actually a fine example of the Bad Science mantra, and Ben makes an important point which many non-nerds might not naturally appreciate about statistics: the need to control for other variables.

If drinking doesn’t give you cancer, then why do drinkers get more cancer? The other two tables offer a beautiful explanation. Of all the drinkers and non-drinkers originally counted, try asking them another question: whether or not they smoke cigarettes. What you get when you do that is the next two tables.

If you just look at the smokers, then the chances of a drinker and a non-drinker getting lung cancer are almost exactly the same. If you look only at the non-drinkers, ditto. In other words, once you know whether someone smokes cigarettes, whether or not they drink makes no difference to their odds of getting lung cancer.

Which is a long way away from the obvious conclusion we were tempted to draw from the first set of data.

What we did here was to control for another variable – namely smoking – before drawing sweeping conclusions from the data. When we give smokers and non-smokers their own separate tables, it means that smoking cigarettes isn’t unfairly weighing the data we’ve already got any more. It becomes clear that drinkers aren’t simply more likely to get cancer; they’re more likely to be smokers.

And although Ben’s right to point out the importance of controlling for other variables like this, what interests me is the reminder of the importance of Bayesian probability.

In particular, the thing to remember is that the probability of an event is a measure of your uncertainty, and not something inherent in the event itself.

For instance, if that first table is all the data you have, then all you know is that drinkers are more at risk of cancer than non-drinkers. If you were to estimate somebody’s odds of getting lung cancer, and the only thing you knew about them is that they’re a drinker, the best you could do is to place it at 16% – the amount of drinkers who developed lung cancer in the study.

If you later acquire the extra data in the second tables, and find out that the individual you’re interested in is not a smoker, then suddenly you can re-adjust your estimate, and give them about a 3% chance of getting lung cancer. They haven’t done anything differently; nothing about their situation has changed for them to suddenly appear much more healthy. You’ve just learned more about them.

And it’s still not true that their odds of developing cancer are exactly 3% in any objective sense. Maybe tomorrow you’ll learn something about their age, or gender, or family history, and adjust your estimate again based on the new data. Maybe you don’t know that a doctor actually diagnosed them with lung cancer yesterday. This, obviously, makes a huge difference to their odds of having lung cancer – but it doesn’t change the fact that they’re in a low-risk group, and a 3% estimate is the best you can do based on your current knowledge.

In conclusion: stats are hard, listen to maths geeks (or become one yourself) before panicking about the latest tabloid healthscare.

Advertisements

Read Full Post »

Yay, another maths lecture!

Click through to see the whole cartoon at XKCD. Really do it. It’s important. Especially if you want the rest of my burblings to make sense.

So. It’s partly funny because it satirises the sensationalism of tabloid news, and the urge to cram as much excitement into a headline as possible only to leave a sober assessment of actual facts to the blogosphere. But it actually addresses a much more common problem with our understanding of probability.

Most people who pay much attention to any kind of sciencey talk are probably familiar with the p-values referenced in the comic. When scientists are testing a hypothesis, they’ll often check whether the p-value (p for probability) of the results from their experiments is less than 5%. The smaller the p-value is, the less likely it is that their results are purely down to chance.

However, the p-value kinda means the exact reverse of what a lot of people assume it means.

When scientists talk about results being “significant at the 5% level”, say, it sounds like this means there’s a 95% chance of a real connection. In this cartoon’s case, it sounds like the scientists are 95% certain of a link between green jelly beans and acne.

Applicants for James Randi’s million dollar challenge are required to meet rather more stringent criteria, but it’s often expressed the same way. For instance, a dowser might have to psychically deduce which of several sealed containers is the one with water in, and repeat it a number of times, so that the p-value becomes very small. They want to be certain there’s really something going on, and it’s not just chance, before the money will be handed over.

But the intuitive idea of what the p-value means in these cases isn’t quite right.

Here’s what you actually need to do. Assume that there is no connection between the things being tested – jelly beans don’t affect acne, and all psychics are just guessing. Then, what are the odds of getting results at least as persuasive as the ones you saw, purely by chance?

That’s your p-value.

So, a p-value of 5% tells us something useful. It means that the results you’ve got are kinda iffy, given what you’d usually expect, if there’s no deeper underlying pattern there. You’d only expect to see results this skewed about 1 time in 20, if you’re relying on randomness. So maybe something’s up.

But if you do a whole bunch of tests, like the jelly bean scientists did, once in a while you will get some iffy results like that just by chance.

Now, clearly one thing this tells us is to be wary of data which has been cherry-picked, like the jelly bean journalists did. There were lots of negative results being ignored, and a single positive outcome highlighted. But the implications for how we assess probabilities more generally are, I think, more interesting.

In particular, it tells us that how likely something is doesn’t just depend on this one set of results. If a 5% p-value means “we’re 95% sure of this”, then this one study has entirely determined your estimate of the likelihood. It fails to take on board any information about how likely or unlikely something seemed before you started – and often this information is really important.

For instance, say you were studying differences between smokers and non-smokers, and the rate at which they get cancer. Any good analysis of data along these lines should easily pass a 5% significance test. It’s a highly plausible link, given what we already know, and 95% sounds like a significant under-estimate of the likelihood of a correlation between smoking and cancer.

But now imagine you’ve done a different test. This time, you just put a bunch of people into two groups, with no information about whether they smoke, or anything else about them, and flipped a coin to decide which group each person would go into. And imagine you get the same, seemingly convincing results as the smoking study.

Are you now 95% convinced that your coin-tossing is either diagnosing or causing cancer in people you’ve never met?

I hope you’re not. I hope you’d check your methodology, look for sources of bias or other things that might have crept in and somehow screwed up your data, and ultimately put it down to a bizarre fluke.

And it makes sense to do that, in this case, even despite the data. The idea that you could accurately sort people by cancer risk simply by flipping a coin is utterly ridiculous. We’d give it virtually zero probability to begin with. The results of your study would nudge that estimate up a little, but not much. Random fluke is still far more likely. If multiple sources kept repeating the experiment and getting the same persuasive results, over and over… then maybe, eventually, the odds would shift so far that your magic coin actually became believable. But they probably won’t.

And this idea of shifting the probability of something, rather than fixing it firmly based on a single outcome, is at the heart of Bayesian probability.

This is something the great Eliezer Yudkowsky is passionate about, and I’m totally with him. That link’s worth a read, though someday I’d like to try and write a similar, even more gently accessible explanation of these ideas for the mathematically un-inclined. He does a great job, but the arithmetic starts to get a bit overwhelming at times.

And if the thrill of counter-intuitive mathematics isn’t enough to convince you that this is fascinating and important stuff, read this. And then this.

Short version: a number of women have been convicted and jailed for murdering their children, then later released when somebody actually did some better statistics.

The expert witness for the prosecution in these trials estimated that the odds of two children in the same family both dying of cot death was 1 in 73,000,000. General population data puts the overall rate of cot deaths at around 1 in 8,500, so multiplying the 8,500s together gives the 1 in 73,000,000 figure for the chance of it happening twice. This was presented as the probability that the children could have died by accident, and thus it was assumed to be overwhelmingly likely that they were in fact deliberately killed.

But, as we learned with the cancer stuff earlier, we should consider these substantial odds against our prior assessment of how likely it is that these women would murder their children. This should start off minuscule, because very few women do murder their children. The fact that both their children died should make us adjust our likelihood estimate up a way – someone with two dead children is a more likely candidate for a child murderer than someone whose offspring are alive and well, after all – but it’s still far from conclusive.

Another way of expressing the central point of Bayesian probability is to consider the probability of A given B, for two events A and B. In this case, the odds of two children randomly picked from the population both dying of cot death may well be around 1 in 73,000,000 – but given that the children you’re considering both died in infancy, and were both siblings and so might have genetic or environmental factors in common, the cot death scenario becomes far more likely.

I wanted to expand on that last point some more, and touch on some other interesting things, but I’m hungry and you’re bored.

Ha. I said “briefly”. Classic.

Read Full Post »

Simpson’s Paradox

I suck at weekends. I’ve done nothing useful today. But something earlier reminded me about this, and for lack of anything else worth saying I’m going to talk about maths some more. I say bug humbah to your Hallowe’en malarkey. If you want spooky monsters and candy, go bother someone else. At my house, you get a lecture on algebra.

Simpson’s paradox is one of those really weird quirks of mathematics, which more people could do with understanding. It’s not even enormously complicated – the deep maths behind it can get pretty weird, but it’s really easy to appreciate how bizarrely counter-intuitive this stuff can be.

So, the paradox, and an example lifted straight from Wikipedia.

Some medical research happened a while back, into treatment for kidney stones. They took 700 people, split them into two groups, and tested a different treatment on each group. Treatment A worked on 273 out of 350 people in the first group, a success rate of 78%. Treatment B worked on 289 out of 350, or 83%.

So Treatment B works better, right?

Well, it turns out there are two different types of kidney stones. Broadly speaking, you can divide them into the “small” kind, and the “large” kind. So, even though Treatment B works better overall, maybe Treatment A is better for either small or large ones specifically. Right?

Well, half-right.

In fact, they found that Treatment A worked 93% of the time on small stones, while Treatment B worked 87% of the time. Meanwhile, with large stones, Treatment A hits 73% to Treatment B’s 69%.

So, for small kidney stones, Treatment A works demonstrably better than Treatment B. And for large kidney stones, A is still more successful than B. Treatment A actually works better in both individual cases.

But for kidney stones in general, Treatment B has a better overall success rate.

I’m a pretty intelligent person who studied mathematics more than anything else in life until I was 22, and I still don’t know how the fuck that works.

I mean, I understand all the maths behind it, it just still hurts my head. So now I’m going to go lie down. (This may also be related to the fact that it’s midnight now.)

Read Full Post »

Numbers and lies

Oh god I love this thread at FAIL Blog so much. You’re probably not a maths geek, so it might not mean much to you, but think about how much fun it is when people who don’t quite know what they’re talking about are convinced they do. Then apply that to a field of study in which absolute truth exists, and any answer or way of doing things is either definitely right or definitely wrong.

I know the actual calculus problem isn’t the point of the fail, but since when does that stop me? I solved it after a minute’s scribbling on a post-it, and got it right, because it’s not actually that complicated a question. What was more interesting was figuring out exactly how the over-confident engineering majors near the start of the discussion came up with their wrong answer. And I pretty quickly figured out what they’d done, and it’s quite funny. But only because I’m a real geek.

It’s interesting because they’re doing some moderately high-end maths, beyond the level most people would have studied to, but at the same time the mistakes they’re making indicate a fundamental lack of understanding about how differential calculus works. And that’s a perfectly okay thing to lack – I know a lot of fine, upstanding citizens with no concept of how differential calculus works at all, and I wouldn’t think to count it against them. But they have the sense not to go on internet message boards and try to teach people maths.

Also, I’ve started a new blog, which I’m planning to post to every weekday, as well as this one. It’s mostly a writing exercise for me, but I may start trying to get it noticed a bit too, now that it’s been going a week and I’m fairly sure my interest isn’t going to just fizzle out. It’s called The Daily Half-Truth, and the idea is to write weird and surreal news stories based on actual topical events, but with some strange and entirely fictional quirks. I’m having fun with it.

Okay, I think I’m done. Have fun noticing Hallowe’en. I’ll be probably not doing that.

Read Full Post »

Been fairly occupied with other things today (partly writing a book, partly watching one of my favourite Buffy episodes), so just have a few bits and pieces to half-heartedly report on.

I’ve also been scribbling more thoughts on this ongoing dialogue of sorts with Eric, my wacky Christian buddy. You can catch up with that in the comments a few posts down from here, or on his blog, but as a bonus DVD extra, here’s a quick thought that didn’t make it into that discussion, after I rewrote the whole piece in a somewhat different direction.

Eric said:

I also agree that we should not be subject to a tyrannical god. However, to suggest that an omnipotent god, no matter his disposition, actually OWES us something, it [sic] to have a view or [sic] humanity a bit higher than our actual position in this universe.

Which sounds unusual, coming from someone who – unless my Christian theology is severely out of wack – believes that the universe was created solely in order for us to live in it. That would seem to set our position pretty high. You could stroll into the Total Perspective Vortex with that attitude and walk out with your sanity, no sweat.

But even leaving that slight glibness aside, I disagree with the entire premise here. If a being who is all-powerful, who knows what’s going on down here, and who loves us unconditionally, has created the world and all the rules by which it runs, and created us and put us here, then yes, he has a responsibility to us, he owes it to us not to neglect us and allow needless suffering.

If, instead of creating the world as he did, he’d simply made Hell and then conjured a few souls into existence solely to throw them into the fire and watch them burn for eternity, would that be just? Would he not owe his creations a little more respect, compassion, and basic decency than that? As it happens, God is not that sadistic, but we’re still apparently bound by his contract and his rules which we never had a chance not to agree to. We’re human beings, you don’t get to walk all over us just because you’re bigger and more powerful. We deserve better. Has Spiderman taught you nothing about what traditionally comes with great power?

So, there’s that.

In exciting maths news (not an oxymoron, so shush), the Great Internet Mersenne Prime Search may have found a new Mersenne prime number – that is, a number into which no smaller number will divide exactly, and which takes the form 2n – 1, for some whole number n. If it checks out, it will be more than 12 million digits in length, far longer than the previous record-holder, which was just under 10 million digits long.

I get excited about these things. I’ll be following this story closely and providing regular updates.

Read Full Post »

%d bloggers like this: