We often evaluate the success of medical treatments or social programs by how much of the population they help. Like, suppose we’re treating a disease that afflicts both people and cats, and among 1 cat and 4 people we treat, the cat and 1 person recover and 3 people die. And of 4 cats and 1 person we don’t treat, three of the cats recover while the person and 1 cat die. In the real world, these numbers might be more like 300 and 100, or whatever, but we’ll keep them small so they’re easier to keep track of. So, in our sample, 100% of treated cats survive while only 75% of untreated cats do, and 25% of treated humans survive while 0% of untreated humans do. Which makes it seem like the treatment improves chances of recovery. Except that if we aggregate the data, among all people and cats treated, only 40% survive, while among all people and cats left on their own, 60% recover. Which makes it seem like the treatment reduces chances of recovery. So which is it? This is an illustration of Simpson’s paradox , a statistical paradox where it’s possible to draw two opposite conclusions from the same data depending on how you divide things up, and statistics alone cannot help us solve it – we have to go outside statistics and understand the causality involved in the situation at hand. For example, if we know that humans get the disease more seriously and are therefore more likely to be prescribed treatment, then it can make sense that fewer individuals that get treated survive, even if the treatment increases the chances of recovery, since the individuals that got treated were more likely to die in the first place.
On the other hand, if we know that humans, regardless of how sick they are, are more likely to get treated than cats because no one wants to pay for kitty healthcare, then the fact that 4 out of 5 humans died while only 1 in 5 cats died suggests that, indeed, the treatment may be a bad choice. So if you’re doing a controlled experiment, you need to make sure to not let anything causally related to the experiment influence how you apply your treatments, and if you have an uncontrolled experiment, you have to be able to take those outside biases into account. As a more tangible example, Wisconsin has repeatedly had higher overall 8th grade standardized test scores than Texas, so you might think Wisconsin is doing a better job teaching than Texas. However, when broken down by race – which, via entrenched socioeconomic differences is a major factor in standardized-test scores – Texas students performed better than Wisconsin students on all fronts: black Texas students scored higher than black Wisconsin students, and likewise with hispanic and white students. The difference in the overall ranking is because Wisconsin has proportionally far fewer black and hispanic students and proportionally more white students than Texas – so the takeaway should not be that Wisconsin has better education than Texas! Just that it has (proportionally) more socioeconomically advantaged people. In some situations there’s also a nice graphical way to picture Simpson’s paradox: as two separate trends that each go one way, but the overall trend between the populations goes the other way. Like, maybe more money makes people sadder, and more money makes cats sadder, but if cats are both much happier and richer than people to start with, the overall trend appears, incorrectly, to be that more money makes you happier. Of course, you can also misinterpret this graph to show that, overall, more money makes you a cat, which I think helps illustrate very well the ability to lie or reach incorrect conclusions by blindly using statistics without context! Of course, this is not to say that statistics are always going to be paradoxical or confusing – it’s quite possible that everything will just make sense from the get-go, like if people and cats both get sadder when you give them more money, and cats are both poorer and happier than people, then the overall trend is no longer paradoxical: more money = more sadness. But it’s important to be aware that paradoxes like Simpson’s paradox are possible, and we often need more context to understand what a statistic actually means. Given the mathiness of my videos, it may not surprise you to hear that I get a lot of practice with math & physics problems while working on them, and this video’s sponsor, Brilliant.org, wants to help you stay sharp on your problem solving, too! (since, unfortunately, watching videos doesn’t require as much problem solving). Practice is pretty much the best way to really get to know a subject, and Brilliant.org is ready to give you plenty with premium courses in probability, logic, and math for quantitative finance. Plus addictive puzzles: for example, “if half of the earth is blown away by the impact of a comet, what happens to the orbit of the moon?” It almost sounds like a MinutePhysics video… but you’re going to have to go to Brilliant.org to solve it (or one of their many others) – and when you do, use the URL brilliant.org/minutephysics to let Brilliant know you came from here.