Simpson's Paradox In Practice

A groundbreaking medical treatment works better for men and simultaneously works better for women. When that data set is combined, is it possible for that very same treatment to perform worse for humans overall? Your intuition would immediately argue against it. The phenomenon seems logically impossible, contradictory, and undoubtedly an error. Our human brains are predisposed to believe that the whole must always equal the sum of its parts, and while this is mostly true, there stands a paradox that defies that intuition.

If this drug is superior for every subgroup of a population, such as gender or ethnicity, common sense dictates it must be superior for the population as a whole. Yet, the unforgiving laws of mathematics and statistics say otherwise. This unsettling irregularity proves that data can be perfectly accurate, meticulously collected, and still lead us to a completely false conclusion.

Simpson’s paradox is a statistical phenomenon where trends appear in different groups of data, but disappear or reverse when those groups are combined. This framework does not solely exist in theory, but practical examples lie in college admissions, medicine and sports. This article aims to investigate the particular causes of the paradox, its past applications and what it teaches us about the dangers of aggregating data without understanding the underlying structure.

To read more, click here.