Simpson's paradox

Welcome to Episode 14 of our Causal Inference course! In this session, we unravel the mysteries of Simpson's Paradox, a statistical phenomenon where a trend that appears in different groups of data disappears or even reverses when these groups are combined. We will explore classic examples to understand how this counterintuitive situation arises and connect it directly to the crucial concept of confounding, which we've discussed previously. You will learn that resolving the paradox isn't a simple statistical choice but requires deep causal reasoning. By the end of this episode, you'll be able to identify potential instances of Simpson's Paradox and understand why asking 'why' is essential before drawing conclusions from data.

Check your understanding

These are the same multiple-choice questions you will see in the Quiz section after you listen to the episode. Use them here to preview or review the answers.

What is the defining characteristic of Simpson's Paradox?

  1. A correlation that is strong in subgroups becomes weak when the groups are combined.
  2. A trend or relationship observed in separate groups of data reverses when the groups are combined into a single dataset.
  3. Two variables are causally linked in subgroups, but only correlated in the aggregate data.
  4. Randomized controlled trials and observational studies yield opposite conclusions.
  5. The average treatment effect (ATE) is positive, but the effect is negative for every individual.

In the classic kidney stone example, what is the primary role of 'stone size' that causes Simpson's Paradox?

  1. It is a mediator in the causal pathway between treatment and recovery.
  2. It is a consequence of the treatment chosen by the doctor.
  3. It is a confounding variable that influences both the choice of treatment and the success of the outcome.
  4. It is a collider variable that creates a spurious association when conditioned on.
  5. It is an instrumental variable for treatment selection.

How should one decide whether to use aggregated or disaggregated (stratified) data when a paradox is observed?

  1. Always use the disaggregated data, as it provides a more granular view.
  2. Always use the aggregated data, as it represents the total population.
  3. Choose the dataset that shows the strongest statistical significance (lowest p-value).
  4. The decision depends on the underlying causal structure of the problem, particularly whether the grouping variable is a confounder.
  5. Perform a statistical test to see which dataset has less variance.

Which of the following previously discussed concepts is most essential for understanding and resolving Simpson's Paradox?

  1. Propensity Score Matching
  2. Confounding
  3. Regression Discontinuity Design
  4. Difference-in-Differences
  5. Causality

A company finds that its new marketing strategy has a lower overall success rate than the old one. However, for both small and large customers, the new strategy has a higher success rate. What are plausible explanations consistent with Simpson's Paradox?

  1. The data was recorded incorrectly.
  2. The new strategy was disproportionately applied to a customer segment that is harder to convert (e.g., large customers).
  3. The old strategy was mostly used on a customer segment that is easier to convert (e.g., small customers).
  4. The new strategy is fundamentally flawed and should be abandoned immediately based on the overall numbers.
  5. The marketing team for the new strategy was less experienced.

Suggested next

Related episodes that are a natural follow-on.

  • Mediation (statistics)

    Welcome to the final episode of our Causal Inference course! Having mastered how to identify *if* a cause has an effect and *how much* of an effect it has, we now turn to the crucial questions of *how* and *why*. This episode introduces mediation ana… Welcome to the final episode of our Causal Inference course! Having mastered how to identify *if* a cause has an effect and *how much* of an effect it has, we now turn to the crucial questions of *how* and *why*. This episode introduces mediation analysis, a powerful statistical technique for understanding the mechanisms or pathways through which a cause produces its effect. We will explore how to decompose a total causal effect into its direct and indirect components, using concepts like Directed Acyclic Graphs that you've learned previously. By the end, you'll be able to look beyond the overall effect and uncover the intricate story of causality that unfolds between a treatment and an outcome, providing deeper insights for policy and decision-making.

  • Instrumental variables estimation

    Welcome to the ninth episode of our Causal Inference course! In previous lessons, we've explored methods like Propensity Score Matching and Difference-in-Differences to handle confounding variables we can observe. But what happens when the confounder… Welcome to the ninth episode of our Causal Inference course! In previous lessons, we've explored methods like Propensity Score Matching and Difference-in-Differences to handle confounding variables we can observe. But what happens when the confounder is hidden or unmeasurable, like innate 'ability' or 'motivation'? This episode introduces Instrumental Variables (IV) estimation, a powerful and clever technique to uncover causal effects even in the presence of such unobserved confounding. You will learn what an instrumental variable is, the two crucial conditions it must satisfy—relevance and the exclusion restriction—and the two-stage logic that allows it to isolate the causal impact of a treatment. By the end, you'll understand how IV provides a solution to one of the toughest problems in causal inference.

  • Average treatment effect

    Welcome to episode 13 of our Causal Inference course. In this session, we introduce a cornerstone concept: the Average Treatment Effect, or ATE. We'll explore how the ATE provides a single, powerful number to summarize the overall impact of an interv… Welcome to episode 13 of our Causal Inference course. In this session, we introduce a cornerstone concept: the Average Treatment Effect, or ATE. We'll explore how the ATE provides a single, powerful number to summarize the overall impact of an intervention across an entire population. Building on our understanding of counterfactuals and randomized controlled trials, you will learn the formal definition of the ATE and why randomization is the gold standard for estimating it. We will also differentiate the ATE from more specific measures like the Average Treatment Effect on the Treated (ATT), discussing when and why these different quantities are important. This episode will equip you with the foundational language for quantifying causal impact.

  • Regression discontinuity design

    Welcome to Episode 10 of our Causal Inference course! This episode introduces Regression Discontinuity Design (RDD), a powerful quasi-experimental method. You will learn how RDD leverages sharp cutoffs in assignment rules—like a minimum test score fo… Welcome to Episode 10 of our Causal Inference course! This episode introduces Regression Discontinuity Design (RDD), a powerful quasi-experimental method. You will learn how RDD leverages sharp cutoffs in assignment rules—like a minimum test score for a scholarship—to create a natural experiment. We'll explore the core intuition behind RDD, distinguishing it from other observational methods by showing how it mimics a randomized controlled trial for subjects right around the threshold. By the end, you will understand the key assumptions that make RDD a credible tool for estimating causal effects, its main variations (Sharp vs. Fuzzy), and its real-world applications in policy, economics, and healthcare.

  • Propensity score matching

    Welcome to Episode 7 of our Causal Inference course! In previous episodes, we established that Randomized Controlled Trials (RCTs) are the gold standard for causal inference, but what can we do when they aren't feasible? This episode introduces Prope… Welcome to Episode 7 of our Causal Inference course! In previous episodes, we established that Randomized Controlled Trials (RCTs) are the gold standard for causal inference, but what can we do when they aren't feasible? This episode introduces Propensity Score Matching (PSM), a powerful technique for analyzing observational data. You'll learn how PSM helps us mimic the conditions of an RCT by balancing covariates between treatment and control groups. We'll explore what a propensity score is, how it's calculated, and the different ways we can use it to match individuals. By the end, you'll understand how PSM addresses confounding and selection bias, moving us closer to making causal claims from non-experimental data.

Often studied before

Episodes that tend to come earlier on similar paths.

  • Correlation does not imply causation

    Welcome to the third episode of our Causal Inference course! Building on our understanding of causality, this episode tackles one of the most fundamental principles in statistics and science: 'Correlation does not imply causation.' We will explore wh… Welcome to the third episode of our Causal Inference course! Building on our understanding of causality, this episode tackles one of the most fundamental principles in statistics and science: 'Correlation does not imply causation.' We will explore what correlation is and, more importantly, why the simple fact that two trends move together doesn't prove that one causes the other. Using relatable examples, from ice cream sales and shark attacks to firefighters and fire damage, we will uncover the common logical traps people fall into. This crucial lesson will highlight the dangers of jumping to conclusions and set the stage for the more advanced methods we'll use later in the course to uncover true causal relationships.

  • Average treatment effect

    Welcome to episode 13 of our Causal Inference course. In this session, we introduce a cornerstone concept: the Average Treatment Effect, or ATE. We'll explore how the ATE provides a single, powerful number to summarize the overall impact of an interv… Welcome to episode 13 of our Causal Inference course. In this session, we introduce a cornerstone concept: the Average Treatment Effect, or ATE. We'll explore how the ATE provides a single, powerful number to summarize the overall impact of an intervention across an entire population. Building on our understanding of counterfactuals and randomized controlled trials, you will learn the formal definition of the ATE and why randomization is the gold standard for estimating it. We will also differentiate the ATE from more specific measures like the Average Treatment Effect on the Treated (ATT), discussing when and why these different quantities are important. This episode will equip you with the foundational language for quantifying causal impact.

  • Regression discontinuity design

    Welcome to Episode 10 of our Causal Inference course! This episode introduces Regression Discontinuity Design (RDD), a powerful quasi-experimental method. You will learn how RDD leverages sharp cutoffs in assignment rules—like a minimum test score fo… Welcome to Episode 10 of our Causal Inference course! This episode introduces Regression Discontinuity Design (RDD), a powerful quasi-experimental method. You will learn how RDD leverages sharp cutoffs in assignment rules—like a minimum test score for a scholarship—to create a natural experiment. We'll explore the core intuition behind RDD, distinguishing it from other observational methods by showing how it mimics a randomized controlled trial for subjects right around the threshold. By the end, you will understand the key assumptions that make RDD a credible tool for estimating causal effects, its main variations (Sharp vs. Fuzzy), and its real-world applications in policy, economics, and healthcare.

  • Directed acyclic graph

    Welcome to Episode 11 of the Causal Inference course. In this episode, we introduce Directed Acyclic Graphs (DAGs), a powerful visual framework for mapping out our causal assumptions. You will learn what DAGs are, what their components signify, and h… Welcome to Episode 11 of the Causal Inference course. In this episode, we introduce Directed Acyclic Graphs (DAGs), a powerful visual framework for mapping out our causal assumptions. You will learn what DAGs are, what their components signify, and how they provide a formal language for reasoning about complex causal systems. We will explore how DAGs make abstract concepts like confounding concrete by identifying 'backdoor paths' and how they warn us against common pitfalls like selection bias through structures known as 'colliders'. This episode will equip you with the foundational knowledge to translate your understanding of a problem into a formal causal model, guiding your choice of statistical methods and variables for analysis.

  • Observational study

    Welcome to the sixth episode of our Causal Inference course. This time, we step away from the experimental ideal to explore the world of Observational Studies. Where Randomized Controlled Trials (RCTs) actively assign treatments, observational studie… Welcome to the sixth episode of our Causal Inference course. This time, we step away from the experimental ideal to explore the world of Observational Studies. Where Randomized Controlled Trials (RCTs) actively assign treatments, observational studies passively observe the world as it is. We will define what these studies are, contrast them with RCTs, and delve into their primary challenge: confounding. You'll learn about the main types of observational designs—cohort, case-control, and cross-sectional—and understand their unique strengths and weaknesses. This episode builds a critical foundation for why more advanced techniques, which we will cover later, are necessary to draw causal conclusions from non-experimental data.