Simpson's Paradox Interview Questions: Aggregates, Segments, and Causal Decisions

Simpson’s Paradox interview questions are rarely solved by saying “segment the data.” The reversal itself is arithmetic: an aggregate relationship can weaken, disappear, or reverse inside subgroups because the groups receive different weights. The interview decision is causal: what does the third variable represent, and which comparison answers the product question?

This guide builds one synthetic conversion example, traces the weighting mechanism, and shows why a confounder, mediator, and collider require different treatment.

Quick answer: Always calculate the overall and stratum-specific rates with their denominators. A reversal can occur when treatment groups have different mixes of high- and low-baseline strata. Do not automatically trust either the aggregate or segmented result. Draw the causal story: adjust for a relevant pre-treatment common cause when it blocks a backdoor path; define whether you want a total or direct effect before conditioning on a mediator; and avoid conditioning blindly on a collider.

What an interviewer wants to hear

A complete answer separates three tasks. First, reproduce the reversal without losing the denominators. Second, explain how unequal stratum weights create it. Third, decide what should be conditioned on using domain knowledge, treatment timing, and a causal graph.

The Stanford Encyclopedia of Philosophy defines Simpson’s Paradox as an association that emerges, disappears, or reverses after dividing a population into subpopulations. Its current review emphasizes that probability alone does not decide whether to condition on the third variable. Pearl’s analysis makes the same practical point: causal knowledge, rather than the visual surprise, determines the appropriate comparison.

Work a conversion table with a real reversal

Suppose an onboarding team compares two flows. “New” and “returning” are pre-exposure user types in this synthetic example.

User segment	Flow A conversions	Flow A rate	Flow B conversions	Flow B rate
New users	18 / 100	18%	17 / 100	17%
Returning users	72 / 100	72%	71 / 100	71%
Overall	90 / 200	45%	88 / 200	44%

That balanced table does not reverse: A wins within each segment and overall. Now change only the allocation. Send most new users to A and most returning users to B.

User segment	Flow A conversions	Flow A rate	Flow B conversions	Flow B rate
New users	162 / 900	18%	17 / 100	17%
Returning users	72 / 100	72%	639 / 900	71%
Overall	234 / 1,000	23.4%	656 / 1,000	65.6%

A is one percentage point higher in both strata, yet B is much higher overall. B receives 90% returning users, whose baseline conversion is high; A receives 90% new users. The aggregate compares different populations. The reversal is not a calculation error.

An interview answer should say what generated that allocation. If users were randomized within segment but the logged sample is badly imbalanced, investigate assignment and filtering. If product rules intentionally route returning users to B, the aggregate may describe the traffic actually served while failing to isolate the flow’s effect. Description and causation are separate questions.

Show the weights explicitly

An overall rate is a weighted average of stratum rates. Flow A’s rate is 0.9 × 18% + 0.1 × 72% = 23.4%. Flow B’s is 0.1 × 17% + 0.9 × 71% = 65.6%. The within-segment comparison holds weights fixed; the aggregate allows different weights.

This gives you a reliable debugging sequence: display counts, compute rates, compute the segment mix in each treatment, and reconstruct the aggregate from weighted stratum rates. If reconstruction fails, the table has a grain, join, or missing-data problem before it has a causal-inference problem.

Do not average the two segment percentages equally unless that matches the target population. Standardization requires declared target weights. Weighting 50/50 estimates performance in a population that is half new and half returning; weighting by next month’s expected traffic answers a different deployment question.

Decide whether the segment is a confounder

Suppose user type exists before exposure, affects which flow a user receives, and also affects conversion. It is a common cause of treatment and outcome. In an observational analysis, conditioning or standardizing on it may help identify the flow effect if the remaining causal assumptions hold.

That “if” matters. You still need consistency, positivity, a correct time order, and control of relevant confounding paths. If almost no returning user receives A, the stratum-specific estimate may be too noisy or unsupported. A regression coefficient does not manufacture overlap.

In a properly randomized experiment, large allocation differences across a strong baseline predictor may signal an implementation problem or chance imbalance. Verify assignment units and exclusions. Stratified estimation can improve precision, but do not rewrite the randomization story after seeing a favorable subgroup.

A mediator asks a different question

Now suppose the third variable is “completed tutorial,” measured after the assigned flow. The flow can change tutorial completion, which can change conversion. Conditioning on completion removes part of the pathway you may want to measure.

If the decision is “Which flow should we launch?”, the total effect usually includes the tutorial pathway. If the question is “What is the direct effect holding tutorial completion fixed?”, you need a direct-effect estimand and stronger assumptions. Comparing completers across flows is not automatically that effect because completion is selected after treatment.

Say the target out loud before segmenting. This prevents a common mistake: using a neat conditional table to answer a total-effect question it was designed to exclude.

A collider can create an association

Suppose both the assigned flow and a user’s unobserved motivation affect whether they contact support. Restricting analysis to support contacts conditions on a common effect, or collider. Within that selected group, flow assignment and motivation can become associated even if they were independent before selection.

The practical rule is not “never segment on a post-treatment variable.” It is “do not condition until you know what causal question the variable serves.” A support-only analysis can be useful descriptively for staffing, while being biased for the causal effect of the flow on conversion.

Aggregate and segmented results can serve different decisions

For capacity planning, the observed aggregate may be the number finance or operations needs. For causal choice between flows, unequal population mix can make that aggregate misleading. For a fairness audit, subgroup rates may reveal harms hidden by an average, but the relevant groups and metrics depend on the decision and cannot be selected only because they produce a reversal.

An interview recommendation should therefore have two sentences: “Descriptively, B has the higher observed overall conversion because it served more returning users. Under the stated pre-treatment user-type adjustment and adequate overlap, A is one point higher within each stratum; I would estimate a standardized treatment effect for the target launch population and audit why routing differed.”

Quantify uncertainty without losing the population question

The synthetic fixture makes the reversal obvious, but a real interview may give smaller cells. Report an interval for each contrast or fit a model that preserves the relevant interaction. A wide subgroup interval does not make the aggregate automatically preferable; it says the subgroup effect is uncertain. Pooling can improve precision while changing the estimand, so name that trade-off.

Do not test the overall result, test every segment, and keep whichever p-value supports the story. If the analysis was designed around a small set of product-critical segments, state that plan. If the reversal emerged during exploration, label it exploratory and validate it on new data or a predeclared follow-up. Multiple comparisons and sparse cells can manufacture unstable reversals.

For a binary outcome, counts matter because 8/10 and 800/1,000 have the same rate but very different uncertainty. If a segment has zero observations under one flow, the conditional causal comparison is unsupported there. You can redefine the target population, collect data that restores overlap, or use a model with explicit extrapolation assumptions; you cannot solve absent support by adding more decimal places.

Reproduce the reversal safely in SQL

At minimum, aggregate from one row per assigned unit. Compute COUNT(*) and SUM(converted) by flow and pre-treatment segment, then compute overall counts from the same eligible table. Avoid averaging already-aggregated percentages because segment sizes differ. Confirm that a user appears in one flow, that conversion is counted inside the same observation window, and that the segment value was known before assignment.

If event data contains retries or repeated conversions, build the user-level outcome first. Joining raw events directly to assignments can multiply rows differently by flow and create a false reversal. Reconcile each table back to unique assignments and expose excluded counts. The interviewer is often testing data-grain discipline as much as statistical vocabulary.

Common traps

“The subgroup result is always correct.” A subgroup variable can be a mediator, collider, measurement artifact, or irrelevant partition. Use the causal graph.

“The aggregate is wrong.” It may correctly describe realized volume. It can still fail to estimate the causal contrast.

“More dimensions will reveal the truth.” Unplanned slicing raises multiplicity, sparse-cell, and storytelling risks. Predefine important dimensions and validate follow-up findings.

“A reversal proves bias.” It flags a weighting or conditioning issue. Bias requires a claim about the data-generating process and estimand.

“A regression with controls settles it.” Controls encode conditioning choices. Check when each variable is measured and which paths it opens or closes.

A concise interview answer

Try this: “I’ll show the counts and denominators overall and by the proposed third variable. Then I’ll reconstruct the aggregate as weighted subgroup rates to prove where the reversal comes from. I won’t choose the aggregate or conditional result from the table alone. I’ll ask whether the variable precedes treatment, affects treatment and outcome, lies on the treatment pathway, or is caused by treatment and outcome. That determines whether adjustment targets the effect we want. Finally, I’ll standardize to a declared population, check overlap and uncertainty, and state which descriptive and causal conclusions differ.”

Candidate-reported practice prompts

Question	Skill to rehearse
How to Analyze Simpson's Paradox	Reproduce the reversal and inspect denominators.
Explain and Resolve Simpson's Paradox	Separate association from the causal decision.
Debug Metrics Computed Before Filtering	Verify the population and transformation order.
Clarify an Ambiguous Product Question and Define a Metric	Define the estimand before choosing a slice.
Resolve a Mismatch Between Regression and Classification Metrics	Check whether apparently conflicting summaries answer the same question.

Final checklist

Keep the counts next to the percentages. Verify the grain and denominators. Compute segment weights for each treatment. Draw a time-ordered causal graph. Decide whether the third variable is a pre-treatment common cause, mediator, collider, or only a reporting dimension. Define the target population and effect, check support and uncertainty, and explain why the business decision follows.

Sources and Further Reading

• Stanford Encyclopedia of Philosophy: Simpson’s Paradox — formal varieties, interpretation, and causal-inference discussion.
• Pearl: Understanding Simpson’s Paradox — why causal information determines whether to condition.
• Bickel, Hammel, and O’Connell: Sex Bias in Graduate Admissions — the classic Berkeley admissions analysis and its limits.
• Kievit et al.: Simpson’s Paradox in Psychological Science — practical examples and related statistical problems.

Research checked September 12, 2026. Both conversion tables are synthetic. Method sources support statistical and causal distinctions.