Heterogeneous Treatment Effect Interview Questions: Segments, Interaction Tests, and Discovery Bias

Heterogeneous treatment effect interview questions ask whether an intervention works differently across people, not whether one subgroup crossed a significance threshold and another did not. Define the treatment effect within each baseline segment, compare those effects directly, and separate exploratory segment discovery from confirmatory evidence.

For product data scientist and causal machine learning interviews, the strongest answer combines an estimand, an interaction test, and an honest validation plan. A model that predicts outcomes well is not automatically a model that identifies who benefits from treatment.

This article adds a worked counterexample and a discovery protocol; it does not promise a particular employer’s questions or selection process.

What is a heterogeneous treatment effect?

A heterogeneous treatment effect is variation in a causal effect across a population. A conditional average treatment effect, or CATE, averages the difference between treated and untreated potential outcomes among units with specified baseline characteristics. It is not an observed individual’s before-and-after change.

Write the target as tau(x) = E[Y(1) - Y(0) | X=x]. Here Y(1) and Y(0) describe outcomes under treatment and control; only one is observed for each unit. Predicting those missing counterfactuals requires identification assumptions, not merely a flexible regression algorithm.

Susan Athey and Guido Imbens distinguish causal-effect estimation from ordinary outcome prediction in Recursive Partitioning for Heterogeneous Causal Effects. Their work motivates honest separation between finding partitions and estimating effects. It does not turn arbitrary post-hoc slicing into valid inference.

In an interview, first ask what “works better” means. A four-percentage-point conversion increase, a 20% relative lift, and an increase in log odds are different effect scales. The segment comparison must use the scale relevant to the decision.

Also define the unit: user, account, household, or marketplace cluster. If assignment occurs at account level but the table contains repeated user events, event counts do not create independent experimental units. Segment precision must reflect the actual assignment and dependence structure.

The significant-versus-nonsignificant trap

Consider this original synthetic experiment. Segments A and B are disjoint baseline groups; treatment and control observations are independent within each group. Each row below gives one arm’s size and conversion count.

Segment	Control	Treatment	Estimated absolute effect
A	200 / 1,000 = 20%	240 / 1,000 = 24%	+4 percentage points
B	20 / 100 = 20%	24 / 100 = 24%	+4 percentage points

Using a simple unpooled normal approximation, A’s standard error is about 1.85 percentage points and its two-sided p-value against zero is 0.0306. B’s standard error is about 5.85 percentage points and its corresponding p-value is 0.4942.

At a 5% threshold, A is significant and B is not. Nevertheless, their estimated treatment effects are identical. The estimated interaction, tau_A - tau_B, is zero percentage points. Its approximate standard error is 6.14 percentage points under the stated independence assumptions; the zero-versus-zero-difference z-statistic is zero.

This numerical exercise illustrates the distinction described by Andrew Gelman and Hal Stern in their significance-comparison paper. It is not evidence that the true segment effects are equal. Wide uncertainty can accommodate substantial differences even when the estimated difference is zero.

The answer to “Should we launch only to A?” is therefore not “yes, because only A is significant.” You need a direct effect comparison and a decision analysis. A business could still prefer A for costs or harms, but that is a separate reason from demonstrating statistical heterogeneity.

How do you test an interaction?

For a binary baseline segment S and randomized treatment T, an illustrative linear probability model is:

E[Y | T, S] = beta0 + betaT*T + betaS*S + delta*T*S

When S=0, the treatment effect is betaT. When S=1, it is betaT + delta. The interaction coefficient delta is the difference between those absolute effects. The hypothesis of equal effects on this scale is delta=0.

The formula interface in statsmodels OLS documentation supports specifying regression models; the appropriate standard errors and causal interpretation remain your responsibility. For binary outcomes, do not silently assume constant residual variance. With clustered assignment, use inference appropriate to the cluster design.

If you use logistic regression instead, the product-term coefficient describes an interaction on the log-odds scale. It is not automatically the difference in percentage-point effects. Obtain predicted probabilities under both assignments, average over a defined population, and calculate the contrast required by the question.

For a continuous characteristic, specify the comparison more carefully. A linear T*age term assumes a particular slope pattern. It does not establish that the effect changes abruptly at age 35. If you search for a threshold after seeing the data, the threshold itself becomes part of the discovery procedure.

With many segments, an omnibus interaction test can answer whether some variation is present before a family of narrower contrasts. It does not identify which comparison matters operationally. State the family of hypotheses and follow-up plan rather than displaying a list of unadjusted winners.

Which segments are safe to define?

Baseline characteristics, measured before treatment, are the natural starting point: geography at assignment, pre-experiment tenure, prior spending, or a frozen baseline risk score. Their measurement and missingness should not depend on the intervention.

“Users who clicked the new feature” is different. Treatment can change clicking, so conditioning on clickers compares selected populations rather than the original randomized groups. Calling this a segment does not preserve the original causal contrast.

Similarly, “active users during the experiment” can be post-treatment selection if the intervention changes activity. An apparent segment win may reflect who entered the analysis rather than who benefited. Define eligibility before assignment, and keep the main analysis tied to that eligible population.

For overlapping segments, remember that effect estimates can be correlated. New users in one geography may appear in several reported cuts. You cannot add independent variances or use an independence-based multiple-testing probability without checking that assumption.

An interview response should also mention support. If almost no treated observations occur in a segment of an observational dataset, a CATE estimate may rely heavily on extrapolation. A sophisticated learner cannot manufacture overlap. Restrict the claim or redesign the data collection before recommending a targeting policy.

Why does discovering the best segment exaggerate its effect?

Searching many noisy estimates selects unusually favorable noise along with any genuine signal. The largest observed lift is not an unbiased forecast of what the chosen segment will achieve next time.

For a deliberately simplified calculation, suppose you run 100 independent null tests, each at significance level 0.05. The probability of at least one false positive is 1 - 0.95**100, about 99.41%. These are original calculated values under independence, not a universal rate for correlated product segments.

The practical problem is broader than counting p-values. You may choose the metric, threshold, exclusion rule, outcome window, and model after seeing results. Each adaptation can enlarge the effective search. Declaring only five final segment tests does not erase the exploration that produced them.

Freeze the segment rule and decision metric before opening confirmation results. Repeatedly checking the holdout converts it into another discovery dataset.

Use this interview protocol:

1. Explore candidate segments or learn effect patterns on discovery data.
2. Freeze the segmentation rule, population, metric, scale, and testing family.
3. Estimate the chosen effects on independent confirmation data with the specified inference procedure.
4. Report uncertainty, multiplicity treatment, and any departures from the plan.
5. Evaluate the resulting policy on new evidence when deployment changes the population or treatment allocation.

These steps reduce a specific source of selection bias; they do not guarantee a positive result. A failed confirmation is informative. Do not keep revising the threshold on the same holdout until it passes.

Where do causal trees and meta-learners fit?

A causal tree searches for partitions with different treatment effects. A T-learner fits separate outcome models for treated and control units; an S-learner incorporates treatment into one outcome model. The important interview distinction is how they estimate a contrast, not whether you can recite every algorithm name.

Flexible effect models can be helpful when baseline covariates are numerous, but they still need justified treatment assignment, overlap, careful sample separation, and suitable evaluation. An outcome model that is accurate mainly because geography predicts baseline conversion can fail to capture treatment-effect differences.

The honest-estimation idea in Athey and Imbens’ paper uses different information for choosing a partition and estimating its effects. This is more specific than saying “use cross-validation.” Ordinary prediction validation measures a different target from causal-effect accuracy, because individual counterfactual differences are not directly observed.

When asked about cross-fitting, explain the role of out-of-fold nuisance estimates and the estimand-specific procedure. Do not imply that an out-of-fold prediction automatically makes every selected subgroup confidence interval valid. Discovery, nuisance fitting, and policy evaluation have related but distinct leakage boundaries.

Turn a subgroup finding into a targeting decision

Suppose one baseline segment has a larger credible uplift. The deployment question still includes treatment cost, eligible reach, uncertainty, guardrail harms, and whether serving only that group changes spillovers or marketplace behavior.

Compare expected incremental value under a proposed policy with a relevant alternative, such as treat nobody, treat everyone, or the existing rule. Ranking by estimated uplift alone can choose expensive or harmful actions. If costs differ across segments, a smaller conversion effect can still create greater net value.

Avoid interpreting a nonsignificant estimate as zero benefit. Conversely, a positive estimate with broad uncertainty is not a guaranteed profitable segment. State the downside tolerance and what evidence would change the launch decision.

A useful decision memo has four sentences: define the eligible baseline group; quantify the effect contrast and interval; label whether the group was discovered or prespecified; recommend confirmation, limited rollout, or no targeting with the reason. That is clearer than a heatmap covered in significance stars.

Before presenting the memo, check whether the segment rule can be applied at decision time. A targeting model built from future renewal behavior may describe retrospective differences beautifully while being impossible to use before treatment. Freeze the available baseline feature set as well as the subgroup threshold.

Finish with the comparison you actually tested

The central rule is simple: test the difference of effects, not the difference of significance labels. Then protect that comparison from post-treatment selection and data-driven discovery bias.

Begin with the effect scale and segment definition; finish with a confirmation plan and a policy decision whose assumptions are visible.

Sources and Further Reading

• Andrew Gelman and Hal Stern: significance versus nonsignificance
• Susan Athey and Guido Imbens: Recursive Partitioning for Heterogeneous Causal Effects
• statsmodels: formula-based OLS reference

Research note: Sources checked October 3–4, 2026. The counts, normal-approximation p-values, interaction calculation, and independent-test probability are original synthetic exercises, checked with NumPy 2.5.3 and SciPy 1.18.1. They are not observed product outcomes or a power guarantee.