Articles
Propensity Score Interview Questions: Overlap, Balance, and Extreme Weights
Calculate propensity-score ATE and ATT weights by hand, then diagnose overlap, covariate balance, extreme weights, and effective sample size.
A propensity-score interview answer can be mathematically correct and causally empty. Computing inverse-probability weights is only the beginning: you need a target population, evidence that treated and untreated units overlap on measured baseline variables, a balance check after weighting, and a plan for the handful of observations that dominate the result. This guide gives you a four-person calculation and an extreme-weight stress test.
Research basis: Rosenbaum and Rubin introduced the propensity-score framework; Austin and Stuart discuss inverse-probability weighting, balance, and positivity diagnostics; Li and colleagues develop overlap weights for a different target population. Candidate reports: no employer-specific question frequency is claimed. Original exercise and inference: all outcomes and estimated scores below are fictional and too small for an actual causal study. The calculations demonstrate what an interview answer must check, not an estimated treatment effect to deploy.
What is a propensity score trying to balance?
For a binary treatment, the propensity score is the probability of receiving treatment conditional on observed pre-treatment covariates: e(X) = P(T = 1 | X). In a product example, treatment might be an invitation to a new feature, with covariates such as prior usage, region, tenure, and eligibility. It is not the probability that a user will convert, and it is not an outcome model. The original Rosenbaum–Rubin paper established the balancing role of the score under its assumptions.
Why estimate it? Treated users may be systematically different from controls before treatment. Weighting or matching can make observed baseline covariate distributions more comparable. That can reduce measured confounding if treatment assignment and potential outcomes are conditionally independent given those covariates. It cannot balance an unrecorded motivation variable merely because the score model has a high AUC. Predicting treatment assignment accurately is not the goal; credible covariate balance in the intended target population is.
Define the estimand first. The average treatment effect (ATE) asks about the source population under hypothetical treatment versus no treatment. The average treatment effect on the treated (ATT) asks about people who actually received treatment. An overlap-weighted effect asks about a population with substantial probability of either assignment. These can differ when effects vary by user type. Switching weights to solve a variance problem can change the question being answered.
The causal conditions deserve plain language. Consistency requires a meaningful, sufficiently defined treatment. Positivity requires a nonzero chance of both assignments in relevant covariate strata. Conditional exchangeability requires no unmeasured confounder after the chosen baseline covariates. If a premium-only feature was impossible for free-tier users, their treatment probability is effectively zero; weighting cannot invent treated free-tier observations.
Calculate ATE and ATT weights on four rows
Original practice fixture: two treated users, T1 and T2, and two untreated users, C1 and C2. Their estimated propensity scores are 0.8, 0.6, 0.2, and 0.4. Observed binary outcomes are 1, 0, 0, and 1. These are illustrative fitted probabilities; no model was trained on this tiny table.
For unstabilized ATE inverse-probability weighting, a treated user's weight is 1/e(X) and a control's is 1/(1-e(X)). Thus T1 weighs 1.25, T2 about 1.667, C1 1.25, and C2 about 1.667. Using normalized weighted means within each group, the treated mean is 1.25/(1.25+1.667) ≈ 0.429. The control mean is 1.667/(1.25+1.667) ≈ 0.571; the illustrative contrast is about −0.143.
For one common ATT weighting convention, treated users receive weight one and controls receive e(X)/(1-e(X)). T1 and T2 each weigh one; C1 weighs 0.25; C2 weighs about 0.667. The treated outcome mean is 0.5; the weighted control mean is 0.667/(0.25+0.667) ≈ 0.727. The illustrative ATT contrast is about −0.227. The different results are expected because the weights emphasize different populations. They do not show that either effect is identified in four observations.
| User | Assignment | Estimated e(X) | Outcome | ATE weight | ATT weight |
|---|---|---|---|---|---|
| T1 | Treated | 0.8 | 1 | 1.25 | 1 |
| T2 | Treated | 0.6 | 0 | 1.667 | 1 |
| C1 | Control | 0.2 | 0 | 1.25 | 0.25 |
| C2 | Control | 0.4 | 1 | 1.667 | 0.667 |
If the interviewer asks whether these are “the” weights, specify the scheme. Stabilized weights multiply by marginal treatment probabilities; matching and overlap weights are different; normalization and variance estimation also matter. State the formula and target population before comparing software output. The Austin–Stuart guidance treats balance and weight diagnostics as part of the analysis, not optional decoration.
What do overlap plots reveal before an effect estimate?
Plot estimated scores separately for treated and control units, using the same axis and enough detail to see sparse tails. If treated users cluster near 0.9 and controls near 0.1, many weighted comparisons depend on a small number of unusual users. Check overlap within important strata and across time, not only in a pooled histogram. An overall overlap plot can hide that one region has no treated controls or that eligibility changed midway through the study.
Do not confuse a low propensity with a bad model. A genuine policy may assign a feature to almost no users of a certain type. That is a design problem for an ATE over all users, not something a flexible classifier can fix. Ask whether the estimand should be restricted to an eligible or overlapping population, whether new data can be collected, or whether a randomized rollout is possible.
The numerical score itself is estimated. Calibrated treatment probabilities matter for inverse weights, but a high treatment-prediction metric does not establish balance. Choose baseline covariates based on causal reasoning and availability before assignment; avoid post-treatment variables such as engagement measured after the feature invitation. If you condition on a post-treatment mediator or collider, a beautiful balance table can coexist with bias.
How do you assess balance after weighting?
Compare baseline covariate distributions before and after adjustment. Standardized mean differences are common because they express imbalance relative to a scale instead of producing sample-size-sensitive significance tests. Report them for each important covariate and for interactions or nonlinear terms relevant to treatment and outcome. Inspect weighted means, spread, and plots; a single average can conceal tail mismatch. A rough absolute standardized-difference threshold can be a screening convention, not a universal proof of exchangeability.
For example, suppose the treated group has 70% prior weekly users and controls have 30%. If the weighted control percentage rises to 68%, that is evidence the observed prior-usage variable is more balanced. It does not establish balance for unmeasured intent or prior revenue if those were not included. Check all prespecified baseline covariates, including features not selected by the propensity model if they are important to the outcome. Balance is a property of the weighted sample, not just the score's predictive fit.
Examine weight distribution and effective sample size. The common Kish diagnostic is (sum of weights)^2 / sum of squared weights. Four nominal observations with weights [1,1,1,20] have effective sample size 23²/(1+1+1+400) = 529/403 ≈ 1.31. One observation dominates. Report maximum and upper quantiles by assignment group as well as ESS. A treatment-effect interval based on nominal N = 4 would conceal that fragility; use an uncertainty method appropriate to the full propensity-estimation and weighting procedure.
What should you do with extreme weights?
An ATE control unit with e(X) = 0.98 receives weight 1/(1-0.98) = 50. It represents many high-propensity people who usually receive treatment. Its outcome can swing the estimate. First verify coding, treatment definition, eligibility, and the propensity model. Then show sensitivity to justified changes: stabilized weights, truncation at a prespecified percentile, trimming nonoverlap, or changing to overlap weights. Do not silently cap weight 50 at 10 and still describe the result as the same estimand without qualification.
Trimming removes units and generally changes the population. An ATT analysis may retain treated users and choose controls comparable to them, but if some treated users have no credible controls, even ATT for all treated users is hard to identify. Overlap weighting emphasizes individuals who could plausibly receive either assignment; for binary treatment, treated units get weight 1-e(X) and controls get e(X) under a common formulation. The Li–Morgan–Zaslavsky paper defines this target and bounded weights. It is useful, but its target is not the entire source population's ATE.
Sensitivity tables should include the estimand, retained population, overlap range, weight quantiles, ESS, balance diagnostics, effect estimate, and uncertainty interval. If the sign flips after a modest, defensible trim, report that instability. Do not choose the trim that makes the result favorable. If positivity fails structurally, a different research question or experiment is the honest recommendation.
What remains unidentified after excellent balance?
Propensity scores adjust for measured pre-treatment covariates. An unobserved factor that affects both assignment and outcome can still bias the estimate. In a feature-launch study, motivated users may seek invitations and also be more likely to retain, even after controlling for recorded tenure and usage. Balance on those recorded fields does not test motivation. Use domain knowledge, negative-control outcomes or exposures where valid, and sensitivity analysis to communicate how strong an omitted confounder would need to be to change the conclusion.
Temporal order matters. The propensity model must use information available before treatment assignment. If a user enters the data only after surviving a period of activity, selection can introduce immortal-time or conditioning bias. If treatment timing varies, a single binary score may not represent the treatment process. Define eligibility, assignment time, follow-up, censoring, and outcome window before fitting any classifier.
A concise interview answer
“I would define treatment, assignment time, eligibility, and whether we want ATE, ATT, or an overlap-population effect. I would estimate treatment probability from prespecified pre-treatment covariates, inspect treated/control overlap, then weight using a formula aligned with the estimand. I would check weighted covariate balance, extreme weights, and effective sample size before interpreting the outcome contrast. Trimming or overlap weights may improve stability but change the target population. Good measured balance does not rule out unmeasured confounding.”
Practice the Explain Propensity Score Matching and Assess Covariate Balance by explaining why the four-row ATE and ATT estimates differ, then replace C2's score with 0.98. The decisive answer is not just “weight 50”; it is what that single weight means for overlap, precision, and the claim you can responsibly make.