Articles
Synthetic Control Interview Questions: Donor Pools, Pre-Trends, and Placebo Tests
Practice synthetic control with a donor-weighted city case, pre-fit, post gaps, spillovers, placebo tests, and limits of causal interpretation.
A synthetic-control case can look persuasive because two lines align before launch and separate afterward. An interview answer earns trust by asking whether the donor markets could represent the treated market without the launch, whether they were themselves affected, and whether similarly large gaps appear when the same method is applied to untreated markets. This guide provides a small numerical case and a falsification checklist.
Research basis: Abadie's methodological review and the original Abadie–Diamond–Hainmueller study describe synthetic controls as weighted comparisons built from pre-intervention data and discuss fit and placebo inference. Candidate reports: none are used to claim an employer uses this case. Original exercise and inference: the market data and launch story below are fictional; their gaps are descriptive until the identifying assumptions are defended.
When is synthetic control the right design?
Imagine one city launched a new delivery promise on July 1, while other cities did not. There is no randomized control group. You have a long panel of city-level weekly orders before and after launch. A synthetic control forms a weighted combination of untreated cities whose pre-launch outcomes and relevant predictors resemble the launch city. After launch, the treated-minus-synthetic gap estimates what changed relative to that constructed counterfactual if untreated donors trace the launch city's no-treatment path.
Start with the estimand: effect on the treated city over specified post-launch weeks, for a defined outcome such as completed orders or contribution margin. The method does not automatically estimate the effect for every possible city or customer. If the launch city was chosen precisely because it was already accelerating, a simple before/after difference is confounded. Synthetic control attempts to model that trajectory from comparable donors; it does not make selection bias vanish by definition.
The Abadie review emphasizes feasibility and situations where the approach can fail. A useful interview answer asks whether enough pre-period observations exist to distinguish stable relationships from a coincidental fit, whether treated and donor outcomes share measurement, and whether the treated unit lies within the donors' plausible combination. If the treated city is structurally outside the donor range, nonnegative weights summing to one may not reproduce it well. A poor pre-fit weakens the post-gap interpretation before any significance test.
The comparison is not “synthetic control always beats difference-in-differences.” Difference-in-differences relies on a parallel-trends counterfactual; synthetic control uses a weighted pre-period match that can accommodate different levels and some pre-trend patterns. Both need a credible no-treatment trajectory and protection from concurrent shocks. If randomized rollout is possible, it may offer a stronger design. Explain why the available assignment process drove your choice.
Ask whether the outcome is measured as a level, rate, or total. Orders can rise because the population grew, while orders per eligible customer stayed flat. If donor markets have different sizes, a raw-level fit may assign weights that mimic scale but not underlying demand dynamics. The outcome transformation and covariates should be chosen for the decision before inspecting the post-launch gap. If the launch itself changes eligibility or logging, even a stable pre-period rate can lose its meaning after the intervention. Explain the numerator and denominator alongside the method.
Build a small donor-weighted counterfactual
Original numerical case: four pre-launch weeks for treated City T have completed orders [91,95,99,103]. Donor City A has [80,84,88,92]; donor City B has [100,104,108,112]. Equal weights, 0.5 on A and 0.5 on B, produce [90,94,98,102]. Each pre-period treated-minus-synthetic gap is one order unit. The pre-period root mean squared prediction error is therefore one.
Now suppose the first three post-launch outcomes are T [112,117,121], A [96,100,104], and B [116,120,124]. The synthetic outcomes are [106,110,114]; the gaps are [6,7,7]. The mean observed gap is 20/3 ≈ 6.67 outcome units per week. The post-period RMSPE is sqrt((36+49+49)/3) ≈ 6.68; divided by pre-RMSPE one, the post/pre ratio is about 6.68.
| Period | Treated T | 0.5 A + 0.5 B | Gap |
|---|---|---|---|
| Pre 1 | 91 | 90 | 1 |
| Pre 2 | 95 | 94 | 1 |
| Pre 3 | 99 | 98 | 1 |
| Pre 4 | 103 | 102 | 1 |
| Post 1 | 112 | 106 | 6 |
| Post 2 | 117 | 110 | 7 |
| Post 3 | 121 | 114 | 7 |
The arithmetic is not the causal conclusion. Four pre-period points are far too short for a serious design, and the example has only two donors. A real analysis would use enough historical variation, pre-specified outcomes and predictors, and a defensible donor pool. It would fit weights on pre-treatment data only, then freeze them before exposing post outcomes. Selecting donors or tuning weights because they make the July effect look larger contaminates the evaluation.
Who belongs in the donor pool?
Each donor must be untreated during the analysis window and reasonably comparable in outcome definition and exposure to shocks. Suppose City C matches T perfectly before July but receives the same national marketing campaign triggered by the launch. Its excellent pre-fit does not make it a valid untreated control; the campaign can move C's post outcome. Conversely, excluding C merely because its post outcome makes the estimated effect smaller is outcome-driven selection. Document eligibility from business facts before inspecting the effect.
Check spillovers. Customers may order from neighboring cities, drivers may be shifted across markets, or an advertising campaign may reach donors. A donor affected indirectly is not a no-treatment counterfactual. Check staggered treatments, policy changes, measurement migrations, and different holiday calendars. If only a few clean donors remain, disclose that the estimate is fragile rather than filling the pool with contaminated markets to reduce variance.
Pre-period outcome fit is necessary but not enough. A model can match four points while failing to match mechanisms: customer mix, capacity, pricing, and local demand trends. In longer panels, hold out a portion of the pre-period: fit on earlier weeks and test prediction in later pre-launch weeks. This checks whether the weights generalize before treatment. If the synthetic line diverges before July, a post-July gap may merely continue an earlier mismatch.
Run leave-one-donor-out sensitivity. In our two-donor toy case, dropping B leaves only A, which sits about 11 units below T pre-launch; dropping A leaves B about nine units above. That exposes dependence on both donors. In a real pool, inspect whether the estimated effect vanishes or reverses when one high-weight donor is removed. A result entirely driven by one city needs a clear case for that city's stability and untreated status.
What do placebo tests tell you?
An in-space placebo pretends each eligible untreated donor was treated at the same date and reruns the full synthetic-control procedure for it, using the remaining eligible donors. Compare the treated gap with the distribution of placebo gaps. The original synthetic-control study uses this logic for inference. It is a falsification exercise: if many untreated cities show gaps as large as T, the observed separation is less distinctive.
Compare like with like. A placebo city with dreadful pre-treatment fit can have a large post gap simply because its synthetic comparison never tracked it. Inspect pre-RMSPE and, where appropriate, compare post/pre-RMSPE ratios or restrict the display to placebos with reasonably comparable pre-fit using a stated rule. Our toy treated ratio 6.68 is not a p-value. With only two donors, a placebo distribution would be nearly useless; the example lacks the pool size for meaningful inference.
An in-time placebo moves the hypothetical intervention date to a pre-launch period and tests whether a similar gap appears when no launch had occurred. It requires sufficient earlier data to fit weights before the placebo date and enough later pre-launch observations to evaluate. With only four pre weeks, our toy case cannot support it. A large fake-date effect suggests instability, prior shocks, or model overfit; a small one is reassuring but not proof against a coincident real-date shock.
There is also a design question about anticipation. If merchants were informed of the delivery change two weeks before July and began staffing differently, a fake intervention in late June might show a “placebo” gap because treatment behavior had already begun. In that case, the date is not truly placebo. Reconstruct the rollout chronology from operational records and define exposure as the earliest plausible behavioral change. Reporting both announcement and launch dates can make the uncertainty visible instead of hiding it inside a single vertical chart line.
Placebo comparisons are conditional on donor eligibility and the design's assumptions. They do not randomize the original assignment retroactively. Do not automatically call the fraction of placebo gaps beyond T a conventional randomized-trial p-value. Explain the finite pool, exchangeability concerns, and effect scale before interpreting it.
What breaks the causal interpretation?
The most damaging failure is an unmeasured event unique to T at the same time: a competitor closes, an app bug is fixed, or a local promotion launches. Synthetic control cannot separate that event from the delivery promise if both begin together. Similarly, if the outcome definition changes only in T, the measured gap can be a logging artifact. Review operational calendars and raw denominators; placebos cannot detect every unique concurrent shock.
Interference is another problem. If the intervention changes donor outcomes, the synthetic series is not untreated. Anticipation matters too: customers or merchants may act before the official launch, making “pre” weeks partially treated. Move the intervention boundary to when behavior plausibly changed, not merely the press-release date. Compare alternate outcomes less likely to respond to the treatment as negative controls, while being honest about their limitations.
A pretty pre-trend plot can be overfit if weights, predictors, and start date were repeatedly tuned after seeing post outcomes. Record the design choices and show robustness across defensible alternatives. If no combination of eligible donors gives adequate pre-fit, say the method is not credible for this case. Choosing another design or limiting claims is a stronger interview response than reporting a precise but unsupported effect.
A concise answer under interview pressure
“I would define the treated unit, intervention date, outcome, and post-period estimand. I would screen donors for treatment, spillovers, and comparable measurement, then fit nonnegative weights using pre-treatment data only. I would report the pre-fit, the frozen-weight post gap, and sensitivity to donor exclusion. In-space and, if history allows, in-time placebos would show whether similarly large gaps arise without treatment. A good fit and unusual placebo gap support a causal story only if concurrent shocks and interference are credible exclusions.”
In the city exercise, compute 0.5 A + 0.5 B and the [6,7,7] gaps quickly, then immediately name what the seven rows cannot establish. Practice the Estimate Causal Impact Using Synthetic Control Methods by changing City C from clean to campaign-exposed and explaining why its pre-fit no longer earns admission to the pool.