Switchback Experiment Interview Questions: Randomization Units, Carryover, and Analysis

A switchback experiment changes treatment over time within a market or other shared system. The central interview challenge is deciding what was randomized, what effect you want to estimate, and which observations share information. A million order rows cannot rescue an analysis that treats one market-wide assignment as a million independent assignments.

This guide develops those decisions through an original, reproducible example. Evidence boundary: DoorDash's published engineering articles document historical experimentation methods; they do not establish a current interview process. The research paper cited below supplies methodological context.

When Does a Switchback Make Sense?

Start with the mechanism that connects people. Suppose a dispatch policy changes which courier receives a job. Giving one order the new policy can change the couriers available to another order. The control experience may therefore depend on treatment assignments elsewhere in the marketplace.

Official engineering example: DoorDash's 2019 analysis describes this dependency and randomization across geographic regions and time windows. It also discusses nested observations and the limited number of more independent units available for analysis. That is a useful reason to consider switchbacks, rather than assuming every marketplace experiment needs them. DoorDash's switchback analysis.

A good opening answer identifies the decision: “We want to estimate the short-term effect of enabling the dispatch policy throughout an eligible market, while reducing competition between simultaneous treatment and control orders.” Then ask whether couriers cross market boundaries, whether changes persist after switching off, and whether the outcome requires prolonged exposure.

For a feature that teaches couriers a new behavior over several weeks, rapid switching may answer the wrong question. Official engineering guidance: DoorDash's 2022 discussion distinguishes network effects, learning effects, and power; its examples explain why different interventions require different designs. Treat this as a trade-off framework, not a promise that any design removes all interference. DoorDash's experimentation trade-offs.

Define the Assignment Unit Before the Metric

Keep four objects separate:

Object	Original example	Why it matters
Assignment unit	One market during one time block	All eligible activity receives the assigned policy
Recorded observation	One job's completion time	Many rows can share an assignment and common shocks
Randomization decision	A coin selecting AB or BA for a matched pair of blocks	The design constrains which schedules are possible
Estimand	Average treatment effect across eligible blocks	Equal block weighting differs from volume weighting

Here, A means control and B means treatment. If blocks contain different numbers of jobs, averaging block means equally answers a different question from pooling every job. Neither target is automatically correct. A team interested in a typical operating period may choose the former; a team interested in the experience of a typical job may require the latter, with an appropriate estimator and uncertainty calculation.

Write the numerator, denominator, eligibility rule, and attribution timestamp before analysis. For example, assign jobs using creation time and follow them to completion, including jobs that finish after the boundary. That measures outcomes for a creation cohort; it does not guarantee those jobs avoided exposure to the next policy. The exposure mechanism still needs examination.

Avoid filtering to jobs that successfully used the new feature after assignment without explaining the resulting selection. A policy can change which jobs complete or receive the feature. Your primary assignment-based analysis and any exposure-based secondary analysis should have distinct interpretations.

Is ABAB a Randomized Schedule?

A fixed alternating sequence is a schedule, but it is not randomized simply because it switches. If treatment always occupies the first period of a recurring demand cycle, the comparison can inherit that pattern.

For the teaching example below, form twelve pairs of comparable blocks in advance. Within each pair, flip a fair coin to choose AB or BA. Each pair contains one treatment block and one control block, and there are 4,096 possible schedules. Pairing and independent coins are explicit features of this exercise, not a universal switchback prescription.

In an interview, explain how you formed pairs using information available before treatment. Pairing neighboring blocks may improve comparability, but it can also put them close enough for carryover. Pairing comparable hours on different days may reduce that problem while introducing other differences. You cannot choose the pairing afterward because one grouping produces a stronger result.

Also preserve the assignment log, schedule version, timezone, deployment status, and any overrides. A beautifully randomized plan is insufficient if the service ran a different policy. An operational override should be visible in diagnostics and handled under a prespecified analysis rule.

Choosing Block Length: Carryover Is Not Autocorrelation

Carryover means a previous treatment can affect a later outcome. Couriers repositioned by one policy may remain in different locations after the policy is disabled. Autocorrelation means outcomes at nearby times are related; recurring demand or weather can create it even if treatment has no lasting effect.

A demand autocorrelation plot therefore cannot, by itself, establish how long a treatment persists. Use a plausible causal mechanism, relevant operational timing, and evidence around randomized transitions to assess carryover. Historical demand data can help characterize ordinary time variation, but it cannot reveal the effect of an intervention that never occurred.

Research context: Bojinov, Simchi-Levi, and Zhao explicitly connect switchback design and inference to assumptions about carryover order. Their paper develops methods under stated conditions; it does not make arbitrary label shuffling valid for every schedule. Design and Analysis of Switchback Experiments.

A defensible preparation answer compares candidate block lengths rather than naming a universal number. Shorter blocks create more assignment opportunities but can devote more of the observation window to transitions. Longer blocks leave fewer opportunities within the same runtime and can still be too short for a lasting behavioral response.

If you exclude a boundary interval, specify its duration and attribution rules before looking at the result. Explain the target: an effect during retained steady-state periods differs from an effect across all operating time. Show sensitivity to reasonable alternative exclusions without selecting the one that delivers significance.

A Reproducible Example: More Rows, Same Randomization

This deliberately artificial exercise isolates one mistake. There are twelve pairs, twenty-four block outcomes, no treatment effect, and no carryover. Outcomes are generated before each pair's fair assignment coin. Every event within a block copies its block outcome, creating perfect within-block dependence.

The second dataset repeats those same values more times. These are duplicated observations, not newly collected jobs. A correct interpretation must not gain independent assignment information from duplication.

Run this standard-library Python example:

import itertools
import random
import statistics as s
from math import sqrt

rng = random.Random(19)
pairs = []
for _ in range(12):
    levels = [20 + rng.gauss(0, 4) for _ in range(2)]
    treated = rng.randrange(2)
    pairs.append((levels[treated], levels[1 - treated]))

d = [t - c for t, c in pairs]
effect = s.mean(d)
possible = itertools.product((-1, 1), repeat=len(d))
p = sum(abs(s.mean(a*b for a, b in zip(z, d)))
        >= abs(effect) - 1e-12 for z in possible) / 2**len(d)

for copies in (10, 1000):
    t = [a for a, b in pairs for _ in range(copies)]
    c = [b for a, b in pairs for _ in range(copies)]
    naive_se = sqrt(s.variance(t)/len(t) + s.variance(c)/len(c))
    pair_se = s.stdev(d) / sqrt(len(d))
    print(len(t) + len(c), round(effect, 4),
          round(naive_se, 4), round(pair_se, 4), round(p, 4))

Executed local results:

Recorded rows	Estimated effect	Naive event SE	Paired-difference SE	Exact randomization p-value
240	0.0405	0.4805	1.7850	0.9829
24,000	0.0405	0.0479	1.7850	0.9829

The naive standard error shrinks roughly tenfold when the same rows are repeated one hundred times. The point estimate, pair-based standard error, and randomization p-value stay unchanged. This run illustrates manufactured precision; it does not produce a false positive, and a single simulation does not estimate a method's false-positive rate.

What Makes This Analysis Valid Here?

The exact p-value tests the sharp null of no treatment effect under the exercise's assignment mechanism. It enumerates every within-pair treatment/control swap and compares absolute mean differences. All 4,096 schedules are equally likely under the twelve independent fair coins. We do not shuffle individual event labels or pretend an impossible schedule could have occurred.

The paired standard error is the usual sample standard deviation of pair differences divided by the square root of twelve. Its conventional sampling interpretation needs suitable independent-pair assumptions; serially dependent real blocks do not automatically satisfy them. The exact randomization calculation here instead uses the known assignment distribution under the sharp null.

Once treatment history affects outcomes, the simple swap calculation above cannot be transferred unchanged. Likewise, unequal assignment probabilities, restricted schedules, or missing outcomes require additional work. The code is an interview teaching example, not a production experimentation library.

What Would You Change for a Real Experiment?

First, reconstruct what actually ran. Compare scheduled assignment with deployment telemetry; distinguish missing logging from noncompliance. Then inspect outcome completeness, eligible population counts, market coverage, and time balance without rewriting eligibility to favor a result.

Next, match uncertainty estimation to the dependence and assignment structure. “Cluster by market-time” is not a complete answer if adjacent blocks remain dependent. Conversely, clustering by market with only a few markets raises its own reliability concerns. Regression adjustment, hierarchical models, time-series methods, and randomization inference each require a defensible specification. Naming one does not establish its assumptions.

For power planning, simulate the proposed schedule against representative historical variation, inject several plausible effects, and evaluate the planned analysis. Include expected carryover, missingness, and volume imbalance where credible. Clearly distinguish a planning assumption from something the data have established. More recorded events can improve measurement within a block, but do not create new assignment decisions.

Finally, separate statistical evidence from launch judgment. A precise improvement in completion time may still be unacceptable if cancellations increase. Specify the primary outcome, guardrails, minimum practically useful improvement, and stopping policy before the experiment. Report uncertainty and unresolved contamination alongside the recommendation.

Sources and Further Reading

• DoorDash: Experiment Rigor for Switchback Experiment Analysis — official engineering article, February 2019.
• DoorDash: Balancing Network Effects, Learning Effects, and Power in Experiments — official engineering article, February 2022.
• Bojinov, Simchi-Levi, and Zhao: Design and Analysis of Switchback Experiments — primary methods paper; arXiv revision September 2025.