Sequential A/B Testing Interview Questions: Peeking, Stopping Rules, and Error Control

Sequential A/B testing interview questions turn on one distinction: a test's error guarantee belongs to its entire decision rule, including when you stop. A p-value below 0.05 on Tuesday does not justify a 5% false-positive claim if Tuesday was selected because it was the first encouraging day.

Your answer should define the hypothesis, monitoring schedule, valid evidence measure, and launch criterion. Then explain what happens when the result is inconclusive. For posterior-based decisions, see the related Bayesian A/B testing guide.

Evidence boundary: Primary research supports the statistical facts below. No verified candidate report establishes this exact exercise or a company's scoring rubric. The numerical paths, monitoring contract, and suggested answers are original practice material. Documentation and research were checked September 21, 2026.

Why does peeking change a fixed-horizon test?

Suppose the null says a checkout change has zero effect on conversion. A valid fixed-sample p-value satisfies P(p ≤ α | null) ≤ α under its assumptions. That statement concerns the prespecified analysis, not the smallest p-value you can find across a growing dataset.

Primary research: Johari, Pekelis, and Walsh develop always-valid inference specifically for decisions made while monitoring experiments. Their construction changes the evidence measure so that a chosen stopping time does not invalidate the intended error guarantee. It does not turn ordinary fixed-horizon p-values into always-valid ones. Always Valid Inference.

The repeated-look event is “reject at least once.” Observations from successive cumulative samples overlap, so the p-values are dependent. Consequently, 1 − 0.95^k is not generally the false-positive probability for k cumulative looks. That expression assumes independent rejection events. Neither “exactly 5%” nor “exactly k times 5%” is a defensible generic answer.

A useful interview explanation is: “Every opportunity to stop on favorable noise changes the selection rule. I need a procedure calibrated for the union of those opportunities.” Looking at a dashboard is not itself the mathematical problem. Using what you see to choose a rejection time, endpoint, segment, or analysis is what changes the inferential procedure.

Operational monitoring can still be necessary. You may stop immediately for a serious outage without claiming that the experiment established a beneficial treatment effect. Keep the safety action and the efficacy claim separate.

Can you protect three planned looks with simple arithmetic?

Original exercise: Plan cumulative analyses at 10,000, 20,000, and 30,000 randomized users. At each fixed checkpoint, assume the selected two-sided test supplies a valid p-value for the same null. Set the overall type I error limit to 0.05.

One conservative construction allocates 0.05 / 3 to each checkpoint. The union bound gives:

P(any false rejection)
≤ P(false rejection at look 1)
  + P(false rejection at look 2)
  + P(false rejection at look 3)
≤ 0.05/3 + 0.05/3 + 0.05/3
= 0.05

Independence is unnecessary for this bound. The test at each checkpoint still needs valid assumptions; Bonferroni cannot repair biased assignment or an incorrect standard error.

Use the following hypothetical path to rehearse the rule. The third row is the continuation you would observe if the experiment reaches its final checkpoint.

Planned look	Ordinary p-value	Decision at threshold 0.05/3
10,000 users	0.042	Continue; no rejection
20,000 users	0.021	Continue; no rejection
30,000 users	0.013	Reject the stated null

An unadjusted stop-at-0.05 policy would stop at the first row. The specified three-look policy reaches the third. Use the exact fraction shown in the figure; rounding the cutoff upward changes the stated bound.

Now change the last p-value to 0.019. None of the three analyses rejects. You cannot quietly restore a 0.05 threshold at the final look: you already reserved error probability for the earlier opportunities. Nor can you add a fourth look without a valid extension rule.

This example is deliberately conservative. It makes the error accounting visible and is easy to defend on a whiteboard. It is not a claim that dividing alpha equally is the most efficient design.

When would you choose group-sequential or always-valid inference?

Start with the team's decision cadence. If a decision is genuinely made once, a planned fixed horizon may be sufficient. If the team needs two interim decisions and one final analysis, a group-sequential design can use boundaries calibrated to the joint behavior of the test statistics.

Official software documentation: rpact illustrates Pocock-style boundaries and the comparatively stricter early boundaries of O'Brien–Fleming designs. Group-sequential boundary examples. State this qualitative difference without inventing critical values. Exact boundaries depend on the design, number and timing of looks, sidedness, and calibration. Information fractions describe statistical information accumulated, which need not equal elapsed calendar fractions.

An alpha-spending approach can accommodate specified flexibility in analysis timing. It still requires the applicable statistical model and spending design. “We have an alpha budget” is incomplete unless you explain how that budget maps to valid rejection boundaries.

Primary research: Howard and coauthors develop confidence sequences with simultaneous coverage over time under stated assumptions. An ordinary confidence interval protects coverage at its chosen analysis time; a confidence sequence protects the full sequence. This distinction allows uncertainty to be reported at a data-dependent stopping time. Time-uniform confidence sequences.

Always-valid methods are useful when continuous monitoring is part of the actual workflow. Their assumptions may involve bounded outcomes, distributional conditions, predictable choices, or a particular construction. Do not present the label as a universal setting that works unchanged for conversion, heavy-tailed revenue, and clustered marketplace outcomes.

The decision is practical as well as statistical. Ask whether the team can implement and explain the method, whether the expected experiment volume justifies added complexity, and whether outcome delay prevents useful early decisions. Continuous calculations do not make a seven-day retention outcome mature sooner.

What must the stopping contract specify?

Original monitoring contract: Imagine a checkout experiment with users assigned once and conversion measured within 48 hours. Before launch, write these decisions in plain language:

• The estimand is the difference in 48-hour conversion probability for eligible randomized users.
• The primary hypothesis, sidedness, and overall error target are fixed before outcome review.
• Each look uses users whose complete 48-hour window is available, with the same inclusion rule across arms.
• The chosen monitoring method governs all efficacy decisions; its implementation and assumptions are recorded.
• A maximum enrollment or calendar limit ends the experiment even if evidence remains inconclusive.
• Outage and harm triggers can halt exposure, while the statistical report states what those stops do and do not establish.

Specify how late events are handled. If control users have mature windows while newly added treatment users have incomplete outcomes, the apparent effect may be a data-timing artifact. Sequential inference protects monitoring under its model; it does not validate inconsistent measurements.

Also define what happens after crossing a boundary. A team may stop new assignment, let already assigned users finish their windows, and publish the analysis required by its design. It should not mix a favorable interim estimate with an unrelated later confidence interval and call the combination prespecified.

Effect estimation needs care because early stopping selects unusual paths. A valid rejection does not automatically make the stopped point estimate unbiased or an ordinary interval correctly covered. Report uncertainty from the chosen sequential method and explain any additional estimation procedure.

Does stopping for futility prove there is no effect?

No. A futility rule can say the experiment is unlikely to achieve its planned objective with the remaining resources. That is different from proving the effect equals zero or ruling out every useful benefit.

Editorial recommendation: In your answer, separate three outcomes: evidence of a positive effect, evidence that any benefit is smaller than a stated practical threshold, and insufficient information. Equivalence or noninferiority questions require their own margins and procedures. A nonsignificant result alone does not establish either.

Official software documentation: rpact distinguishes binding futility, which affects critical-value calibration, from nonbinding futility. In a nonbinding design, type I error calibration does not require the team to obey that boundary. Design reference. A binding boundary is part of the calibration. Treating these as interchangeable can invalidate the claimed design. Ask which convention the experiment uses before recommending continuation after a futility signal.

Business stopping is also legitimate: a competing launch may remove the value of further learning. Record that reason. Do not retroactively describe a budget-driven stop as statistical proof that the feature failed.

The same distinction applies to launch decisions. A statistically detectable improvement can be too small to cover implementation cost, or it can accompany unacceptable latency. A boundary crossing is evidence about a hypothesis; the shipping decision also needs magnitude, uncertainty, guardrails, and business value.

What if the team already peeked or changed metrics?

First reconstruct the actual decision process. How many analyses occurred? Were their times fixed? Did the team change the metric, filter users, select a segment, or stop enrollment based on results? Preserve the complete record, including unfavorable analyses.

If exactly three analyses were prespecified and each p-value was valid, the conservative ledger above offers a transparent evaluation. If the number or timing of ordinary tests was outcome-dependent, merely dividing 0.05 by the number remembered afterward is not a general repair.

A valid procedure might be reconstructed from complete data under additional assumptions, but that requires methodological review. A clean confirmation experiment with a specified rule is often easier to explain. Label the exploratory result as exploratory while deciding whether confirmation is worthwhile.

Likewise, correction over time does not automatically correct over hypotheses. Testing five metrics, many countries, and several variants creates another family of opportunities. Define the family and its error objective. Family-wise error control limits the probability of any false rejection in that family; false discovery rate control targets a different quantity. Choose a method that supports both the multiplicity and monitoring structure.

Bayesian updating addresses a different inferential question. A posterior probability is not a frequentist false-positive guarantee. If a team wants operating characteristics for a Bayesian stopping policy, evaluate that complete policy under explicit data-generating scenarios.

Practice defending the decision aloud

These related questions are practice prompts, not verified transcripts of this exact case. For each answer, name the error event before naming a method.

Practice question	What to rehearse
Apply sequential testing without p-hacking	State the error event across all analysis times.
Compute cluster-aware significance and sequential corrections	Separate correlation within assignments from correction across looks.
Plan and analyze an A/B test	Write the complete monitoring and outcome-maturity contract.
Design and assess a video-pin increase experiment	Distinguish a launch threshold from an encouraging interim result.
Design and power an A/B test	Explain how stopping flexibility changes sample-size planning.

A concise response to “Can we stop at p = 0.04?” is: “I need the monitoring rule. Under a single planned valid test, that may reject at 5%. After repeated unadjusted looks, it does not support the same claim. I would apply the specified sequential boundary, check measurement and guardrails, and separate statistical evidence from whether the improvement is worth launching.”

Rehearse the three-look ledger without notes, then change one assumption: a fourth look, a delayed outcome, or a newly selected segment. Explain exactly which guarantee needs to be reconsidered.

Sources and Further Reading

• Johari, Pekelis, and Walsh: Always Valid Inference — primary research on sequential inference for A/B testing.

• Howard and coauthors: Time-uniform, nonparametric, nonasymptotic confidence sequences — primary research on simultaneous coverage over time.

• Howard and coauthors: Time-uniform Chernoff bounds via nonnegative supermartingales — primary mathematical background for time-uniform concentration.

• rpact: Group-sequential boundary examples — official software examples of calibrated monitoring.

• rpact: Group-sequential design reference — official design arguments, including binding futility.