A/B Test Sample Size Interview Questions: MDE, Traffic Limits, and Feasibility

“How many users do we need?” is usually the second question in an A/B test interview. The first is what change is worth detecting, on which metric, among which eligible users, by what deadline. A calculator can return an impressive number even when the team has confused a 10% relative lift with a 10-percentage-point lift, counted page views instead of randomized users, or assumed every visitor is eligible. This original exercise turns sample size into a feasibility decision for a product team with a two-week launch window.

The linked questions are practice prompts, not evidence that a particular employer uses this scenario. Evidence boundary: statistical software and experimentation documentation support the power and minimum-detectable-effect concepts. The 8% baseline, traffic, deadline, calculations, and interview rubric below are an original teaching case, not observed experiment results or a promise that a future test will achieve exactly 80% power.

Start by making the planning contract explicit

Assume a binary purchase-conversion metric measured once per randomized eligible user. Historical control conversion is 8%. The product team would act on a 10% relative lift, from 8.0% to 8.8%. It asks for a two-sided 5% significance level, 80% power, and equal 50/50 allocation. Traffic is 12,000 visitors per day, but only half pass the experiment's eligibility filters. So the planning pool is 6,000 randomized users per day, or about 3,000 per arm. The deadline is 14 calendar days. Those assumptions are part of the answer; hiding them behind one sample-size number is not.

The metric's analysis unit must match the randomization unit. If assignment is by user, counting every page view as a fresh independent observation inflates apparent sample size. If the same person can appear under multiple anonymous IDs, eligibility and identity resolution become relevant. If the business wants a user-level revenue metric rather than binary conversion, this particular two-proportion calculation is the wrong model. Ask these questions before choosing a formula.

Planning input	Value in this exercise	Question to ask
Baseline conversion	8.0%	Is the estimate from a comparable eligible cohort?
Minimum detectable effect	10% relative	Is 0.8 percentage point worth acting on?
Target treatment rate	8.8%	Was relative lift converted correctly?
Type I error and power	Two-sided 0.05, 0.80	Is this the planned decision rule?
Allocation	50/50	Are there operational reasons to deviate?
Eligible traffic	6,000 users/day	Is this unique randomized users, not sessions?
Deadline	14 days	Does the required sample fit full usage cycles?

Optimizely's MDE guidance frames MDE as a planning choice. It is not the smallest effect the experiment could ever observe, and it is not a promise that any significant result must be at least that large. The team should choose an effect tied to a decision—incremental value, implementation cost, and downside risk—not whichever MDE makes the sample size fit a preferred launch date.

Convert relative lift into the quantity the test detects

With baseline probability p0 = 0.08, a 10% relative increase means p1 = 0.08 × 1.10 = 0.088. The absolute difference is Δ = 0.008, or 0.8 percentage point. A 10-percentage-point lift would mean 18% conversion, more than double the baseline; that is a wholly different business and statistical claim. Say both scales in your interview answer. It is easy for a stakeholder to hear “ten percent” and imagine a much larger effect than the experiment is powered to detect.

For two independent proportions with equal allocation, a standard normal-approximation planning formula uses the baseline and target proportions, a two-sided critical value, and the target power. One common expression is:

p_bar = (p0 + p1) / 2
n_per_arm ≈ [ z_(1-alpha/2) * sqrt(2*p_bar*(1-p_bar))
              + z_(power) * sqrt(p0*(1-p0) + p1*(1-p1)) ]^2
            / (p1-p0)^2

Using alpha = 0.05, power 0.80, p0 = 0.08, and p1 = 0.088 gives about 18,872 eligible users per arm, or 37,744 total after rounding each arm up. That is a planning approximation under its model, not a substitute for a simulator when users are clustered, outcomes are unusual, or the decision rule differs. statsmodels' NormalIndPower.solve_power provides a software route for two-sample normal power calculations; always check the effect-size convention and parameters of the function you actually call.

At 6,000 eligible users per day, the arithmetic duration is 37,744 / 6,000 ≈ 6.29 days, so the earliest full-day completion is about 7 days if traffic is stable and enrollment begins immediately. This is a traffic floor, not an automatic stopping rule. A product with weekday/weekend behavior may intentionally run at least a full weekly cycle; ramp-up, exclusions, bot filtering, and missing outcomes can extend it. Fixed-horizon inference also requires a planned stopping rule. Repeatedly peeking and stopping whenever p < 0.05 changes error behavior unless the analysis method accounts for sequential monitoring.

Now shrink the MDE: the deadline stops working

The interviewer changes one requirement: “We want to detect a 5% relative lift instead.” The target becomes 0.08 × 1.05 = 0.084, an absolute difference of 0.004, or 0.4 percentage point. Holding the other assumptions fixed, the same approximation needs about 73,853 per arm and 147,706 total. At 6,000 eligible users per day, that is 24.62 days, rounded up to about 25 days. The two-week deadline no longer supports that sensitivity under the stated traffic and model.

Relative MDE	Target rate	Absolute difference	Approximate users per arm	Total	Traffic-floor duration
10%	8.8%	0.8 percentage point	18,872	37,744	7 days
5%	8.4%	0.4 percentage point	73,853	147,706	25 days

The sample requirement grows by roughly four times when the effect difference is halved because the denominator of the approximation contains Δ². It is not exactly four here because the variance terms also shift slightly with p1. More importantly, the second scenario is not a small scheduling inconvenience; it is a different experiment design choice. Under a 14-day window and 6,000 eligible users daily, expected total enrollment is about 84,000, far below the 147,706 requirement. Do not pretend that one week of extra optimism or a last-minute p-value check will replace the missing information.

The first case is arithmetically feasible within two weeks, but do not claim guaranteed success. If conversion drops, eligible traffic falls, allocation becomes uneven, or outcome latency extends beyond the last enrollment day, the final analyzable sample can miss the target. The planner should distinguish enrolled, exposed, and measured users. A person may be assigned but never see the feature, or may purchase days later. Define the analysis population and outcome window before drawing the calendar. A simple total / daily traffic calculation assumes these losses are negligible or already reflected in the eligible-traffic estimate.

What should the team do when traffic and sensitivity conflict?

Do not silently weaken the power requirement or change the MDE after seeing results. Offer an explicit menu. Extend the test to about 25 days plus any ramp or outcome-lag buffer. Expand eligibility only if the additional users belong to the target decision population; adding irrelevant traffic can dilute the effect. Increase traffic allocation if there is unused eligible traffic and risk permits. Consider a more sensitive pre-specified analysis, such as covariate adjustment, if it is valid for the design and estimated variance reduction is defensible. Or accept that a 5% relative effect cannot be resolved by this test on the deadline and make a different product decision.

Proposed response	When it helps	Risk to name aloud
Extend duration	The decision can wait and traffic is stable	Seasonal changes, concurrent launches, delayed outcomes
Increase eligible traffic	Additional users represent the same target population	Different user mix or interference
Adjust analysis method	Pre-specified covariates genuinely reduce variance	Post-hoc method shopping or model mistakes
Accept larger MDE	Only a larger effect is economically worth acting on	Smaller real gains may remain unresolved
Stop the experiment idea	Deadline is fixed and uncertainty is unacceptable	Requires another decision process, not a false claim of proof

Optimizely's prioritization guidance similarly treats MDE and available traffic as planning inputs. A credible interview answer chooses one option based on business value and constraints. “Just run it for two weeks and see” is not a power plan. If the result is inconclusive, the honest statement is that the test did not sufficiently distinguish effects of the desired size under its design—not that treatment and control are “the same.”

Follow-ups: allocation, multiple arms, and guardrails

Equal allocation is efficient for comparing two proportions when per-user cost and variance are similar. A 90/10 rollout may reduce exposure to a risky treatment, but it generally needs more total traffic for the same comparison power. If there are two treatments against one control, the calculation and multiplicity plan change; do not reuse a two-arm total by dividing it three ways. If the primary metric is rare or heavily skewed, estimate variance from comparable historical data and consider a simulation of the actual analysis. For a clustered randomization, account for within-cluster correlation instead of counting all users as independent.

Sample-size planning also needs a trust checklist. Check that assignment is stable, no one appears in both arms, sample-ratio mismatch is monitored, exposure is logged, and the outcome window is complete. Set a primary metric and guardrails before launch. If the team changes its metric, filtering rule, or stop date after seeing treatment effects, the nominal power and false-positive interpretation no longer correspond to the original plan. Optimizely's fixed-horizon discussion underscores that analysis timing is part of the method, not administrative trivia.

The interviewer's last trap may be “If we collect 18,872 per arm, will we definitely detect a 10% relative lift?” No. At the modeled 10% lift, 80% power means a long-run probability of rejecting the null under the specified assumptions and decision rule, not certainty in this one experiment. The observed effect can be smaller or larger; confidence intervals convey uncertainty after the test. If the test is not significant, inspect the interval and data quality before claiming the feature has no effect. Explain power as a property of the planned procedure, not a score awarded to an individual result.

Sources and Further Reading

• statsmodels: NormalIndPower.solve_power — two-sample normal power interface.
• Optimizely: Use MDE when designing an experiment — MDE as a design input.
• Optimizely: Use MDE to prioritize experiments — feasibility and prioritization.
• Optimizely: Frequentist fixed-horizon statistics — planned analysis timing.