A/A Testing Interview Questions: False Positives, Calibration, and Pipeline Failures

An A/A test sends users into two groups that receive the same experience. It sounds as though the result should always be zero. It should not. Random groups vary, and a valid test at a prespecified 5% significance level can flag a difference under a true null some of the time. An interview answer must separate that ordinary false-positive behavior from a platform that assigns users unevenly, loses events, joins duplicates, or calculates the metric at the wrong grain.

This guide gives you a reproducible calibration experiment and three diagnostic failures to explain aloud. Evidence boundary: Microsoft Research describes trustworthy experimentation checks; the simulation and broken pipelines below are original editorial exercises. No candidate report is used as a process fact.

What should an A/A test establish?

First define the randomization unit, assignment ratio, eligibility rule, exposure condition, metric, analysis unit, and decision threshold. “The UI is the same” is insufficient if assignment, logging, or analysis differs between arms. For example, a treatment flag could alter an event schema even when the rendered page looks identical. A healthy A/A provides evidence that the experiment machinery and statistical procedure behave as expected for the checks you ran. It does not prove that every future metric or product experiment is unbiased.

Microsoft Research's pre-experiment discussion treats A/A-style and retrospective checks as part of platform trustworthiness, while warning that a retrospective check can be a lagging indicator of bias. Its during-experiment guidance emphasizes ongoing data-quality and analysis checks. These are official Microsoft engineering discussions, not a universal rule that every company runs the same A/A gate.

In an interview, say which question each check answers. Assignment counts test whether the configured split appears in the randomization service. Exposure counts show who reached the experiment. Metric counts and values test logging and aggregation. Repeated A/A results test the combined statistical pipeline's calibration. A single apparently insignificant A/A result cannot establish calibration: under a broken pipeline, one run can still return a large p-value by chance.

Question 1: why can a correct A/A be significant?

Under a correctly specified null and continuous test statistic, p-values are approximately uniform from zero to one. At an alpha of 0.05, about 5% of independent, prespecified tests should fall below 0.05 in the long run. That does not mean exactly one in every twenty tests, and it does not make a significant A/A automatically harmless. One test gives weak evidence about calibration; repeated tests plus targeted data checks give a stronger diagnosis.

If you look at 20 metrics and call the experiment broken whenever any metric has p < .05, your chance of at least one chance finding can be much larger than 5%. The exact calculation depends on dependence among metrics. Likewise, rerunning an A/A until a favorable result appears is not a calibration procedure. Define the metric family, threshold, analysis window, and alert rule before looking. Track both a per-test false-positive rate and how often the scorecard-level rule alarms under the null.

A useful mental model is a smoke detector. It can alarm when there is no fire, but a detector that alarms on every other cooking session is poorly calibrated. Conversely, silence once does not prove its battery works. The interviewer wants the sampling logic and the engineering checks, not the phrase “A/A means no difference.”

Repeated A/A simulation: a reproducible calibration check

This original exercise simulates 2,000 independent experiments, each with 500 users per arm. Every user in both arms has the same 20% chance of a binary outcome. For each replicate, we compute a pooled two-proportion z statistic and a two-sided p-value. The fixed seed makes the demonstration reproducible, not more statistically valid.

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(2026)
replicates, users_per_arm, baseline_rate = 2000, 500, 0.20
successes = rng.binomial(users_per_arm, baseline_rate,
                        size=(replicates, 2))
pooled = successes.sum(axis=1) / (2 * users_per_arm)
se = np.sqrt(pooled * (1 - pooled) * 2 / users_per_arm)
z = (successes[:, 1] - successes[:, 0]) / users_per_arm / se
p_values = 2 * norm.sf(np.abs(z))

print((p_values < 0.05).sum())
print(np.histogram(p_values, bins=np.linspace(0, 1, 11))[0])

With this seed, 99 of 2,000 tests fall below .05, or 4.95%. The ten p-value bins contain [207, 211, 204, 220, 189, 222, 208, 115, 255, 169]. Do not demand exactly 100 flags or perfectly equal bars. Binary outcomes produce discrete counts and a z approximation, so the histogram has visible irregularity. If you rerun with another seed, the count moves. The practical question is whether the observed deviation is consistent with sampling variability and whether design and data checks pass.

This simulation is deliberately simple. A production metric may be continuous, heavy-tailed, ratio-valued, clustered by user, or delayed. If you randomize users but compute per-event independent standard errors, false positives can inflate even when treatment truly changes nothing. To validate the actual platform, replay its assignment, metric query, variance estimator, missingness rules, and monitoring policy—not a toy z test that bypasses them. A/A calibration is about the full path that an A/B decision would use.

Question 2: distinguish three pipeline failures

Suppose a 50/50 A/A has balanced raw assignments but a significant conversion gap. The fastest useful response is not “increase sample size.” Trace distinct randomized IDs across four stages: assignment, exposure, eligible analysis population, and metric outcome. Compare counts and missingness by arm at each transition. Preserve event timestamps and join keys so the discrepancy can be located rather than merely observed.

Deliberately broken case	What remains healthy	Expected diagnostic	Correct next move
Exposure logger drops some B events after assignment	Assignment IDs are 50/50	B loses distinct IDs between assignment and exposure; missingness differs by arm	Fix the logging path and rerun, rather than interpret lift
Metric join multiplies B event rows	Assignment and exposure counts may match	Distinct users stay balanced but event rows per user and metric totals inflate in B	Enforce one row per analysis unit or deduplicate with documented rules
“Active users” filter uses a post-assignment event	Raw allocation is balanced	Eligible counts diverge after a condition affected by the pipeline or treatment	Analyze the intended population and treat the filter as a diagnostic

These are mechanisms, not claims that every unequal count is a bug. For example, a planned ramp or stratification can make a temporary overall ratio look odd unless expected allocation is calculated by time and stratum. A genuinely treatment-dependent exposure can also change downstream counts in an A/B, which is why the assignment population and estimand matter. In an A/A, where variants should be identical, a persistent arm-specific loss deserves investigation even if the aggregate outcome p-value is unremarkable.

A sample ratio mismatch check targets counts against a known allocation, whereas a metric test targets outcome differences. Both can pass while another part is broken: an equal number of duplicated rows in each arm could bias the metric's variance without causing SRM; a logging loss could create SRM yet, by chance, no significant outcome gap. Do not use one test as a substitute for a pipeline trace. Record the first stage where counts, schemas, or values diverge.

Question 3: what if the p-value histogram is suspicious?

An excess of very small p-values across many A/A splits can indicate underestimated standard errors, leakage, unbalanced covariates from flawed assignment, or analysis after selectively choosing metrics. A pile near one can arise from conservative tests, rounding, discreteness, or duplicated comparisons. Diagnose the expected null distribution for the actual test before labelling any visual departure a failure. Ten-bin plots are a screen, not a formal proof.

Start with the unit. If users are randomized but multiple sessions per user are treated as independent, the denominator of the standard error can be too optimistic. Then inspect assignment stickiness: did a user move between arms across devices or sessions? Examine late or missing events, attribution windows, bot filters, and joins. Rerun a minimal metric derived directly from assignment records. If that is calibrated but the production metric is not, the error lies downstream. If even assignment counts or pre-treatment covariates are distorted, look upstream.

Check the analysis plan too. Repeated interim peeks, filtering out inconvenient cohorts, or choosing the metric with the smallest p-value can inflate apparent discoveries without a software bug. Predefine a fixed horizon or a valid sequential procedure, and state how multiplicity is handled. A platform can be mechanically correct while an analyst workflow is uncalibrated. Conversely, a perfect statistical plan cannot repair missing telemetry.

Question 4: how would you turn this into a release gate?

Make the gate concrete. Before launch, verify stable assignment by unit and allocation schedule, event schema parity, expected exposure behavior, a documented metric definition, and no unexpected arm-specific missingness. Run repeated null splits through the same query and estimator used for A/B reports. Compare the empirical false-positive fraction with a tolerance justified by the number of replicates, and investigate histogram shape rather than hard-coding “exactly 5%.” Keep a small set of sentinel metrics that are not supposed to move, while recognizing that an A/A cannot expose every future treatment-specific failure.

If a gate fails, preserve raw assignment and event samples, record the first divergent stage, and name an owner for the fix. After repair, rerun the same diagnostic and a clean A/A. Do not “correct” a bad A/A by manually moving users between groups after assignment or by quietly dropping a metric. A credible postmortem says what part of the pipeline was wrong, which decisions may be affected, and what evidence now supports reuse.

An interview-ready response might be: “At alpha .05, a single significant A/A can happen by chance. I would predefine the metric and horizon, then run many independent null splits to inspect calibration. For a suspicious result, I would trace distinct user IDs from assignment through exposure, eligibility, joins, and outcomes, checking the first arm-specific divergence. I would separate SRM, telemetry defects, variance-estimator mistakes, and analysis peeking before declaring the platform healthy or broken.”

Sources and Further Reading

• Microsoft Research, Patterns of Trustworthy Experimentation: Pre-Experiment Stage — platform preparation and retrospective A/A limitations.
• Microsoft Research, Patterns of Trustworthy Experimentation: During-Experiment Stage — monitoring experiment data quality and analysis.
• Microsoft Research, Treatment Effect Assessment at Scale — recent discussion of A/A false positives across correlated metrics.
• SciPy, Normal Distribution — norm.sf used in the reproducible simulation.