Articles
Multiple Testing Interview Questions: FWER, FDR, and Metric Families
Compare FWER and FDR, define metric families, and work through Bonferroni, Holm, and BH with a checked Python example.
A dashboard shows twenty metrics, and one has p = 0.04. Is that a discovery, a launch reason, or an expected result of looking in many places? The answer depends on the claims you planned to make, the tests you ran, and the kind of error you want to control.
Multiple testing interview questions are rarely solved by saying “use Bonferroni.” A stronger answer defines the family of hypotheses, chooses between family-wise error rate and false discovery rate, and explains what the adjusted result permits you to claim.
This article separates official statistical software guidance from our original numerical examples and decision framework. We do not use candidate reports to claim how frequently this topic appears in a particular interview.
Why twenty ordinary tests change the interpretation
Suppose all twenty null hypotheses are true, each test has an exact 5% rejection probability, and their rejection events are independent. The probability of no false rejection is 0.95**20. Therefore, the probability of at least one false rejection is:
1 - 0.95**20 = 0.6415, approximately 64.2%
This original calculation illustrates the problem; it is not a universal dashboard false-positive rate. Correlated tests change that probability. Conservative individual tests, discrete p-values, and false null hypotheses also change the setup.
The expected number of false rejections in this all-null example is one. An expectation of one does not mean every dashboard contains exactly one false positive. Some contain none, and others contain several.
Before calculating anything, ask what the analyst was allowed to select. Could any metric justify the same launch? Were multiple variants, time windows, or audience slices searched? Did a nominally primary outcome become primary only after results were visible? Those choices determine the selection problem more directly than the number of charts in a presentation.
FWER and FDR protect different claims
Let V be the number of rejected hypotheses whose nulls are actually true, and R the total number of rejected hypotheses. Family-wise error rate, or FWER, is P(V >= 1): the probability of at least one false rejection in the defined family.
False discovery rate, or FDR, is E[V / max(R, 1)]. It averages the false-discovery proportion across repetitions, taking that proportion as zero when nothing is rejected. It is not the ratio E[V]/E[R] in general.
Official methodological context: the R statistical manual distinguishes methods designed for FWER control from the BH and BY procedures for FDR control. It also documents their dependence conditions. See R's p.adjust reference.
Consider an original screening program that nominates many possible follow-up experiments. Accepting a controlled expected fraction of false leads may fit that discovery objective. A small family of consequential confirmatory claims may instead require control of the chance of making any false claim. The error criterion follows the decision; “FDR is more powerful” is not enough to choose it.
FDR at 5% does not mean each selected hypothesis has a 5% probability of being false. Nor does it guarantee that exactly 5% of this particular result set is false. Those are different statements from the repeated-procedure expectation.
If every null is true, every rejection is false. In that special case the false-discovery proportion is one whenever there is a rejection, so FDR equals FWER. That identity is a useful check on whether you understand the definitions.
Define the metric family before seeing the winners
Here is an original planning example for a checkout experiment:
| Planned use | Candidate claims | Family decision to record |
|---|---|---|
| Confirm whether either variant improves conversion | A versus control and B versus control on the primary conversion metric | Treat both opportunities for the improvement claim together |
| Explore which of twenty behaviors might explain a change | Twenty prespecified diagnostic outcomes | Use a declared exploratory family and distinguish follow-up leads from confirmation |
| Investigate a surprising mobile result after readout | Many device, geography, and customer-tenure slices inspected | Disclose the search; avoid presenting the chosen slice as prespecified confirmation |
| Check acceptable harm on a guardrail | A predefined non-inferiority margin and population | Design the guardrail test and decision logic explicitly |
Separate labels do not automatically justify separate error budgets. If a team can choose any of three “primary families” to announce the same success, the broader selection remains relevant. Conversely, every unrelated test performed anywhere in a company need not be thrown into one indiscriminate correction. State the scope of the claims the guarantee covers.
A metric family includes hypotheses, not just metric names. Three variants compared with one control produce three comparisons for a single metric. Five selected segments can multiply those opportunities further. Repeated looks over time introduce another issue: the validity of the underlying p-values under the stopping rule.
An interview-ready family declaration might be: “Our confirmatory improvement claim can come from either of two variants, evaluated on one prespecified metric at the planned readout. I will control FWER across those two comparisons. The diagnostic panel is exploratory and cannot independently trigger that claim.”
Calculate three procedures on the same five p-values
Use this original, already sorted vector:
0.004, 0.012, 0.023, 0.041, 0.200
Assume the individual p-values are valid and that the conditions for the chosen procedure hold. Use a target level of 0.05. The comparison is educational; selecting whichever procedure gives the preferred outcome after seeing this vector would change the analysis plan.
Bonferroni compares every p-value with 0.05/5 = 0.01. Only the first hypothesis is rejected. Equivalently, multiply each p-value by five and cap at one. The underlying union-bound argument does not require independent tests. NIST presents this inequality and its use for simultaneous inference in its Bonferroni reference.
Holm examines the sorted values from smallest upward. Its thresholds here are 0.01, 0.0125, approximately 0.0167, 0.025, and 0.05. The first two pass. The third fails, so the step-down procedure stops and rejects only the first two. Do not restart at a later rank after that failure.
Benjamini–Hochberg, or BH, finds the largest rank k with p(k) <= k * 0.05 / 5, then rejects all hypotheses through that rank. Its thresholds are 0.01, 0.02, 0.03, 0.04, and 0.05. Rank three is the largest passing rank, so the first three are rejected. This is a step-up rule, not Holm with a different multiplier.
| Hypothesis | Raw p | Bonferroni adjusted | Holm adjusted | BH adjusted |
|---|---|---|---|---|
| H1 | 0.004 | 0.020 | 0.020 | 0.020 |
| H2 | 0.012 | 0.060 | 0.048 | 0.030 |
| H3 | 0.023 | 0.115 | 0.069 | 0.03833 |
| H4 | 0.041 | 0.205 | 0.082 | 0.05125 |
| H5 | 0.200 | 1.000 | 0.200 | 0.200 |
Official guidance: Holm provides FWER control without a dependence restriction beyond valid individual p-values. BH's standard guarantee holds under independence or specified positive regression dependence, not arbitrary correlation. BY offers FDR control under arbitrary dependence and can be more conservative. The R manual and SciPy FDR documentation explain these distinctions.
Check the arithmetic and preserve hypothesis identity
This Python example was tested with SciPy 1.18.1. It keeps names attached to values and checks the expected adjusted outputs.
import numpy as np
from scipy.stats import false_discovery_control
names = np.array(["H1", "H2", "H3", "H4", "H5"])
p = np.array([0.004, 0.012, 0.023, 0.041, 0.200])
m = len(p)
order = np.argsort(p)
ranked = p[order]
bonferroni = np.minimum(m * p, 1)
holm_ranked = np.minimum(
np.maximum.accumulate((m - np.arange(m)) * ranked), 1
)
holm = np.empty_like(p)
holm[order] = holm_ranked
bh = false_discovery_control(p, method="bh")
assert np.allclose(holm, [0.020, 0.048, 0.069, 0.082, 0.200])
assert np.allclose(bh, [0.020, 0.030, 0.115/3, 0.05125, 0.200])
for method, adjusted in [("Bonferroni", bonferroni),
("Holm", holm), ("BH", bh)]:
print(method, names[adjusted <= 0.05].tolist())
Expected rejections are H1; H1 and H2; and H1, H2, H3, respectively. The cumulative maximum in Holm's adjusted values preserves the step-down constraint. Merely multiplying each sorted value by its remaining count can produce an invalid adjusted sequence.
Official implementation detail: SciPy returns adjusted p-values in the input order. If your own implementation sorts values, restore their original positions before joining results back to metric names. The API reference documents this behavior.
Test a permuted input order and confirm that each name keeps the same adjusted value. Test a single hypothesis, for which these adjustments reduce to its original p-value. Reject malformed inputs rather than silently dropping failed calculations and shrinking the declared family.
Adjustment cannot repair an invalid experiment
A multiple-testing procedure assumes appropriate individual p-values. If repeated sessions were incorrectly treated as independent users, multiplying those p-values by the family size does not fix the standard errors. If analysts stop whenever a fixed-sample p-value crosses a threshold, ordinary endpoint correction does not automatically fix optional stopping.
Likewise, a broken exposure join, biased missingness, or a post-treatment filter can invalidate the underlying comparison. The adjustment operates on the resulting numbers; it cannot reconstruct the correct experimental population.
Explain what remains outside the guarantee. Effect estimates selected for extremeness may be exaggerated. Ordinary individual confidence intervals are not automatically simultaneous confidence intervals just because p-values were adjusted. A discovery is still a claim about a specified outcome and population, not a general promise that the product is beneficial.
A guardrail p-value above 0.05 also does not establish safety or non-inferiority. The experiment may be too imprecise to exclude a harmful effect. Use a predeclared harm margin and suitable interval or test, and explain how that requirement combines with the success criterion.
Answer the launch follow-up with evidence
Suppose H3 is conversion and is rejected by BH but not Holm. Do not respond, “BH says launch.” First state whether discovery-oriented FDR control was the planned criterion, whether the family was complete, and whether dependence conditions were justified. Then report the estimated conversion change, its uncertainty, practical importance, and guardrail evidence.
If the planned confirmatory rule was Holm, H3 did not meet that rule. It can motivate a follow-up experiment without being relabeled as a confirmed win. A clear answer preserves the analysis decision even when a different calculation looks more attractive.
Use these verified practice records to rehearse that explanation; they do not establish current employer-specific interview content.
| Practice question | What to demonstrate |
|---|---|
| Control error under multiple testing | Define the family and choose the error criterion |
| Explain P-value, Confidence Interval, and Multiple Testing Adjustments | Separate adjusted significance from effect-size uncertainty |
| Explain p‑values to a product manager | Avoid saying that FDR is a probability for each selected claim |
| Compute A/B significance, CI, and power | Connect the error budget to design and precision |
| Quantify improvement and compute required sample size | Explain why statistical significance alone does not settle launch value |
Practice Control error under multiple testing aloud using the five-value example. State the family, the guarantee, the calculation, and the claim that remains supportable after adjustment.