Bootstrap Interview Questions: Resampling Units, Paired Data, and Confidence Intervals

Bootstrap interview questions are rarely about typing a for loop 10,000 times. The difficult part is deciding what the data-generating unit is, what quantity you want to estimate, and which relationships a resample must preserve. A polished implementation can still answer the wrong question if it samples events when users are independent, or samples two model outputs separately when they were measured on the same cases.

This guide uses current SciPy documentation and one original synthetic dataset. It distinguishes documented library behavior from editorial interview guidance and from results observed in the local example. The example demonstrates mechanics; it does not prove that one bootstrap design is always more conservative or that bootstrap intervals are valid for every dataset. Start with Calculate 95% Bootstrap Confidence Interval for Order Values, then return here to change the resampling unit and preserve pairing.

Start with the estimand, not the loop

An estimand is the population quantity the analysis is trying to learn. “Model B is better” is not an estimand. “The mean reduction in loss per evaluation event” and “the mean reduction for an equally weighted user” are different estimands. They can produce different point estimates before any bootstrap begins.

Use this sequence in an interview:

1. State the target quantity and its population.
2. Identify the units that were sampled or randomized independently.
3. Identify observations that must stay together.
4. Define the statistic on one resampled dataset.
5. Only then choose the interval method and number of resamples.

For the synthetic example, lower loss is better and each row contains user_id, Model A loss, and Model B loss for the same event. Improvement is A - B. There are 17 rows from eight users. Users 1–3 contribute multiple rows; users 4–8 contribute one each. If the estimand weights events equally, the observed mean improvement is 0.0335. If it first averages within each user and then weights users equally, the observed mean improvement is 0.0350.

That numerical difference is small here, but the conceptual difference is not. An event-weighted answer lets high-activity users contribute more. A user-equal answer gives each user the same weight. Neither is automatically correct. The product question and sampling design decide.

Preserve pairing when two measurements share a case

Suppose Models A and B score the same 17 events. Their outputs are paired by event. The statistic can be written as the mean of the 17 within-event differences:

def mean_improvement(model_a, model_b, axis=-1):
    return np.mean(model_a - model_b, axis=axis)

Official SciPy behavior: when scipy.stats.bootstrap receives multiple samples with paired=True, it resamples one array of indices and applies those same indices to every sample. With paired=False, it resamples the samples independently. The current function defaults to paired=False, so paired data require an explicit choice.

Why does that matter? If index 6 refers to one event, a paired draw takes both A[6] and B[6]. An independent draw might compare A[6] with B[12]. The second operation discards the within-event relationship and estimates variability for a different sampling process.

The local exercise used the same random seed and 10,000 percentile resamples. The paired row-level 95% interval was [0.0100, 0.0565]. Independently resampling the two model arrays produced [-0.0371, 0.1035]. Its bootstrap standard error was 0.0356, compared with 0.0119 for the paired design.

Those widths are an observed result for this positively correlated synthetic dataset. Do not memorize “independent is always wider.” With a different dependence structure, the direction can change. The defensible conclusion is that independent resampling breaks the design and estimates the wrong variability when observations are paired.

Choose between rows and users as the resampling unit

Pairing solves one dependency: Model A and Model B stay aligned for an event. It does not solve repeated events from the same user. Treating 17 rows as independent suggests 17 independent pieces of information, even though behavior within one user may be correlated.

For a user-level bootstrap, first build eight user records. In each resample, draw eight user IDs with replacement and include the selected user-level statistic. For a user-equal estimand, the implementation averages each user's improvement and then averages the sampled user means:

user_effects = frame.groupby("user_id")["improvement"].mean().to_numpy()

rng = np.random.default_rng(20270910)
replicates = []
for _ in range(10_000):
    idx = rng.integers(0, len(user_effects), len(user_effects))
    replicates.append(user_effects[idx].mean())

The original user-cluster percentile interval was [-0.0162, 0.0825]. It is wider than the paired row interval in this example because there are only eight independently resampled users and their effects vary. Again, that single comparison does not prove a universal ordering. It exposes how much the answer depends on the assumed independent unit.

If the desired estimand remains event weighted while uncertainty is clustered by user, simply averaging user means changes the target. You may need to resample users while retaining all of their rows and recompute the event-weighted statistic, or use a method designed for the sampling structure. State how repeated selections and cluster sizes are handled. “Bootstrap by user” is incomplete unless the statistic and weighting are explicit.

Read the Python implementation as a statistical contract

An interviewer may give working code and ask what is wrong. Trace the index construction rather than focusing first on syntax.

Implementation	What one draw samples	Relationship preserved	Main question
rng.choice(n, n, replace=True) applied to both arrays	Paired rows	A/B match for each case	Are rows independent?
Separate index draws for A and B	Marginal A and B rows	Neither case pairing nor user grouping	Was independence intended?
User IDs drawn, then all selected-user rows included	User clusters	Within-user rows and A/B pairs	Is the statistic user or event weighted?
Blocks of adjacent timestamps drawn	Time blocks	Local temporal dependence	How was block length justified?

Check five implementation details:

• Replacement: an ordinary nonparametric bootstrap samples with replacement. A permutation without replacement serves a different purpose.
• Sample size: a typical replicate has the same number of units as the observed sample, although specialized methods can differ.
• Recomputation: calculate the complete statistic in every replicate. Do not bootstrap an already summarized number that removed needed structure.
• Random generator: seed or pass a generator for a reproducible exercise. SciPy's current documentation recommends the rng keyword for new code.
• Vectorization and memory: SciPy can call a vectorized statistic in batches. More resamples can require more memory; batch changes computation, not the estimand.

The local script records NumPy 2.5.3, SciPy 1.18.1, seed 20270910, both point estimates, three percentile intervals, and six assertions. A reviewer can rerun it and compare the short JSON output. That is stronger evidence than saying “I tested the bootstrap.”

Explain percentile, basic, and BCa intervals carefully

Official SciPy behavior: scipy.stats.bootstrap currently offers percentile, basic, and BCa methods, with BCa as the default and 9,999 resamples by default. BCa adjusts for estimated bias and acceleration. The methods transform the bootstrap distribution differently, so they need not return identical endpoints.

Using SciPy's default BCa method with 9,999 paired row resamples, the local example returned [0.0088, 0.0559]. The manual 10,000-resample percentile interval was [0.0100, 0.0565]. The closeness here does not license choosing a method after seeing which answer is more attractive.

In an interview, explain the method you selected, its assumptions, and a diagnostic. For BCa, SciPy documents a DegenerateDataWarning; endpoints may be NaN when the bootstrap distribution is degenerate, such as when all observations are identical. Switching mechanically to percentile may produce finite numbers without creating missing information. Inspect the data, statistic, effective number of independent units, and whether another analysis is more appropriate.

More resamples reduce Monte Carlo noise in the estimated bootstrap distribution. They do not increase the number of observed users, repair selection bias, make dependent rows independent, or guarantee good coverage. Report the seed and run a stability check when endpoints materially affect a decision.

Interpret a 95% confidence interval without a probability shortcut

A frequentist 95% confidence procedure is evaluated over repeated samples: under its conditions, about 95% of intervals produced by the procedure contain the fixed population quantity. After one interval is computed, avoid saying there is a 95% probability that the fixed parameter lies inside it. That Bayesian-sounding statement requires a probability model for the parameter.

Also avoid equating “the interval excludes zero” with “the change is important.” If loss units have a business interpretation, compare the interval with a smallest meaningful improvement. If they do not, translate the metric before making a decision. An interval crossing zero signals that the data and procedure do not separate the effect from zero at that confidence level; it does not prove equality.

Coverage claims depend on the sampling assumptions and the interval method. A bootstrap cannot reproduce users, seasons, devices, or treatment assignment mechanisms that are absent or misrepresented in the observed data. Discuss external validity separately from resampling uncertainty.

Diagnose four common interview failures

Two models, same examples: The candidate resamples predictions independently. Fix the indices first: draw cases and use the same draw for both arrays. State that paired=True implements this behavior in SciPy.

Many events per user: The candidate samples rows and calls the resulting interval “user uncertainty.” Ask which units are independent and whether high-activity users should receive more weight. Rebuild the replicate at the user level.

All values identical: BCa returns a warning or unusable endpoint. Do not hide it. A degenerate empirical distribution cannot reveal variation that was not observed. Verify data preparation and reconsider the inferential question.

Very few clusters: Running one million resamples still rearranges only the few observed clusters. Simulation precision may look excellent while inferential information remains weak. Report the small effective sample, use domain-appropriate methods, and avoid a confident product claim.

Structure a strong interview answer

Use a compact six-part explanation:

1. Estimand: “I want the equally weighted mean user improvement in loss.”
2. Unit: “Users are the independent sampled units; events repeat within users.”
3. Dependence: “Model outputs are paired within each event, and events remain grouped within a user.”
4. Algorithm: “Draw users with replacement, retain their paired observations, recompute the user-level statistic, and repeat with a fixed generator.”
5. Interval: “Use a preselected method, report its endpoints, and inspect warnings and stability.”
6. Limit: “Eight users provide limited information; more resamples do not add users.”

If the interviewer changes the estimand to average event impact, revise the statistic and weighting openly. If they reveal randomization happened by household, move the unit again. The ability to update the design when the data-generating process changes is the core skill.

Sources and Further Reading

• SciPy: scipy.stats.bootstrap — official API behavior for pairing, methods, defaults, RNG, batching, and degenerate data.
• scikit-learn: sklearn.utils.resample — official consistent-array resampling utility and replacement behavior.
• Field and Welsh: Bootstrapping clustered data — peer-reviewed treatment of bootstrap choices for clustered observations.