Permutation Test Interview Questions: Exchangeability, Paired Data, and a Python Check

A permutation test can produce a convincing p-value from an invalid shuffle. If treatment was assigned to households, shuffling individual household members invents assignments the experiment never allowed. If two measurements belong to the same person, pooling them can discard the pairing that makes the comparison meaningful.

The strongest answer to a permutation test interview question begins with one sentence: Under this null hypothesis, these particular rearrangements are allowed. Then define the statistic, enumerate or sample the permitted arrangements, and compare the observed result with that reference distribution.

This guide uses an original five-pair exercise and executable Python checks. Official software documentation supports the API details; the design choices and interview responses are our reasoning. No candidate report is used to claim interview frequency.

What does exchangeability actually require?

Exchangeability means that the joint distribution is unchanged by the rearrangements used in the test under the stated null. It is a property of a model or assignment mechanism, not a synonym for “the rows look similar.”

Consider six people randomly assigned to two equally sized groups. Under a sharp null of no treatment effect for any person, the outcomes stay fixed while treatment labels vary according to the actual randomization. There are twenty ways to select the three treated people. A randomization test can compare the observed mean difference with all twenty assignments.

That argument comes from the experiment's design. An observational comparison needs another justification for exchanging labels. Different age distributions, unequal outcome variability, or selection into treatment can undermine a naive pooled-label test. Equality of population means alone does not generally establish the stronger distributional invariance needed for an exact unstudentized permutation test.

A useful interview response is: “I would first ask whether this is a randomized experiment or an observational comparison. Then I would identify the assignment unit and the null under which outcomes can be rearranged.” This exposes the assumption before the interviewer has to prompt for it.

“Distribution-free” also does not mean assumption-free. A test can avoid a normal-distribution formula while still depending critically on the sampling design, independence structure, and permitted transformations.

Choose the rearrangement before choosing the library option

Official implementation detail: SciPy distinguishes independent sample-label reassignment, within-pair sample swaps, and changes to the pairings used in an association test. These correspond to independent, samples, and pairings. They answer different questions. See the SciPy permutation-test documentation.

Use this original decision table to explain your choice before coding:

Data and question	What may move?	What must stay together?	Warning to state
Independently randomized users in A and B	User treatment labels under the assignment design	Each user's outcome record	Preserve any allocation restrictions
Two treatments randomized within matched pairs	Treatment labels within each pair	Pair membership	Do not pool all measurements
Association between two independent sequences of measurements	One sequence's alignment with the other, under an appropriate independence null	Values within each sequence	This tests association, not a paired treatment shift
Treatment randomized to stores	Store-level assignments	All customers within each store	Customers are not independent assignment units
Repeated measurements over time	Only transformations justified by a time-series model or design	Relevant temporal dependence	Arbitrary timestamp shuffling destroys autocorrelation

The table is a reasoning aid, not a universal recipe. For example, store randomization stratified by region requires preserving the regional allocation constraints. A valid permutation scheme can be narrower than “shuffle whole stores.”

Official methodological guidance: FSL's PALM documentation describes exchangeability blocks and distinguishes rearrangements within blocks from rearrangements of whole blocks. That is useful language for explaining dependence, although its application-specific encoding is not required for this exercise. See exchangeability blocks.

Work through five matched pairs by hand

Our original experiment has five independent matched pairs. Within each pair, one unit was randomly assigned treatment and the other control with equal probability. Assume no interference and a sharp null that treatment changes no unit's outcome.

The observed treatment-minus-control differences are:

d = [1, 2, 3, 4, 5]
observed mean difference = 3

Swapping the treatment labels inside one pair reverses that pair's difference. Each pair has two allowed assignments, so there are 2**5 = 32 equally likely sign patterns. Pair membership never changes.

Define extremeness as an absolute mean at least as large as the observed absolute mean. The all-positive pattern gives a mean of 3; the all-negative pattern gives −3. Every mixed pattern has a smaller absolute mean. The exact two-sided p-value is therefore 2/32 = 0.0625.

All five differences favor treatment, yet this test does not reject at a prechosen 0.05 level. That is not a contradiction. The experiment has only five independent randomized pairs, and the exact reference distribution is coarse. Report the observed difference of 3 alongside the p-value rather than translating the result into “there is no effect.”

Now change the story: the five differences are before-and-after measurements from five employees who all received training. The same arithmetic still runs, but the random assignment argument disappears. A sign-flip test would need an appropriate symmetry assumption for differences under its null; it would not remove seasonality, learning effects, or other explanations for a change.

This is the key follow-up: identical arrays can require different inferential claims because their data-generating processes differ.

Verify the exact answer with a small Python oracle

This independent enumeration makes every permitted assignment visible. The second calculation checks a library call against it. Run with NumPy and SciPy installed; the example was checked with SciPy 1.18.1.

from itertools import product
import numpy as np
from scipy.stats import permutation_test

control = np.array([10, 20, 30, 40, 50], dtype=float)
treatment = control + np.array([1, 2, 3, 4, 5])
d = treatment - control

signs = np.array(list(product([-1, 1], repeat=len(d))))
null_means = (signs * d).mean(axis=1)
observed = d.mean()
p_exact = np.mean(np.abs(null_means) >= abs(observed) - 1e-12)

def mean_difference(x, y):
    return np.mean(x - y)

result = permutation_test(
    (treatment, control),
    mean_difference,
    permutation_type="samples",
    vectorized=False,
    n_resamples=np.inf,
    alternative="two-sided",
)

assert len(null_means) == 32
assert np.isclose(observed, 3.0)
assert np.isclose(p_exact, 0.0625)
assert np.isclose(result.pvalue, p_exact)
print(observed, p_exact)

The tiny tolerance is appropriate for this controlled floating-point example, not a universal threshold to copy into every test. Large or poorly scaled statistics need more deliberate numerical treatment.

Official API detail: SciPy's two-sided convention doubles the smaller one-sided tail. Our absolute-tail calculation agrees here because the sign-flip distribution is symmetric. Do not assume different two-sided definitions agree for every statistic or null distribution; record the convention explicitly. The same documentation describes exact enumeration and randomized resampling.

Useful checks go beyond obtaining the expected number. Reverse all differences: the two-sided p-value should remain unchanged. Replace every difference with zero: the p-value should be one. Reorder pairs consistently in both arrays: the result should remain unchanged. Reordering only one array changes pair membership and therefore changes the problem.

Why should a sampled permutation p-value not be zero?

Exact enumeration includes the observed arrangement in the complete reference set. Its tail proportion needs no extra pseudocount. Randomly sampling a limited number of arrangements is a different computation.

For a standard Monte Carlo permutation procedure with B random rearrangements and b statistics at least as extreme as observed, the familiar corrected estimate is (b + 1)/(B + 1). The original paper by Phipson and Smyth explains why treating a sampled reference distribution as continuous, and returning zero when no sampled statistic exceeds the observation, can understate p-values.

For example, zero exceedances among 999 sampled rearrangements gives 0.001 under that correction, not zero. This calculation concerns the specified tail-count procedure; blindly adding one to an already complete exact enumeration changes its answer.

Choose the simulation budget in relation to the decision. A coarse Monte Carlo result near a rejection threshold may need more draws or a stated simulation uncertainty. Reproducibility requires the seed, generator, statistic, alternative, preprocessing, and permutation scheme. A seed alone does not document the test.

Do not repeatedly change seeds until a result crosses 0.05. Likewise, selecting the most favorable statistic after inspecting several versions introduces a selection problem that a single permutation run does not correct.

What follow-up questions reveal a weak answer?

“Can I shuffle rows when each user contributes ten sessions?” Identify whether treatment was assigned to users or sessions. If assigned to users, retain the user's session bundle under the reassignment. Also decide whether the estimand weights users equally or weights every session equally. Preserving a cluster does not automatically choose the correct business metric.

“What if the groups have equal means but very different variances?” Do not promise exact validity for an unstudentized pooled-label test under a mean-only null. Discuss a suitable studentized statistic and the conditions supporting its approximation, or choose another justified procedure. State the inferential target before comparing software outputs.

“Is permutation the same as bootstrap?” In this exercise, rearrangement creates a reference distribution under a null while holding observed values fixed. A bootstrap typically resamples observational units with replacement to approximate sampling uncertainty. Neither method excuses choosing the wrong unit or ignoring dependence.

“Does p = 0.0625 mean the null has a 6.25% chance of being true?” No. It describes the specified extremeness probability over allowed assignments under the null and design. It is not a posterior probability, an effect size, or the probability that replication will fail.

“Can I keep collecting pairs until the test rejects?” Not under the original fixed-sample interpretation without accounting for the stopping rule. Plan the design and decision rule first, or use a method that explicitly handles sequential monitoring.

Practice the reasoning, not just the shuffle

Practice question	Permutation-test follow-up to explain
Explain p-value and choose correct test	Define the null and the meaning of extremeness before computing a number
Test classifier difference with McNemar's test	Explain why predictions on the same cases are paired
Compare two rare-event detection models statistically	Identify the independent unit and preserve shared evaluation cases
Connect Probability, Experiment Sizing, and Bootstrap Inference	Distinguish a null reference distribution from uncertainty estimation
Design a causal evaluation without A/B testing	Explain why observational reassignment requires additional assumptions

Start with the Test classifier difference with McNemar's test, then explain which records must remain paired and why. For this article's five-pair exercise, a complete answer names the design, preserves pair membership, computes 32 arrangements, reports 0.0625 with the effect estimate, and limits the claim to the assumptions actually supplied.

Sources and Further Reading

• SciPy: permutation_test API and null-distribution conventions
• Phipson and Smyth: Permutation p-values should never be zero
• FSL PALM: Exchangeability blocks