T-Test Interview Questions: Paired Data, Unequal Variance, and Effect Sizes

An interviewer gives you two columns of numbers, asks whether the means differ, and watches which question you ask first. The answer depends less on memorizing a formula than on identifying what one row represents. If the columns come from the same people before and after a change, their pairing is evidence. If they come from different people, treating row 1 in one group as related to row 1 in the other invents a relationship. This guide uses a deliberately small, original dataset to show how the same group averages can support sharply different answers when the pair IDs are preserved or lost.

Evidence boundary: SciPy documentation defines the tests and their software behavior; the numbers, interview prompts, and answer framing below are original editorial exercises. No candidate report is used to claim a specific employer's format.

First interview question: what is the unit of comparison?

Ask how observations were collected before calculating anything. In a before/after design, each participant, device, store, or matched site contributes a pair. Define a difference for each unit, such as after - before, then test whether the mean difference is zero. SciPy's ttest_rel is explicitly for related samples and calculates its statistic from the mean of the pairwise differences divided by their standard error. The samples must align and have the same shape. A row-order mistake changes the estimand's variance even when neither column's mean changes.

In a randomized experiment with distinct users in control and treatment, the groups are independent. The relevant standard error uses variation within both groups. SciPy's ttest_ind is for independent samples; its default equal_var=True uses an equal-variance assumption, while equal_var=False performs Welch's test. If the data are repeated sessions from the same user, neither pretending that all sessions are independent nor pairing arbitrary rows solves the clustering problem. Say what was randomized, what was measured, and where dependence can enter.

Data-generating design	Observation used in the analysis	First test to consider	Failure to avoid
The same eight people measured twice	Eight within-person differences	Paired t-test	Sorting one column independently and destroying IDs
Eight people in each of two separate groups	Sixteen people, split by group	Welch two-sample t-test	Pairing rows only because both files have eight rows
Matched pairs of stores formed before assignment	One treatment-control difference per matched set	Paired analysis of set differences	Discarding the matching variable
Multiple sessions per user	Users or a model of within-user dependence	User-level or cluster-aware analysis	Counting sessions as independent users

The table is a selection aid, not a claim that a t-test always works. Check the outcome scale, design, missingness, extreme values, and the question the analysis must answer. A binary outcome, a small sample with severe skew, or a time series with dependence may need a different method or a more careful sensitivity analysis.

Worked exercise: identical marginal means, different pairing

Imagine eight technicians taking a timed diagnostic task before and after practice. Their baseline scores vary widely. Each person's later score is only four to six points higher. These are synthetic teaching data, not an observed intervention.

Technician ID	Before	After	After minus before
1	100	104	4
2	120	125	5
3	140	146	6
4	160	165	5
5	180	184	4
6	200	206	6
7	220	225	5
8	240	245	5

The before mean is 170 and the after mean is 175. The observed mean improvement is 5. Yet the baseline standard deviation is about 48.99, while the standard deviation of the eight differences is about 0.756. That is the entire point of pairing: each person's stable baseline is removed before estimating the variability of change. The paired standard error is 0.756 / sqrt(8) ≈ 0.267, giving t ≈ 18.708 with seven degrees of freedom. A two-sided paired test returns p ≈ 3.10 × 10^-7 and a 95% interval for the mean change of approximately [4.368, 5.632].

Now reverse the order of the after column but leave every value in that column untouched. Its mean is still 175. The before mean is still 170. If you mistakenly run a paired test on these misaligned rows, the differences swing from about +145 to -135. Their standard deviation is about 98.16; the resulting paired t ≈ 0.144 and p ≈ 0.890. The confidence interval expands to roughly [-77.06, 87.06]. This is not a legitimate alternative analysis of the same matched study. It is a data-join failure. The correct response is to restore IDs and check uniqueness, not to pick the p-value you prefer.

If these actually were two independent groups with the same marginal values, a Welch test would use both groups' standard deviations and return t ≈ 0.204, p ≈ 0.841, and about 14 degrees of freedom. Its conclusion differs from the legitimate paired analysis because the design and standard error differ. The numerical example intentionally makes within-person changes unusually consistent so the contrast is visible; real data rarely look this neat.

A strong interview answer says: “First I would verify that the person ID maps one before score to the same person's after score. With valid pairs I analyze the eight differences. If these are independent groups, I use a two-sample method, usually Welch, and I do not manufacture pairs from row position. The group averages alone cannot tell me which design I have.” That sentence demonstrates data integrity, method choice, and interpretation before any arithmetic.

Reproduce the answer, including the broken join

The following Python uses SciPy's documented functions. It checks means before comparing results so a reviewer can see that only the relationship between columns changed. Run it in a pinned environment for a truly reproducible submission; SciPy result object features can vary by installed version.

import numpy as np
from scipy import stats

before = np.array([100, 120, 140, 160, 180, 200, 220, 240], dtype=float)
after = np.array([104, 125, 146, 165, 184, 206, 225, 245], dtype=float)
broken_after = after[::-1]

assert before.mean() == 170
assert after.mean() == broken_after.mean() == 175

for label, candidate_after in [("matched", after), ("broken IDs", broken_after)]:
    differences = candidate_after - before
    result = stats.ttest_rel(candidate_after, before)
    ci = result.confidence_interval()
    print(label, differences.mean(), differences.std(ddof=1),
          result.statistic, result.pvalue, ci.low, ci.high)

independent = stats.ttest_ind(after, before, equal_var=False)
print("independent Welch", independent.statistic,
      independent.pvalue, independent.df)

Expected rounded output is: matched difference 5.0, SD 0.756, t=18.708, p=3.10e-7; broken IDs difference 5.0, SD 98.157, t=0.144, p=0.890; independent Welch t=0.204, p=0.841, df≈14.000. The ttest_rel interval is for after - before because that is the argument order. Swapping arguments flips the sign of the statistic and interval without changing a two-sided p-value. This sign detail is easy to miss in an interview answer.

Before trusting a production dataset, assert one row per ID and measurement occasion, inspect unmatched IDs, and decide whether complete-case analysis changes the population represented. A missing after score does not justify pairing that participant's before score with the next row. If matching was probabilistic rather than exact, describe the linkage uncertainty and whether the pairing was specified before observing outcomes. These checks often matter more than another decimal place in the test statistic.

Second question: what if the independent groups have unequal variance?

Unequal variance is a separate issue from pairing. For genuinely independent groups, Welch's test estimates the mean difference using sqrt(s1²/n1 + s2²/n2) as its standard error and uses an approximate degrees-of-freedom calculation. SciPy exposes it with equal_var=False. Its equal-variance default uses a pooled estimate instead. In a live interview, say the assumption out loud and choose the method consistent with the design; do not run every test and report the smallest p-value.

Third question: what does “effect size” mean here?

Start with the raw mean difference in domain units. Five score points may be important or trivial depending on the task and its decision threshold. The paired example's estimate is a five-point mean gain; its interval conveys sampling uncertainty under the test model. Statistical significance does not tell you whether the change improves a real decision, whether the design supports a causal claim, or whether the effect will generalize.

A standardized effect divides a difference by some standard deviation, so the denominator must be named. For paired data, one possible statistic is d_z = mean(differences) / sd(differences). In our synthetic matched example that is about 6.61, strikingly large because the constructed changes are almost constant. It is not interchangeable with an independent-groups Cohen's d based on pooled outcome variability. Standardized numbers from differently paired designs may look incomparable because they use different sources of variation. State the formula and report the raw effect alongside it.

For independent samples, a standardized mean difference may use pooled outcome SD, with a small-sample correction such as Hedges' g when appropriate. If group variances differ strongly, ask whether a single common SD is meaningful before presenting one number as universal. An effect size is a descriptive scale, not a substitute for a confidence interval or for the metric's business threshold. In an interview, I would lead with “a five-point difference, interval X to Y, relative to a threshold of Z,” then add a clearly defined standardized measure only if it helps comparison across studies.

Assumptions and edge cases an interviewer may probe

For a paired t-test, the independent units are the pairs, and the t approximation concerns the distribution of their differences. Inspect those differences, not just the two marginal histograms. Extreme differences can dominate a sample of eight. The synthetic exercise is a mathematical illustration, so its p-value is not empirical evidence of a real treatment. With very few pairs and severe skew or outliers, discuss a paired sign or signed-rank analysis, a permutation approach that respects pairing, or a bootstrap interval, while explaining that each may target a different quantity or need its own assumptions.

For independent groups, observations should be independent at the chosen analysis unit. Random assignment of users can support causal interpretation if exposure and measurement are correct, but a t-test cannot itself fix selective logging or interference among users. If a platform randomizes by household and measures individuals, account for households rather than assuming each individual contributes an independent draw. If a before/after study has no control group, keep the wording descriptive unless the design justifies stronger causal claims.

Missing values deserve their own answer. SciPy exposes nan_policy controls, but “omit” is a computational behavior, not proof that missingness is ignorable. In a matched design, dropping incomplete pairs may change who remains. Report how many pairs were expected, how many were analyzed, and why. If scores are censored or the measurement scale has a hard ceiling, say so before interpreting the mean. If you test many outcomes or repeatedly inspect the result, address the increased false-positive risk through a planned analysis rather than treating p < .05 as a magic boundary.

Sources and Further Reading

• SciPy, ttest_rel — official API and definition for related samples, statistic sign, and confidence interval.
• SciPy, ttest_ind — official API for independent groups and the equal_var=False Welch option.
• SciPy, bootstrap — official documentation on resampling, including paired index resampling.