Articles
ANOVA Interview Questions: Group Differences, Unequal Variance, and Follow-Up Comparisons
Practice ANOVA interviews with executed ordinary and Welch tests, unequal-variance Games-Howell comparisons, assumptions, and interpretation limits.
An ANOVA interview question may begin with “Do these three groups differ?” and end with a harder decision: which analysis is defensible when group sizes and variances differ, and what does a significant result actually establish? The worked example below is constructed for practice and its calculations were run with SciPy 1.18.1. It is not a real experiment or a reported employer question. SciPy's f_oneway documentation describes ordinary one-way ANOVA and the equal_var=False Welch option; the documentation distinguishes their variance assumptions.
If you want a quick hypothesis-testing warm-up, try Choose and Explain a Classical Hypothesis Test before calculating the fixture below.
What question does a one-way ANOVA answer?
Suppose a product analyst has three groups, A, B, and C, each with a numeric outcome. The one-way ANOVA null hypothesis is that the three population means are equal. The alternative is that at least one population mean differs. The omnibus F statistic compares variation between group means with variation within groups. A small p-value can reject the joint equality claim under the test assumptions. It does not say every pair differs, identify which pair differs, measure business importance, or prove that the grouping caused the outcome. State those limits early in an interview.
The word “population” matters. The observed sample means almost never match exactly, even when the population means do. ANOVA asks whether the between-group separation is large relative to the variability expected under the null model. The result depends on the study design: independent observations and a suitable residual model cannot be created by selecting a convenient function call. NIST's one-way ANOVA reference presents the decomposition and assumptions. Use the formula to explain the mechanism, then return to the data-generating process.
Work the unequal-size example by hand
Take three small samples. A has 8, 9, 10, 11. B has 8, 9, 10, 11, 12, 13. C has 13, 16, 19, 22, 25. These are invented observations with unequal group sizes and visibly unequal spreads, chosen to make a method choice discussable. They are not patient, customer, or candidate data. The summary was computed from the exact values below.
| Group | Values | n | Mean | Sample variance |
|---|---|---|---|---|
| A | 8, 9, 10, 11 | 4 | 9.5 | 1.667 |
| B | 8, 9, 10, 11, 12, 13 | 6 | 10.5 | 3.500 |
| C | 13, 16, 19, 22, 25 | 5 | 19.0 | 22.500 |
There are 15 observations. The grand mean is 13.0667. For ordinary one-way ANOVA, sum each group size times its squared distance from the grand mean: the between-group sum of squares is 266.4333. Sum each observation's squared distance from its own group mean: the within-group sum of squares is 112.5000. The degrees of freedom are 2 between and 12 within, so the mean squares are 133.2167 and 9.3750. Their ratio is F = 14.2098, with p = 0.000685 under the ordinary F reference distribution.
An interviewer may ask what the F ratio means. It is not “14 times more likely that the groups differ.” It is a ratio of two estimated variance components under this model. The p-value is the chance, assuming the null model and its assumptions, of an F statistic at least as extreme as the observed one. It is not the probability the null is true. The arithmetic tells you what the software computed; it does not certify that the ordinary test was the right one.
The variability is a warning. C's sample variance is 22.5, far above A's 1.667, and C does not have the same sample size as the other groups. A ratio from such a tiny constructed fixture is not a reliable variance test by itself, but it is enough to require a method discussion. Ask how the data were collected, inspect group distributions and outliers, and consider a variance-robust procedure. A normality test on groups of four to six observations cannot rescue a flawed independence assumption or establish a population model with confidence.
When does Welch ANOVA change the answer?
Ordinary one-way ANOVA pools within-group variance. Welch ANOVA changes the weighting and degrees of freedom to accommodate unequal variances. On the same fixture, SciPy f_oneway(A, B, C, equal_var=False) returns F = 8.4293 and p = 0.012632. Both ordinary and Welch results reject at a pre-specified 0.05 threshold here, but the p-values differ materially. The point is not to shop between them for a smaller number. The point is to choose a procedure based on the design and variance concern before seeing which result is most convenient.
With unequal sizes and visibly different spreads, Welch is a reasonable primary answer for this constructed case. State what it does and does not fix. It addresses unequal variances in the mean comparison; it does not make repeated measurements independent, remove selection bias, correct a mislabeled outcome, or make an outlier harmless. If all observations came from the same person measured across conditions, the one-way independent-groups model is wrong. If assignment to groups was not randomized, a group difference remains an association and may reflect confounding.
What if group distributions are strongly skewed or bounded? Inspect the outcome's scale and scientific question. A transformation may alter interpretation; a robust or permutation-based approach may be more appropriate for some designs. These are not automatic “nonparametric fixes.” A permutation test requires an exchangeability argument, and rank-based tests generally answer a different question from equality of means. An interview answer should explain why a method matches the estimand and sampling process rather than reciting a menu of alternatives.
Why does a significant omnibus test not settle the pairs?
The null says A, B, and C all share one population mean. Rejecting it means at least one mean differs under the model. It does not identify A versus B, A versus C, or B versus C. Testing each pair with an unadjusted 0.05 threshold would inflate the chance of at least one false claim across the family. Decide which contrasts matter before looking at all the p-values. If the task truly asks for all three pairwise comparisons, use a method that addresses the family.
For unequal variances, SciPy's tukey_hsd documentation implements Games–Howell when equal_var=False. In this exact fixture, the adjusted pairwise p-values are A–B 0.5974, A–C 0.0200, and B–C 0.0289. At a pre-specified 0.05 family-wise threshold, the two comparisons involving C are supported, while A–B is not. These numbers were executed locally; they are not generic thresholds or evidence about a real experiment. With tiny n and high dispersion, report uncertainty and avoid treating the observed differences as stable effect estimates.
| Question | Suitable output for this fixture | What it cannot tell you |
|---|---|---|
| Do any group means differ? | Welch omnibus F = 8.4293, p = 0.0126 | Which pair differs or why |
| Which pairs differ under unequal variances? | Games–Howell adjusted A–C and B–C p-values below 0.05 | Whether the differences are practically important |
| Is A different from B? | Games–Howell adjusted p = 0.5974 | That the true means are identical |
| Did group membership cause the gap? | Requires design and confounding analysis | Cannot follow from ANOVA alone |
If the interviewer asks about Tukey HSD, distinguish it from the Welch-compatible choice. SciPy's pairwise function uses Tukey-Kramer for unequal sample sizes when equal variances are assumed, and Games–Howell when equal_var=False. Unequal n alone does not mandate Games–Howell; the variance assumption matters. NIST's multiple-comparison discussion gives primary background on the family of pairwise comparisons. If only a few preplanned contrasts matter, an appropriate contrast and multiplicity plan may be more efficient than all pairs. Say which question you are actually answering.
Check assumptions before interpreting the calculation
First ask whether observations are independent. Were participants randomized once, or did one participant contribute several records? Were measurements clustered by team, clinic, or week? Independence is a property of collection, not of a histogram. If one customer contributes repeated transactions, ordinary one-way ANOVA underestimates the complexity of the sampling process. A design-aware repeated-measures, mixed, clustered, or hierarchical analysis may be needed. Do not choose among these without the actual study structure.
Second, inspect the outcome within each group. Plot individual values with a useful scale, then look for a single value dominating C's spread. The fixture's C values are widely spaced by construction. A boxplot can reveal spread but conceal the tiny sample size; individual points make n visible. Third, check variance differences and consider whether Welch addresses them. Fourth, define the estimand: means of the measured outcome, a transformed mean, or a business threshold may lead to different decisions. Fifth, consider power and uncertainty. Failing to reject A–B in this fixture does not establish equivalence; that would require an equivalence margin and appropriate design.
An interviewer may ask whether to run a preliminary variance test and switch methods only when it is significant. That two-stage strategy can make inference depend on a low-power diagnostic, especially with n this small. Better to use domain and design knowledge, graphical diagnostics, and a pre-specified robust default when variance heterogeneity is plausible. Describe any sensitivity analysis transparently: ordinary and Welch lead to the same broad omnibus conclusion here, but the ordinary p-value is more optimistic. Do not select the ordinary result merely because it looks more impressive.
Give a complete answer in two minutes
A concise answer can sound like this: “The null is equal population means across A, B, and C. Ordinary ANOVA gives F(2, 12) = 14.21, p about 0.000685, but C has much larger sample variance and the groups have unequal n. I would check independence from the study design and use Welch as the primary mean comparison here: F about 8.43, p about 0.0126. That supports at least one mean difference, not a particular pair or a causal effect. If all three pairs matter, Games–Howell adjusted comparisons flag A–C and B–C in this toy example, but not A–B. I would report effect sizes, intervals, and the design limitations before recommending an action.”
If the interviewer changes the story—say the same subjects received all three treatments—stop using these results as the answer. The arithmetic remains correct for the listed numbers, but the inferential reference distribution no longer matches the repeated-measures design. That distinction between a correct calculation and a valid conclusion is the most important skill this question can reveal.