Articles
Data Scientist Statistics Screen: Short Questions With Follow-Up Reasoning
Practice short statistics answers with follow-ups on p-values, intervals, base rates, sample independence, conversion lift, and bootstrap assumptions.
You answer “a p-value below 0.05 is significant,” and the next question is: “Significant enough to launch?” The definition has ended; the reasoning has started. Statistics interview preparation becomes more useful when every short answer is followed by a change in the assumptions, the data, or the decision.
This guide gives you a set of original practice prompts with answers you can say aloud, numerical checks, and follow-ups that expose common mistakes. Use the Data Scientist Statistics and Math collection to extend the practice afterward.
Evidence boundary: statistical definitions below use primary educational and professional sources. The prompts, examples, and rehearsal structure are our preparation recommendations. No candidate report is used to claim that every employer runs this format, asks these exact questions, or applies a particular scoring rubric.
Give an answer that can survive one follow-up
For rehearsal, use three parts: a direct claim, the assumption that makes it valid, and the consequence for the problem. Start briefly, then expand when asked. A concise answer should leave room for discussion without hiding an essential condition.
For example: “A larger independent sample usually reduces the standard error of a mean. That requires a stable population and finite variance. Collecting more biased observations will not repair the sampling bias.” You have answered the question and identified where the answer could fail.
Avoid listing every test you know before identifying the estimand: the population quantity you want to learn. A mean transaction value, median user spend, and probability of any purchase require different summaries even when they come from the same table.
1. What does a p-value actually tell you?
Short answer: under a specified null model, it is the probability of a test statistic at least as extreme as the observed statistic, with “extreme” defined by the test. It is not the probability that the null hypothesis is true.
Professional guidance: the American Statistical Association statement also distinguishes statistical significance from effect size and business importance. A threshold alone does not settle a decision.
Follow-up: “The p-value is 0.001. Should we launch?” Ask for the estimated effect, uncertainty, decision threshold, guardrail outcomes, and study design. A precisely measured improvement of 0.001 percentage points could be too small to pay for the implementation. A tiny p-value does not remove selection bias or repair broken randomization.
A second follow-up might change 0.001 to 0.2. Say that the result does not provide strong evidence against the specified null at a conventional threshold. Do not conclude that the effects are identical. The data may be too imprecise to distinguish useful gains from losses.
2. What does a 95% confidence interval mean?
Short answer: in the frequentist interpretation, the method produces intervals covering the fixed population parameter in 95% of repeated samples, under its assumptions. The confidence level describes that procedure.
NIST’s confidence-interval explanation distinguishes repeated-sample coverage from assigning a 95% probability to the fixed parameter being inside the particular observed interval. A Bayesian credible interval uses a different framework and requires a model and prior.
Follow-up: “The interval includes zero. Is there no effect?” No. Inspect its width and the decision-relevant values it includes. An interval spanning substantial harm and substantial benefit leaves a different decision problem from a narrow interval entirely inside a prespecified negligible-effect range.
For a rough numerical check, suppose a large independent sample has mean 20 and estimated standard error 2. A normal approximation gives 20 ± 1.96 × 2, or about 16.08 to 23.92. Explain why you chose the approximation; do not treat 1.96 as a universal critical value for every sample or statistic.
3. Does the central limit theorem make the data normal?
Short answer: no. In its familiar independent, identically distributed, finite-variance form, the theorem concerns the standardized sample mean as sample size grows. The original observations can remain skewed. See these primary lecture notes on the central limit theorem.
Follow-up: “We have 40 observations, so can we assume the mean is normal?” There is no universal sample-size cutoff that guarantees a good approximation. Tail behavior and the required accuracy matter. Inspect the data and consider how sensitive the conclusion is to unusual observations.
A more revealing change is “40 million events from 500 users.” Events from the same user may be dependent. Forty million rows are not automatically forty million independent units. Identify the sampling or randomization unit and account for clustering before calculating an impressively small standard error.
If asked whether quadrupling the independent sample size halves the standard error, answer yes for the usual mean formula with otherwise comparable variance. Then state that the underlying observations do not become half as variable; the estimate becomes more precise.
4. Why can a good detector produce mostly false alerts?
Short answer: precision depends on the base rate as well as sensitivity and false-positive rate. Reversing a conditional probability requires the prevalence of the condition. MIT’s probability lecture develops Bayes’ rule and conditional probability.
Consider an original toy fraud detector applied to 10,000 accounts. Assume 1% are fraudulent, sensitivity is 90%, and the false-positive rate is 5%.
| Actual group | Accounts | Flagged | Not flagged |
|---|---|---|---|
| Fraudulent | 100 | 90 | 10 |
| Legitimate | 9,900 | 495 | 9,405 |
| Total | 10,000 | 585 | 9,415 |
Only 90 of 585 alerts are true positives: about 15.4%. Sensitivity of 90% did not imply 90% precision. The larger legitimate population supplies many false alerts even at a relatively low false-positive rate.
Follow-up: “What if fraud prevalence rises?” Holding the two conditional error rates fixed, precision increases. In practice, a population change may also change those rates, so verify that assumption. An operational recommendation must also consider review capacity and the costs of missed fraud and false accusations.
5. Which test would you use to compare two groups?
Short answer: first establish the outcome, estimand, and dependence structure. For an independent two-group comparison of means, Welch’s t-test does not require equal variances. For matched before-and-after measurements, the pairing changes the analysis.
NIST’s two-sample test reference distinguishes paired and unpaired data and equal- versus unequal-variance formulations.
Follow-up: “Both samples contain the same users.” Analyze within-user differences when that matches the design and question. Treating the groups as independent discards their correspondence. A paired analysis still needs appropriate assumptions about differences across users.
Another follow-up changes the outcome from spending to conversion. Now each independent user contributes a binary outcome, and the target might be a difference in proportions. Do not choose the procedure merely because both datasets contain numbers. Explain the parameter and sampling structure before the formula.
6. What does a two-point conversion increase establish?
Take an original fixed-horizon experiment with 1,000 independent users per group: 100 control users convert, compared with 120 treatment users. Assume valid random assignment and complete measurement.
The observed rates are 10% and 12%. The absolute increase is 2 percentage points; relative lift is 20%. Those statements describe the same estimate on different scales.
For an illustrative large-sample interval, use the unpooled estimated standard error:
difference = 0.12 - 0.10 = 0.02
SE = sqrt(0.12 * 0.88 / 1000 + 0.10 * 0.90 / 1000)
≈ 0.0140
approximate 95% interval = 0.02 ± 1.96 * 0.0140
≈ [-0.0074, 0.0474]
That is roughly −0.74 to +4.74 percentage points. It includes zero and also includes gains the business might consider valuable. This example supports “the estimate is positive but uncertain,” not “the treatment definitely works” or “the treatment has no effect.”
Follow-up: “We checked every hour and stopped when it looked good.” The fixed-horizon interpretation no longer follows automatically. Discuss a prespecified stopping rule or an appropriate sequential method. If assignment is by household, revisit independence as well. Correct arithmetic cannot compensate for a mismatched design.
7. Should you report a mean or a median?
Short answer: choose the summary that answers the question. For values 0, 0, 0, 0, and 100, the mean is 20 and the median is 0. Neither calculation is wrong. They describe different features of the distribution.
If the stakeholder asks about a typical user, the median may be informative. If they ask about aggregate revenue per user, the mean has a direct relationship to the total. Report the concentration or relevant quantiles when a single number hides the distribution.
Follow-up: “Can we remove the 100?” Investigate whether it is a measurement error or a valid high-value observation. Removing valid outcomes changes the target quantity. A trimmed mean can be useful when explicitly chosen, but it should not silently replace the requested metric because the answer looks more comfortable.
If many users have exactly zero spend, discuss purchase probability and spending conditional on purchase. That decomposition can reveal why a stable median conceals an important business change.
8. Does bootstrapping remove the need for assumptions?
Short answer: no. Resampling can approximate a statistic’s sampling variability, but its validity depends on how the observed sample represents the population and how the resampling respects dependence.
The official SciPy bootstrap documentation describes sampling with replacement and explicitly distinguishes paired resampling from independent resampling of multiple arrays. This is a useful implementation-level reminder that the resampling unit matters.
Follow-up: “Each user has 20 sessions. Can we resample sessions?” If sessions within users are dependent, ordinary independent row resampling can misrepresent uncertainty. Consider resampling users, keeping their relevant records together, when that matches the sampling design and estimand.
More resamples reduce Monte Carlo variability in the bootstrap computation. They do not create more original users, recover an unobserved population segment, or guarantee reliable tail inference from a weak sample. Explain what the procedure approximates before naming a library option.
Rehearse the follow-up, not just the definition
Use a partner to change one condition after each answer: independence becomes clustering, a fixed horizon becomes optional stopping, or a customer-level table becomes an event-level table. Keep your initial claim only if its assumptions still hold; otherwise revise it explicitly.
| Verified practice question | Follow-up to rehearse |
|---|---|
| Analyze Central Limit Theorem in User Comment Distribution | Distinguish the observation distribution from the sampling distribution |
| Compare Mean and Median for a Distribution with a Point Mass | Explain which population feature each summary describes |
| Maximize Covariance from Known Variances | State when a bound is attained, not just its numerical value |
| Approximate a Binomial Upper Tail with a Normal Distribution | Explain the discrete boundary and approximation limitations |
| Assess Routing Experiment Validity | Identify assignment, measurement, and dependence before interpreting lift |
These links are practice resources, not a prediction of your interview. After each attempt, check whether you named the population, the unit of analysis, an assumption, and a conclusion the evidence cannot justify. Return to the Data Scientist statistics collection with whichever gap was hardest to explain aloud.