Interview guides · Duke University

Duke University Data Scientist Interview Questions & Guide 2026

Causal inference drills for Duke University data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

As a Data Scientist at Duke University, you serve as a critical analytical engine driving research initiatives, operational efficiencies, and data-informed decision-making across academic and administrative domains. This role sits at the intersection of advanced statistical modeling, data engineering, and strategic insight generation. You will be responsible for transforming complex, disparate datasets into clear, actionable recommendations that influence key stakeholders, university leadership, and research teams.

SQL is seldom the hardest round and is often the one that eliminates people. The working bar is usually window functions, correct deduplication, and joins that do not silently fan out rows, rather than obscure syntax.

Focus areas

Align cohorts to term weeks, not calendar weeks
Model seat activation as the renewal leading indicator
Read item difficulty and discrimination before trusting accuracy

Overview

Your work directly impacts major institutional projects, ranging from optimizing student success metrics and healthcare analytics to supporting large-scale academic research grants and administrative planning. Because Duke University manages vast amounts of sensitive and diverse data, this role requires both technical rigor and a high degree of ethical responsibility. You will collaborate closely with researchers, domain experts, and engineering teams to deploy models, build robust reporting pipelines, and design rigorous validation frameworks.

Expect a stimulating environment where intellectual curiosity is prized, but be prepared for a culture that values meticulous validation over hasty conclusions. Success in this position requires balancing deep technical capability with clear communication, enabling you to translate complex algorithmic outputs into narratives that non-technical leaders can act upon with confidence.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Adaptive item selection holds observed accuracy flat by construction.

A selector targeting a fixed success probability, say 70%, will keep measured first-attempt accuracy near 70% whether learners are improving or not, because as theta rises the engine simply serves harder items. Reporting 'accuracy improved 3 points' on adaptive content therefore usually means the selector got more conservative, not that anyone learned more. Accuracy is only interpretable on a fixed form with unchanged item versions, which is why the first-attempt accuracy metric above carries both restrictions.

Roster sync creates and deactivates accounts in bulk, and it is not behaviour.

An automated roster import at term start can create thousands of accounts in one hour and deactivate thousands more at term end. Signup, activation, and churn series computed without filtering enrollment_source IN ('roster_sync','admin_bulk') will show spikes and cliffs driven entirely by an integration job. It also breaks cohort retention: a roster-created account that never activates is a provisioning artefact, not a churned learner.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Never asking what decision the analysis will inform

Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.

Questions & practice

Implement verified mastery events with a delayed check

Medium · first crossing · window logic · metric implementation

From fct_assessment_response (learner_id, objective_id, content_item_id, submitted_at, attempt_no, is_correct, score_points, max_points, scoring_mode, scored_at, mastery_state_after), count verified mastery events for one calendar month. A mastery event is the first time a (learner_id, objective_id) pair reaches mastery_state_after 'mastered'. It is verified when the earliest scored response on that objective submitted at least 7 days later scores score_points/max_points >= 0.8. Return the monthly count and the verified rate, with the denominator restricted to events that have an eligible later check.

Approach

Filter to rows with mastery_state_after = 'mastered', then take the earliest submitted_at per (learner_id, objective_id). This is not the same as the first row overall for the pair, since earlier attempts sit in 'learning'.
Restrict those crossings to the target month in UTC. A pair that first crossed in an earlier month never enters this month's count even if it crosses again after a lapse, so the first-crossing filter must run over all history, not just the month.
Build the candidate check set by joining the full response table back on (learner_id, objective_id) and keeping rows with submitted_at >= crossing + 7 days, score_points not null, scored_at not null, and max_points > 0.
Take the earliest eligible candidate per pair, not the best one. Taking the maximum score turns a retention check into an ever-passed-later check, which is a much weaker claim and a materially higher number.
Mark verified where score_points / max_points >= 0.8. The denominator is crossings with at least one eligible candidate; report the count of crossings with none alongside, because that share moves with the reporting lag and a large share means the month is not ready to publish.

Worked solution (30 min)

mastered = df[df.mastery_state_after == 'mastered']; crossings = mastered.groupby(['learner_id','objective_id'], as_index=False).submitted_at.min().
month_crossings = crossings[crossings.submitted_at.between(month_start, month_end)].
Merge the full response frame onto month_crossings on (learner_id, objective_id), keep rows where submitted_at >= crossing + 7 days and score_points.notna() and scored_at.notna() and max_points > 0.
Sort candidates by submitted_at and take the first per pair with drop_duplicates on the pair key.
verified = (score_points / max_points) >= 0.8; report verified.sum(), len(candidates), and len(month_crossings) - len(candidates).

Expected result: Three numbers for the month: the count of first crossings with an eligible delayed check (the denominator), the share of those meeting the 0.8 ratio (the verified rate), and the count of crossings with no eligible check. Pairs that first crossed in an earlier month contribute nothing.

Verify it yourself

A pair whose only response lands in 'mastered' with nothing after it appears solely in the no-eligible-check count, in neither numerator nor denominator.
Moving a delayed check from day 6 to day 8 after the crossing flips the pair from ineligible to eligible and changes the denominator by exactly one.
Shifting the month window by one day moves crossings in and out but leaves each crossing's chosen check unchanged.

Follow-up

A pair crosses to 'mastered', lapses, and crosses again inside the same month. What does your code count, and what should it count?
Why does this metric need a 30-day reporting lag, and what shape does the series take if you publish it at 7 days?
The 0.8 ratio is a product choice. How would you check it actually separates learners rather than just passing almost everyone?

Compute pace adherence at term-week six

Easy · metric implementation · calendar alignment · null handling

Given fct_enrollment (enrollment_id, learner_id, course_id, section_id, term_id, enrollment_source, enrolled_at, scheduled_start_date, due_date, completed_at, units_total, units_completed, status) and dim_term (term_id, term_start_date, term_end_date, term_length_weeks), compute the share of enrollments at or ahead of pace at the end of term-week 6. An enrollment is on pace when units_completed >= units_total * (6 / term_length_weeks). Count only enrollments active at that moment. Handle NULL scheduled_start_date and units_total of zero explicitly, and return one value per term with its denominator.

Approach

Join to dim_term on term_id and derive the cutoff as term_start_date plus 42 days. The metric is anchored to the term calendar, not to each enrollment's own start, which is what makes it comparable across institutions on different calendars.
Resolve NULL scheduled_start_date by falling back to term_start_date, and count how often the fallback fired. A large fallback share is a finding about roster sync, not a detail to bury.
Reconstruct active-at-cutoff rather than trusting the status column: enrolled_at <= cutoff and (completed_at is null or completed_at > cutoff). If status is current-state only, say so, because then filtering on it removes learners who fell behind and dropped later.
Exclude units_total = 0 from both numerator and denominator and report that count separately. Left in, 0 >= 0 marks every empty enrollment as on pace and drags the rate up.
Group by term_id and return numerator, denominator, and rate together, never the rate alone.

Follow-up

A two-week holiday falls inside weeks one to six for one term. How do you stop that punishing those enrollments without hand-editing the threshold?
Two terms have term_length_weeks of 10 and 16. Is a single week-6 number comparable across them, and what would you report instead?

Attach the calibration in force without merge_asof

Medium · as-of join · searchsorted · vectorisation · slowly changing dimensions

You have responses (response_id, content_item_id, submitted_at, is_correct) and calibration_history (content_item_id, valid_from, valid_to, irt_a, irt_b), where intervals within an item are non-overlapping and the open interval carries valid_to as NaT. Attach to each response the irt_a and irt_b in force at submitted_at. You may not use pandas.merge_asof and you may not apply row-wise. responses has about 5 million rows, calibration_history about 40 thousand. Return the input frame plus two columns, NaN where no interval covers the timestamp.

Approach

Sort calibration_history by (content_item_id, valid_from) and factorise content_item_id across both frames into a shared integer code, so an unknown item on the response side is detectable immediately rather than joining to nothing.
Convert both timestamps to int64 seconds and build one composite key per side, code * 2**32 + seconds. With codes well under two billion and seconds near 1.8e9, that stays inside int64 and makes a single global search possible. State the second-level resolution assumption; interval boundaries are set at second granularity, so nothing is lost.
Call np.searchsorted(calibration_keys, response_keys, side='right') - 1 once over the whole array. That gives, for each response, the position of the latest interval starting at or before it within the same item, because the composite key orders by item first.
Invalidate bad candidates in two places: index -1, and a candidate whose code does not match the response's code, which happens when a response precedes its item's first interval and lands on the previous item's last row.
Invalidate a third case that is easy to miss: the candidate has a non-null valid_to and submitted_at is at or after it, meaning the response falls in a coverage gap. Without this the previous interval's parameters leak onto uncovered responses.
Take the parameters positionally with np.take and write NaN where any invalidation fired.

Follow-up

A backfill wrote overlapping intervals for 200 items. How do you detect that before the join rather than after, and what do you do with those responses?
The join is correct but peak memory is unacceptable. What changes, and which part of the approach survives?
How would you test this without a golden output to compare against?

Write a query using SQL window functions to calculate running totals and moving averages for research grant expenditures over time.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Daily scored submissions under rubric scoring lag

Easy · date boundaries · null semantics · reporting lag

fct_assessment_response(response_id, learner_id, content_item_id, content_version_no, submitted_at, is_correct, score_points, max_points, scoring_mode, scored_at) stores both timestamps in UTC. scored_at is NULL until a rubric item is graded and lags submitted_at by days when scoring_mode = 'rubric_human'. Produce two dense daily series over the last 90 days: submissions keyed on submission date, and completed scorings keyed on scored date. Then state the reporting lag after which each series stops moving, computed from the data rather than asserted, and say what biases your lag estimate.

Approach

Key the two series on different columns on purpose: date(submitted_at) for volume, date(scored_at) for graded throughput, with scored_at IS NOT NULL on the second. They answer different questions and are not two versions of one number.
Define scored as scored_at IS NOT NULL rather than is_correct IS NOT NULL. Check first whether rubric rows carry score_points with is_correct left NULL after grading; if they do, the is_correct test undercounts by exactly the rubric share.
Estimate lag empirically with percentiles of scored_at - submitted_at grouped by scoring_mode. This is computed only over rows already scored, so it is survivorship-biased downward. Report alongside it the share of submissions in each age bucket still unscored.
Generate a dense date spine and LEFT JOIN both series onto it so a zero-volume day is a 0, not an absent row that a chart will interpolate across.
Name the timezone problem instead of guessing: these tables carry a locale on the learner but no timezone, so a local-calendar day series is not computable from this schema. Say that rather than pretending date(submitted_at) is an institution's day.

Worked solution (15 min)

Build the date spine for the 90-day window, then the two aggregates, then join.
Run a distribution of is_correct for rows with scored_at IS NOT NULL to decide which column defines scored.
Compute median and p95 of scored_at - submitted_at per scoring_mode, and the unscored share by 1, 3, 7 and 14 day age buckets.
Write one sentence stating the reporting lag you are imposing and what it costs.

Expected result: Two gap-free 90-row daily series, submitted_count and scored_count, plus a lag table per scoring_mode with median lag, p95 lag, and the still-unscored share by age bucket.

Verify it yourself

SUM(submitted_count) over the window equals COUNT(*) of responses whose submitted_at falls in the window.
For a window ending today, SUM(scored_count) is at or below SUM(submitted_count), and the gap is concentrated in the final days rather than spread evenly.
Rows with scoring_mode = 'auto' should show a lag near zero. A large auto lag means scored_at is written by a batch job, and the whole lag model needs rebuilding around that job's schedule.

Follow-up

The last five days of the submitted series will be revised upward as late rows land. How do you present a series whose recent points are known to be incomplete?
Rubric grading is done by people with weekly schedules. What does that do to a daily scored series, and would an ISO week grain be more honest?
If you had to pick a single cut-off date after which the data is considered final, how would you derive it and what percentage of eventual scorings would you be discarding?

How would you evaluate the success of a newly implemented campus-wide communication campaign?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

If we notice a sudden drop in user retention for a campus portal, how would you investigate and isolate the root cause?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you design a metric to measure student engagement across our online learning management platforms?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Define a set of key performance indicators for a new digital health initiative launched by the university medical center.

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Metrics that survive an adaptive item selector

Medium · adaptive selection · irt · measurement design

Content selection targets a 70% success probability per learner. First-attempt accuracy reads 69.8% before a model change and 70.3% after. Using dim_content_item (irt_a, irt_b, calibration_n, version_no, status) and fct_assessment_response (attempt_no, is_correct, objective_id), propose a measurement design that can detect a real learning change under this selector. Specify the fixed-form probe set, its size and calibration requirement, the statistic you report, and say why the adaptive accuracy series carries no information about learning.

Approach

Show why the series is pinned. Under a 2PL item, P(correct) = 1 / (1 + exp(-a(theta - b))). A selector targeting 0.70 solves for b given theta: a(theta - b) = ln(0.7/0.3) = 0.847, so b = theta - 0.847/a. As theta rises the selector raises b with it and measured accuracy stays at the target.
Demote accuracy accordingly: it measures how well the selector hit its own target, so keep it as a selector-calibration guardrail and never as an outcome.
Build a fixed-form probe: 10 to 15 items with frozen (content_item_id, version_no), status='published', calibration_n at least 200, spanning irt_b roughly -1.5 to +1.5, served on a fixed schedule identically to every arm and never chosen adaptively.
Report a theta estimate or mean probe score. For 2PL, test information is I(theta) = sum of a_i^2 P_i Q_i, so 12 items with a = 1.0 measured near their peak give I = 3 and SE = 1/sqrt(3) = 0.58 logits per learner. Individual estimates are noisy, but a 400-learner group mean has SE near 0.029 logits, which is where the comparison lives.
Protect the probe: exclude its items from the adaptive pool so practice cannot leak into the measurement, and freeze versions, because editing an item changes the meaning of every historical response to it.
Add the difficulty audit as a second guardrail: mean irt_b of the items backing mastery decisions, which exposes a selector that simply got easier.

Worked solution (30 min)

Derive the selector identity b = theta - ln(0.7/0.3)/a = theta - 0.847/a, and evaluate it at a = 1.2, where the served item sits 0.71 logits below the learner's theta.
Specify the probe: 12 frozen item versions, calibration_n >= 200 each, irt_b spread across -1.5 to +1.5, administered every four weeks to all arms.
Compute probe precision: I = sum a_i^2 P_i Q_i = 12 x 1.0 x 0.25 = 3, SE = 0.58 logits per learner, SE of a 400-learner mean = 0.58/20 = 0.029 logits.
Write the reporting rule: probe mean as the outcome, adaptive accuracy and mean irt_b of mastery-backing items as guardrails.

Expected result: A 10 to 15 item fixed-form probe on frozen versions with calibration_n >= 200, excluded from the adaptive pool, reporting a group mean theta or probe score with SE derived from test information; adaptive first-attempt accuracy demoted to a selector-calibration guardrail alongside mean irt_b, with 69.8% versus 70.3% declared uninformative about learning.

Verify it yourself

b = theta - ln(0.7/0.3)/a = theta - 0.847/a, and at a = 1.2 the offset is 0.71 logits.
Probe items are frozen at one version_no and removed from adaptive selection.
Per-learner SE from 12 items at a = 1.0 is about 0.58 logits, and you did not present individual theta as precise.

Follow-up

How many probe items would you need before reporting an individual learner's theta rather than a group mean?
What happens to the probe if the selector begins serving items that share a template with probe items?
The probe costs learner time every cycle. How do you justify that cost to a curriculum owner?

Net revenue retention slid nine points in one quarter

Hard · revenue-retention · cohort-definition · mix-shift

Net revenue retention on institutional licenses fell from 104 percent to 95 percent quarter over quarter. Source: fct_subscription_period (subscription_period_id, account_id, org_id, plan_tier, billing_interval, seats_purchased, seats_provisioned, mrr_usd, period_start, period_end, is_renewal, status, canceled_at, churn_reason). Decompose the nine points into expansion, contraction and non-renewal, and establish whether renewal behaviour changed at all. Deliverable: the decomposition, the cohort definition you used, and a call on whether this is a trend or account news.

Approach

Pin the cohort in writing before computing anything: orgs holding an active period twelve months before each renewal date, new logos excluded from both numerator and denominator, and non-renewals entering the numerator as zero rather than being dropped. Dropping non-renewals is the most common way this metric is overstated, and correcting it can move the headline by more than the effect under investigation.
Rebuild the ratio from components on that cohort: starting annualised revenue, expansion, contraction, non-renewal. Check the components sum back to the reported ratio in both quarters; a residual means the two quarters were not computed on the same cohort rule and the comparison is void before any interpretation.
Test cohort comparability on billing_interval. Multi-year contracts do not present a renewal in every window, so a quarter whose renewal cohort holds a different multi_year share is not comparable to its predecessor. Recompute both quarters holding the interval mix fixed and report that alongside the raw number.
Rank per-org contributions to the nine points and state how much the top three carry. Institutional revenue is concentrated, so a decomposition that leaves the concentration unstated invites a trend reading of what is account news.
Audit the inputs: confirm mrr_usd normalisation is consistent (annual divided by twelve) and look for overlapping period_start and period_end within a single account_id, which double counts revenue after a mid-term upgrade.
Close the loop to something actionable by joining the lost orgs back to seat activation and pace adherence in the preceding term, and say whether the signal was visible early enough for anyone to have acted on it.

Follow-up

Build the renewal-risk view from this: which signals are actionable while there is still time to act, and at what days-to-renewal would you surface an org?
An org contracts from 900 seats to 600 but moves to a higher tier and raises MRR. Expansion or contraction, and does your decomposition handle it without double counting?
How would you present NRR so that a quarter with an unusual renewal cohort is not read as a trend by an executive audience?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Explain a risk score to a non-technical sales executive

Easy · communicating uncertainty · renewal risk · stakeholder communication

Your renewal-risk score joins fct_subscription_period to seat activation from dim_learner.first_activity_at and to pace adherence from fct_enrollment. For one institution with 900 seats_provisioned and 31% activation at term-week 6, it outputs a 0.62 probability of non-renewal. A sales VP asks whether you are losing the account or not. You have two minutes and no slides. Deliverable: the spoken answer, plus the one line you would put in the weekly account review so the number is not later quoted as a certainty. Probed: whether you can carry uncertainty and still give a decision.

Approach

Answer the question that was asked before qualifying it. 'Treat this as at-risk and act this week' is the answer; the probability is the support for it.
Translate the score into a frequency on a reference class the VP already trusts, such as how many of the last fifty accounts scored in the 0.55-0.70 band did not renew, instead of explaining calibration in the abstract.
Name the driver that is still movable before period_end: 31% seat activation at term-week 6 can be changed this term, whereas last term's usage cannot, so that is where the conversation goes.
State what would move the score and by when, so the number arrives attached to an action rather than as a verdict.
For the written line, use a band and a review date rather than a bare decimal, because a single number in a document will be re-quoted without its interval.

Follow-up

The VP asks for every account scored above 0.50. What do you tell them about where that threshold came from?
How would you check whether 0.62 is calibrated at all, and on what sample?

Account for an analysis you later discovered was wrong

Hard · postmortem · analytical rigor · ownership

Describe a case where you reported a result and later found it was wrong. Choose one where the error was yours and the number had already been used for a decision. Cover: the claim, the actual defect at the data or design level, how you found it, how long it had been live, who you told and in what order, and what you changed so the same class of defect cannot recur. The interviewer will push on the mechanism, not the apology. Probed: whether you understand your own failure modes well enough to have engineered around them.

Approach

Pick an error with a nameable mechanism, such as a grain mistake, a survivorship filter, or a join that silently dropped age_gated learners. 'I misread a chart' has nothing testable in it.
State the blast radius in decisions rather than in dashboards: what was decided, what was spent or promised, and to whom.
Describe detection honestly, including whether someone else caught it and how long it ran undetected. A slow external catch is a usable answer; a vague one is not.
Give the disclosure order and the reason for it: the decision owner first, with the corrected number and its direction in the opening sentence, then the wider audience.
Close on the systemic fix and say whether it has fired since. A control that has never triggered on anything is a resolution, not a fix, and saying so is worth more than claiming otherwise.

Follow-up

What did the corrected number change about the decision? If nothing changed, why did you escalate at all?
Has the check you added caught anything since, and on what?
What class of error would still get past you today?

Defend a null result against an already-announced launch

Medium · null results · experiment design · stakeholder communication

A section-randomised test of a new practice-sequencing feature ran across 62 class sections in 11 orgs. Primary metric: verified mastery events per active learner. With section-clustered standard errors the effect is +1.8% (95% CI -4.1% to +7.9%). Leadership has already told two district administrators that the feature improves outcomes. You have ten minutes in a review. Deliverable: what you say, what you put on one slide, and what you propose next. Probed: whether you can hold a null under commercial pressure without either caving or lecturing the room on statistics.

Approach

Lead with the decision, not the p-value: state the interval in the unit the audience already uses (mastery events per active learner per week) and say plainly what the study could and could not have detected.
Separate the interval's half-width from the minimum detectable effect before you put either on a slide. The stated 95% CI of -4.1 to +7.9 has a half-width of 6.0 points, so the clustered standard error is 6.0 / 1.96 = 3.06 points; the MDE at 80% power and two-sided alpha = 0.05 is (1.96 + 0.8416) * 3.06 = about 8.6 points, roughly 1.43 times the half-width. Quote 8.6, and show that it follows from 62 sections at the section size and intraclass correlation you assumed via the design effect 1 + (m - 1) * rho, so that 'no effect found' is visibly separated from 'too small to see'.
Check the pre-registered guardrails before conceding anything: median active_seconds per mastery event and the calibrated irt_b distribution of the items backing mastery decisions. If both are flat, the null is about learning, not about a broken pipeline.
Handle the promise made to the two districts as its own item with a named owner, rather than leaving the correction implicit in a statistics discussion.
Close with the cheapest decision-grade next step: extend to enough additional already-instrumented sections to reach the effect size leadership actually cares about, with the stopping rule fixed before the extension starts.

Follow-up

If the interval had been -0.5% to +8.0%, would you support launch, and what exactly changes in your reasoning?
Leadership asks you to re-run with learner-level standard errors since it is the same data. What do you say?
What would you have set up at design time so that this meeting was easier?