Interview guides · Definitive Healthcare

Definitive Healthcare Data Scientist Interview Questions & Guide 2026

A two-week prep plan for Definitive Healthcare data scientist candidates: write your answer before reading the solution, revisit misses three days later.

The 60-second brief

As a Data Scientist at Definitive Healthcare, you are at the forefront of transforming the healthcare commercial intelligence industry. Our mission is to help clients—ranging from biopharma and medical device companies to healthcare providers—navigate the incredibly complex healthcare market. In this role, you will synthesize massive, disparate datasets, including medical claims, prescription data, and provider affiliations, to build predictive models and uncover actionable insights.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Risk-adjust outcomes before comparing providers or panels
Correct for claims runout before reporting recent months
Build member-month denominators from overlapping coverage spans

Overview

The impact of this position is profound. You are not just building models in a vacuum; you are directly influencing how life sciences companies launch new therapies, how hospitals optimize their networks, and how the broader healthcare ecosystem operates. As a Senior Data Scientist or Healthcare Analytics Leader, you will spearhead high-visibility projects, shape our analytical product roadmap, and elevate the technical rigor of the entire data organization.

What makes this role uniquely challenging and rewarding is the sheer scale and complexity of the data. Healthcare data is notoriously messy, highly regulated, and deeply nuanced. You will need to balance advanced machine learning techniques with deep domain expertise to solve problems that have no textbook answers. If you are passionate about using data to drive strategic business outcomes in a sector that impacts human lives, this is the role for you.

The Definitive Healthcare interview process

Definitive Healthcare candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Screen

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Take-Home Assignment

Before any modelling, the dataset is itself the first test. Take-home data usually carries something broken: rows duplicated at an unexpected grain, a join that silently drops part of the population, timestamps stored in more than one timezone, or missingness correlated with the outcome. An hour spent profiling row counts, key uniqueness and date ranges is not overhead, because it decides whether every number after it is real. What separates submissions is whether you report the defects you found and adapt the analysis to them, rather than modelling over them quietly and hoping the aggregate absorbs it.

What to demonstrate

Whether you established the grain of each table and checked row counts after every join, and said so in the writeup
Whether data defects you found are surfaced with their effect on the conclusion, instead of being dropped without comment
Whether filters and exclusions are reproducible from the submitted code, with the size of the excluded population quantified

How to prepare

Write a short profiling script you can point at any unfamiliar table: row count, distinct key count, null rate per column, and the min and max of every date field, then run it before anything else
Write the funnel or the join chain as one query and check the row count at each grain, so a silent fan-out shows up as a number rather than as a wrong answer later
On a past dataset, list every exclusion you applied and how many rows each one removed, then draft the single sentence about it you would put in a report

Onsite Interview

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Where candidates lose points

Ignoring clustering by provider, facility, or site

Patients within a provider panel, and subjects within a trial site, are correlated on both case mix and practice pattern. Standard errors computed as if observations were independent are too narrow by roughly the design effect, 1 + (m - 1) * ICC, where m is the average cluster size. With 50 patients per provider and an intracluster correlation of only 0.02, variance is understated by about a factor of two, which manufactures significant provider differences out of noise. Cluster-robust errors or a random intercept per provider, and cluster-level randomisation when you design the test, are the corrections.

Reading the most recent months of a claims-based series as real

Claims incur before they are reported and paid, so recent incurred months are systematically undercounted until runout completes. The lag is not uniform: pharmacy adjudicates in days, professional claims in weeks, inpatient facility claims in months. That means recent data is both too low and mix-shifted toward cheap services, which reads as a cost improvement and a utilisation drop at once. The fix is to hold the last three incurred months back or apply completion factors, and to state the paid-through date on every chart.

Reading an observational correlation as a causal effect

Name the confounder you are most worried about and the design that would remove it: an experiment, a difference-in-differences with a checked pre-period trend, an instrument, or a regression discontinuity. When none is available, state which direction the bias likely runs and bound the claim accordingly.

Ignoring interference between units in a marketplace experiment

Ask whether one unit's treatment can change another unit's outcome through shared inventory, a matching pool, a social graph or a common budget. Where it can, randomise at a level that contains the spillover, such as region or time slice, and say explicitly what that costs you in statistical power.

Questions & practice

Explain the bias-variance tradeoff and how you manage it in your day-to-day modeling work.

Medium · statistics and probability

Approach

Translate the result into the decision it informs, in one plain sentence.
Sanity-check the answer against a simple bound or a simulated case.
Write down the assumption the method needs before you use the method.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

If your model's performance degrades over time in production, how do you diagnose and fix the issue?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Collapse claim versions before summing allowed amounts by member

Easy · claim versioning · groupby · deduplication

You are given medical_claim_line as a pandas DataFrame: claim_line_id, claim_id, claim_version, frequency_code (1 original, 7 replacement, 8 void), member_id, service_start_date, procedure_code, allowed_amount, claim_status. Return total allowed_amount per member for service dates in one stated calendar month, counting each claim once at its surviving version and excluding any claim whose surviving version is a void. Multi-line claims must keep every line of that surviving version. Output columns: member_id, allowed_amount, sorted descending. Do not deduplicate on claim_id alone.

Approach

Separate the two grains out loud before writing code: versions live at the claim level, dollars live at the line level. Every bug in this exercise comes from mixing them.
Compute the surviving version per claim with a groupby transform of max over claim_version, then keep the lines whose claim_version equals that value. A transform keeps the frame at line grain, which a sort plus head(1) does not.
Drop whole claims whose surviving version carries frequency_code 8. A void is not a zero-dollar line, it retracts the claim, so filtering rows rather than claims leaves the prior version's lines behind.
Filter to claim_status 'paid' and state the assumption: denied, pended and reversed lines carry an allowed_amount that never became a liability, so including them overstates cost.
Restrict service_start_date to the month boundaries, then groupby member_id and sum. Sort descending and return.

Worked solution (20 min)

Build a max-version lookup per claim_id with groupby('claim_id')['claim_version'].transform('max') and keep rows where claim_version equals it.
From the surviving rows, collect the claim_ids whose frequency_code is 8 and drop those claim_ids entirely.
Filter claim_status to 'paid' and service_start_date into the month window.
Group by member_id, sum allowed_amount, sort descending, reset the index.

Expected result: A two-column frame member_id, allowed_amount. On a claim with versions 1, 7, 7 only the third version's lines contribute; a claim whose latest version is frequency_code 8 appears nowhere. Total allowed is strictly below a naive sum over all rows whenever any claim in the sample was adjusted, and equal to it when none was.

Verify it yourself

The distinct claim_id count in your filtered frame equals the count of distinct claim_ids whose surviving version is not a void.
The line count you kept for each surviving claim equals that claim's line count at its max version, not 1.
A naive sum over every row is greater than or equal to your total, with equality only when no claim has claim_version above 1.

Follow-up

A claim has versions 1, 7, 7 where the second replacement has fewer lines than the first. What does your code return and is that correct?
How would you handle a claim whose lines span two calendar months at the boundary of the reporting window?
The same logic has to run over 400 million lines in a warehouse. What changes, and what stays the same?

How do you optimize a SQL query that is running too slowly on a table with 500 million rows?

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Given a table of prescription fills, how would you identify patients who are non-adherent to their medication?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Turn overlapping coverage spans into fractional member-month denominators

Medium · intervals · gaps and islands · denominators

member_enrollment holds span_id, member_id, plan_id, product_type, effective_date and termination_date, which is NULL while the span is active. A member can hold several spans in a year, and back-dated plan changes make spans overlap. Return one row per calendar month of 2025 with total member-months, where a member contributes covered days in the month divided by that month's length, and a day covered by two overlapping spans counts once. Both span endpoints are inclusive. Collapse overlaps with window functions before joining a month calendar, and do not assume one span per member.

Approach

Normalise the open end first. COALESCE(termination_date, DATE '9999-12-31') or clip to the reporting end, because a NULL termination_date makes every BETWEEN comparison unknown and drops exactly the members who are still covered.
Merge overlapping spans per member before the calendar join. Order by effective_date, carry a running MAX of prior termination_date with a frame ending one row before, and open a new island where effective_date exceeds that running max plus one day, which keeps adjacent spans contiguous rather than splitting them.
Clip each merged island to the reporting year, then cross join to a twelve-row month calendar and keep pairs that intersect.
Compute covered days as LEAST(island_end, month_end) minus GREATEST(island_start, month_start) plus one, and divide by the actual length of that month rather than a flat 30, so February and the 31-day months are not distorted.
Sum the fractions by month. Merging after the calendar join also works but forces a DISTINCT over member-days, which stops scaling at population size.

Worked solution (30 min)

CTE bounded: clip spans to 2025-01-01 through 2025-12-31 with COALESCE on termination_date, discarding spans that do not intersect the year.
CTE merged: gaps-and-islands per member using LAG and a running MAX over preceding rows to collapse overlapping and adjacent spans.
CTE months: twelve rows of month_start, month_end and days_in_month.
Join merged to months on interval intersection and compute covered_days per pair.
SUM(covered_days / days_in_month::numeric) grouped by month, ordered by month.

Expected result: Twelve rows of (month, member_months) with fractional values. Any single member contributes at most 1.0 to a month and at most 12.0 across the year. A member covered 2025-01-01 through 2025-02-20 contributes 1.0 in January and 20/28 = 0.714 in February.

Verify it yourself

For a member with spans 2025-01-01..2025-03-31 and 2025-03-15..2025-06-30, March contributes exactly 1.0, not 1.5.
Total member-months is at most 12 times COUNT(DISTINCT member_id) and at least the count of members covered the whole year.
Forcing every termination_date to NULL raises the total. If it drops to zero, the NULL handling is missing.

Follow-up

Where does a continuous-enrolment filter with an allowable 45-day gap belong, and how does that change the island boundary rule?
One human appears under two member_ids after a master person index merge. What does that do to member-months, and how would you detect it?
How do you treat a span that terminates on the day it starts, and what does that row usually represent?

We have access to a new, unstructured dataset of clinical trial summaries. How would you extract value from this data?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize your time when balancing urgent ad-hoc requests with long-term strategic modeling projects?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

A pharmaceutical company wants to launch a new drug for a rare autoimmune disease. How would you use our data to help them identify target physicians?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Describe how you would build a network graph to map the relationships between primary care physicians and specialists.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Estimate a programme effect at a fixed risk-score enrolment cutoff

Medium · regression discontinuity · regression to the mean · local average treatment effect

Care management enrols members whose prospective_risk_score in member_enrollment crosses a fixed annual threshold of 1.75; above the line enrolment is near-automatic and below it unavailable, and about 85 percent of members above the line actually enrol. Leadership refused a randomised holdout and wants the programme's effect on next-year allowed PMPM. Two years of scores and claims are available. State the design, the estimand it identifies, the assumptions, and the diagnostics you would run before reporting anything.

Approach

Propose a fuzzy regression discontinuity on prospective_risk_score at 1.75. Uptake jumps but is not complete, so the estimator is the ratio of the jump in outcome to the jump in enrolment probability at the cutoff, which is a Wald ratio.
State the estimand honestly up front: a local effect for members near 1.75, who are the least sick enrollees. It does not identify the effect on the top risk decile, which is usually the number leadership actually wants, and saying so before the result lands is what keeps the analysis credible.
Say explicitly why the obvious alternative fails. A pre-post on enrolled members selects on a high score in year one, and such cohorts regress toward the mean in year two whether or not anything is done, so a pre-post design reports savings every time. The discontinuity is immune because it compares members on either side of an arbitrary line.
Specify the estimation: local linear fits on each side, triangular kernel, a data-driven MSE-optimal bandwidth, and robust bias-corrected inference rather than a naive local-linear confidence interval. Do not fit a high-order global polynomial.
Run the falsification suite before looking at the effect: a density test for manipulation of the running variable, continuity of pre-determined covariates at the cutoff, sensitivity across bandwidths, and placebo cutoffs away from 1.75.

Worked solution (30 min)

Plot the outcome and the enrolment rate against the centred running variable in uniform bins to confirm a visible jump in both.
Estimate the first stage (jump in enrolment probability at 1.75) and the reduced form (jump in next-year PMPM), then take the ratio for the fuzzy RD estimate.
Select the bandwidth by an MSE-optimal rule with a triangular kernel and local linear fits, and report robust bias-corrected intervals.
Run the density test at 1.75 and the covariate continuity tests on age, prior-year cost, dual_eligible_flag, and product_type.
Re-estimate at 0.5x, 1x, and 2x the selected bandwidth, and at placebo cutoffs of 1.50 and 2.00.

Expected result: A fuzzy RD estimating a local average treatment effect on compliers near the 1.75 cutoff, equal to the outcome jump divided by the enrolment jump, with local linear fits, a triangular kernel, an MSE-optimal bandwidth, and robust bias-corrected inference. The result is explicitly not generalisable to the highest-risk members.

Verify it yourself

The density test should show no discontinuity at 1.75; a spike just above the line indicates manipulation and invalidates the design.
Pre-determined covariates should show discontinuities indistinguishable from zero at the cutoff.
The point estimate should be stable across 0.5x, 1x, and 2x bandwidth.
Placebo cutoffs away from 1.75 should return null effects.

Follow-up

Risk scores are built from coded diagnoses, and coding intensity is something people can influence. What would manipulation look like in the density, and what do you do if you see it?
The MSE-optimal bandwidth leaves 900 members. How do you decide whether to report an underpowered estimate at all?
Design an encouragement version instead: randomise the invitation rather than enrolment. What does instrumental variables buy you, and which assumption is most at risk?

First-pass denial rate doubled in one received week

Medium · pipeline failure · segment cuts · denominator definition

First-pass denial rate jumped from 6.1 to 11.8 percent in a single received_date week and stayed at the new level. Nothing in the adjudication rules changed that week. You have medical_claim_line: claim_id, claim_version, frequency_code, claim_status, denial_reason_code, member_id, received_date, adjudicated_at, billing_provider_npi. The metric counts lines whose first adjudication returns 'denied', over lines received in the window at claim_version = 1. Find the cause, say whether denial behaviour actually changed, and deliver the corrected series plus the query change that fixes it.

Approach

Cut the excess by denial_reason_code before anything else. A behaviour change spreads across codes; an upstream failure concentrates in one. If a single eligibility-related code carries nearly all of the 5.7 point excess, the claims are fine and the eligibility data was not.
Join the denied lines to member_enrollment and compare the span's created_at to the claim's received_date. Lines denied for members whose coverage span loaded after the claim arrived are a load-ordering failure, not a coding failure.
Check resolution downstream: take those claim_ids and look for a later version with frequency_code 7 that adjudicates to 'paid'. A high resubmit-and-pay share confirms the denial was transient and recoverable.
Verify the denominator is not the mover. Count lines at claim_version = 1 by received_date and check the weekday mix, because submission batching makes received_date strongly day-of-week patterned and a short or holiday week shifts the denominator without changing behaviour.
Confirm runout: the metric should be held until 60 days of adjudication have passed, so check that the comparison weeks are equally mature rather than comparing a settled week to a fresh one.
Restate the series with the affected reason code broken out as its own line, and change the query to carry denial_reason_code into the reported breakdown rather than only the headline rate.

Follow-up

The affected lines all get paid on resubmission. Is the first-pass denial rate then wrong, or right and uninteresting?
How would you monitor for this class of failure without waiting for someone to notice a chart?
What changes if the excess is spread evenly across denial_reason_code instead of concentrated?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Give an example of how you have mentored a junior team member to improve their technical skills.

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Defend a null result against a programme sponsor

Medium · regression to the mean · difference-in-differences · stakeholder management

A care management programme enrolled the top 1 percent of members by prior-year allowed spend. The sponsor's deck shows allowed PMPM for enrollees falling 34 percent from the year before enrollment to the year after, and asks for budget to triple the programme. You rebuild the evaluation with a concurrent comparison group selected by the identical spend rule in the same period. The difference-in-differences estimate is a 3 percent reduction with a confidence interval spanning zero. Deliverable: how you present this, to whom, in what order, and what you propose next.

Approach

Name the probe: whether you can deliver a finding that costs somebody their programme without softening it into uselessness or creating an adversary who routes around you next time.
Lead with the mechanism, not the verdict. Show the comparison group's own unadjusted drop, which will be large, because a cohort selected on an extreme of the outcome regresses toward the mean whether or not anyone intervenes. The sponsor's 34 percent is mostly that, and it is a property of the selection rule rather than a criticism of their clinicians.
Give the sponsor the finding privately before it appears in any deck their leadership sees. Being surprised in a room is what turns a methods disagreement into a political one.
State the estimate with its interval and say what it rules out as well as what it fails to establish. A 3 percent point estimate whose interval crosses zero is not evidence of no effect; it is insufficient power to separate a modest effect from none, and those are different claims.
Arrive with a design rather than only an objection. Propose a regression discontinuity at the enrollment threshold if the rule is applied sharply, or a randomised rollout across the next wave of eligible members, and state the sample size needed to detect the effect size the sponsor believes in.

Follow-up

The sponsor says withholding the programme from a comparison group is unethical. What do you propose instead?
Leadership wants one number for the board next week. What do you give them?
What result would change your mind and make you believe the 34 percent?

Own an analysis that shipped wrong and was acted on

Easy · retrospective · ownership · data quality

Describe an analysis you delivered that was wrong, where someone acted on it before the error surfaced. Pick something with a mechanism you can name, not a typo. Cover what the number claimed, what was actually true, how much time passed before it was caught, who caught it, and what you told the people who had already acted. Deliverable: a five-minute account that ends with the specific control you added afterwards, plus one occasion since when that control has fired.

Approach

Read the probe: the interviewer is testing whether you detect your own errors and whether the correction was structural. Pick an error with a nameable mechanism (a denominator built from distinct members instead of member-months, a series read before claims runout completed, a cohort selected on the outcome) so the story has a diagnosis rather than a mood.
Open with the decision that was made on your number, not with the bug. The severity of an analytic error is measured in the action it caused: a rate filing, a staffing plan, a programme expansion.
State the counterfactual number. 'The true figure was 14.1 admissions per 1,000 member-years, not 9.8' is a fact; 'it was off by a lot' is not.
Describe the detection path honestly, including whether you found it or somebody else did. Say how long it took, because time to detection is the part you actually control next time.
Close on the control, and make it something that protects a reader who has never met you: a unit test on the denominator, a paid-through date stamped on every chart, a required exclusions block in the spec template. Name one later date it caught something.

Follow-up

What would have had to be true for you to catch this before delivery rather than after?
Did the stakeholder change how much they trust your work, and did you want them to?
Give an example of a control you added that turned out to be the wrong control.