cswork / libraryOpen practice workspace

Interview guides · athenahealth

athenahealth Data Scientist Interview Questions & Guide 2026

athenahealth data scientist prep: probability, sampling distributions, and stating the assumption behind every test you run.

The 60-second brief

The role of a Data Scientist at athenahealth is pivotal in leveraging data to drive decision-making and enhance healthcare solutions. As a Data Scientist, you will be responsible for analyzing complex data sets to uncover insights that can improve patient care, streamline operations, and support the development of innovative products. This position is critical as it directly influences the effectiveness of the services provided to healthcare providers and patients alike, placing you at the intersection of technology and healthcare.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

  • Risk-adjust outcomes before comparing providers or panels
  • Separate censoring from events in time-to-event models
  • Correct for claims runout before reporting recent months

Overview

Working at athenahealth, you will engage with vast amounts of healthcare data, applying advanced statistical techniques and machine learning algorithms to solve real-world problems. You will collaborate closely with product teams, engineers, and other stakeholders, making your input essential for the development of data-driven strategies that can significantly impact the business. The opportunity to work on meaningful projects that contribute to better health outcomes makes this role both challenging and rewarding.

The athenahealth interview process

athenahealth candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Call

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

  • Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
  • Whether your reason for wanting the role points at the work itself rather than the company's reputation
  • Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

  • Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
  • Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
  • Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Interviews

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

  • Whether your row counts survive each join, and whether you notice on your own when they do not
  • Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
  • Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
  • Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

  • Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
  • Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
  • Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Interviews

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

  • Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
  • Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
  • Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

  • For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
  • Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
  • Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Final Discussions

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

  • Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
  • Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
  • Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

  • Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
  • For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
  • Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Ignoring clustering by provider, facility, or site

Patients within a provider panel, and subjects within a trial site, are correlated on both case mix and practice pattern. Standard errors computed as if observations were independent are too narrow by roughly the design effect, 1 + (m - 1) * ICC, where m is the average cluster size. With 50 patients per provider and an intracluster correlation of only 0.02, variance is understated by about a factor of two, which manufactures significant provider differences out of noise. Cluster-robust errors or a random intercept per provider, and cluster-level randomisation when you design the test, are the corrections.

Reading the most recent months of a claims-based series as real

Claims incur before they are reported and paid, so recent incurred months are systematically undercounted until runout completes. The lag is not uniform: pharmacy adjudicates in days, professional claims in weeks, inpatient facility claims in months. That means recent data is both too low and mix-shifted toward cheap services, which reads as a cost improvement and a utilisation drop at once. The fix is to hold the last three incurred months back or apply completion factors, and to state the paid-through date on every chart.

Optimising accuracy on a heavily imbalanced target

State the base rate first, then choose the metric from the relative cost of a false positive against a false negative: precision and recall at the operating threshold, PR-AUC, or expected cost. At a 1 percent positive rate, predicting the majority class for everyone scores 99 percent accuracy and is worthless.

Never asking what decision the analysis will inform

Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.

Questions & practice

Write a function to calculate the mean and variance of a dataset.

Medium · statistics and probability

Approach

  1. Translate the result into the decision it informs, in one plain sentence.
  2. Sanity-check the answer against a simple bound or a simulated case.
  3. Say what the estimate is of, and over what population it generalises.

Follow-up

  • How would you explain this result to someone who does not know statistics?
  • What sample size would you need to detect an effect half this size?

Describe the difference between overfitting and underfitting.

Medium · machine learning and modelling

Approach

  1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
  2. Check what information would not exist at prediction time, and exclude it.
  3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

  • What would you monitor after launch to know the model is still valid?
  • Where could label leakage enter this setup?

Can you implement a simple decision tree algorithm in Python?

Medium · machine learning and modelling

Approach

  1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
  2. Say how the offline result would be validated online before it is trusted.
  3. Check what information would not exist at prediction time, and exclude it.

Follow-up

  • What would you monitor after launch to know the model is still valid?
  • How would you choose the decision threshold, and who owns that choice?

Expand overlapping coverage spans into fractional member-months

Medium · interval expansion · denominators · vectorisation

member_enrollment has span_id, member_id, plan_id, product_type, effective_date, termination_date (NULL while active). Both dates are inclusive. Write a function returning one row per calendar month of a stated year with total member-months, where a member contributes covered days in the month divided by days in that month, capped at one month per member even when two spans overlap after a plan change or a retroactive span. Do not expand to one row per member-day: the sample has 400,000 members. Output: month, member_months.

Approach

  1. Resolve NULL termination_date to the reporting end date and say so in a comment. An open span is not an infinite span, and clipping it at the year end is what keeps the denominator finite and auditable.
  2. Merge overlapping and adjacent spans per member before touching months. Sort by member_id and effective_date, carry a running maximum of the end date, and open a new merged group when effective_date exceeds that running max plus one day. Adjacent means a gap of zero days, which a plan change produces constantly.
  3. Cross join the merged spans to the 12 month boundaries rather than to days. Overlap days = (min(span_end, month_end) - max(span_start, month_start)).days + 1, clipped below at 0. That is at most 12 rows per span instead of 365.
  4. Divide overlap days by the number of days in that month, so February and July are weighted correctly, then sum by month.
  5. Validate on a constructed member before trusting the aggregate: a single span covering the whole year must sum to exactly 12.0.

Worked solution (30 min)

  1. Fill termination_date nulls with the reporting end date and clip all spans to the reporting year.
  2. Sort by member_id, effective_date, then compute a running max end per member with cummax shifted by one, flag a new group where effective_date > prior_max_end + 1 day, and cumsum the flag to get merged group ids.
  3. Aggregate each group to min start and max end, producing disjoint spans per member.
  4. Cross join merged spans to a 12-row month frame, compute clipped overlap days, divide by days in month.
  5. Group by month and sum.

Expected result: Twelve rows. For a member with spans 2026-01-01 to 2026-03-15 and 2026-03-01 to 2026-06-30, March contributes exactly 1.0, not 15/31 plus 31/31. For a member with spans covering January and March only, January is 1.0, February is 0.0, March is 1.0, and the member totals 2.0.

Verify it yourself

  • A member with one span covering the full year sums to exactly 12.0 across the twelve rows.
  • Total member-months is less than or equal to (distinct members) x 12, and equality holds only if every member is covered all year.
  • Re-running the span merge on its own output changes nothing, which proves the merged spans are disjoint.
  • Recompute one member by brute-force day expansion and confirm the two agree to floating point tolerance.

Follow-up

  • Product type changes mid-year. The metric must be reported by product_type. Where does the cap now apply and what breaks?
  • An eligibility file arrives with a retroactive termination that shortens a span you already reported on. How do you restate?
  • Why member-months rather than distinct members, in one sentence, for a director who wants the simpler number?

Solve a problem related to data manipulation, like finding duplicates in a dataset.

Medium · sql and data manipulation

Approach

  1. Say which table is the grain you start from, and join outward from it.
  2. Compute rates by summing numerator and denominator separately, never by averaging rates.
  3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

  • How does the query change if the join becomes one-to-many?
  • How would you verify this result without re-running the same query?

Measure ninety-day primary care contact for an enrollment cohort

Easy · cohort analysis · conversion rate · date windows

member_enrollment gives span_id, member_id, product_type, effective_date and termination_date, which is NULL while a span is active. medical_claim_line gives member_id, service_start_date, procedure_code and claim_status. The cohort is members whose earliest 2025 span has effective_date in January 2025 and who stay covered at least 90 days from it. Return, by product_type, cohort size and the share with at least one paid claim line whose procedure_code is in a supplied list of evaluation-and-management office visit codes and whose service_start_date falls in the half-open window from effective_date to effective_date plus 90 days.

Approach

  1. Build the cohort as one row per member before touching claims. Take the earliest 2025 span per member, require its effective_date in January, and require coverage to effective_date + 90 days, which means termination_date IS NULL OR termination_date >= effective_date + 89.
  2. Use EXISTS for the numerator rather than a join. A member with four office visits must count once; joining and counting rows inflates the numerator and can push the rate above 1.
  3. Keep the window half-open and anchored per member. Every member has a different effective_date, so a single fixed calendar window is a different metric and will not match the cohort definition.
  4. Aggregate numerator and denominator in one pass with a conditional sum grouped by product_type, and carry the raw counts alongside the rate so the reader can see the denominator.
  5. State the reportability rule on the output: the January cohort is not final until 90 days plus professional-claims runout have passed, so label the paid-through date.

Worked solution (15 min)

  1. CTE first_span: per member, the 2025 span with the minimum effective_date, filtered to January effective dates.
  2. Add the continuous-coverage requirement to that CTE using COALESCE(termination_date, DATE '9999-12-31').
  3. Add an EXISTS subquery on medical_claim_line correlated on member_id, restricted to the code list, claim_status = 'paid', and service_start_date >= effective_date AND < effective_date + 90.
  4. Group by product_type: COUNT(*) as cohort_size, SUM(CASE WHEN contacted THEN 1 ELSE 0 END) as contacted, and their quotient.
  5. Print the paid-through date used, and re-run at a 30-day window to confirm the rate falls.

Expected result: One row per product_type with cohort_size, contacted_members and contact_rate = contacted_members / cohort_size, where contacted_members never exceeds cohort_size.

Verify it yourself

  • contact_rate lies in [0, 1] for every product_type. Above 1 proves the numerator was counted on rows rather than members.
  • SUM(cohort_size) across product_types equals COUNT(DISTINCT member_id) in the cohort CTE. A larger value means a member with two January spans was counted twice.
  • A 0-day window returns a rate at or near zero, and a 365-day window returns a rate no lower than the 90-day one.

Follow-up

  • A member changes plan on day 40, closing one span and opening another. Is she still in the cohort, and which product_type does she report under?
  • What moves if denied lines are allowed into the numerator, and which definition is right for an access metric?
  • How would you show this as a cumulative curve by day since effective_date instead of one 90-day point?

Given a dataset, how would you approach cleaning and preparing it for analysis?

Medium · product sense and metrics

Approach

  1. Restate the decision this analysis has to support, and who acts on the answer.
  2. Fix the population and the time window before naming any metric.
  3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

  • Which segment would you cut first, and what would that rule out?
  • What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize tasks when working on multiple projects?

Medium · product sense and metrics

Approach

  1. State what result would change your recommendation, so the answer is falsifiable.
  2. Restate the decision this analysis has to support, and who acts on the answer.
  3. Fix the population and the time window before naming any metric.

Follow-up

  • How would you detect that the metric is being gamed rather than genuinely improving?
  • Which segment would you cut first, and what would that rule out?

Explain the concept of tf-idf and its use in text analysis.

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Work from the decision backwards to the evidence you would need.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Critique a denial rate slide before it is shown

Easy · metric critique · claim versions · runout · mix shift

A weekly slide says first-pass denial rate improved from 9.1 percent to 7.4 percent month over month. It was computed from medical_claim_line as rows with claim_status 'denied' divided by all rows with a received_date in the month. You cannot query anything. List every assumption the number rests on, restate the metric so two analysts computing it independently would get the same value, and give the three questions you would ask before the slide is presented.

Approach

  1. Work the grain first. The denominator counts all rows, so a claim resubmitted after a denial contributes its original line, its replacement (frequency_code 7) and possibly a void (8). Both numerator and denominator are inflated by exactly the claims most likely to have been denied, which makes the rate a function of resubmission behaviour rather than adjudication.
  2. Work maturity second. A line received late in the month may still be pended at extract time, and pended lines are neither paid nor denied. If pends resolve to denials later, the most recent month is understated by construction, which looks exactly like improvement. Ask for the paid-through date and the adjudication runout hold.
  3. Work the unit third. A denial rate on claim lines and a denial rate on claims are different numbers, and a single denied line on a twenty-line facility claim is a different operational event from a wholly denied professional claim. The spec has to pick one and say so.
  4. Work mix fourth. Denial reasons have different owners and different volumes: eligibility denials belong to enrollment operations, coding denials to provider relations. A shift in submission mix across providers, place of service or reason code moves the blended rate with no change in any component, so the restated metric reports by denial_reason_code group.
  5. Work precision last. A 1.7 point move is decisive on two million lines and meaningless on three thousand. The restated metric carries the denominator count, and a slide without it cannot be read.

Worked solution (15 min)

  1. Write the assumption list as a numbered set: single version per claim, adjudication complete at extract, line grain intended, stable reason-code and provider mix, denominator large enough for the move to be real.
  2. Restate the metric with numerator as lines denied at first adjudication, denominator as lines received in the month counted at claim_version 1, reported by received_date month, held 60 days, broken out by denial_reason_code group, and published with the denominator count.
  3. Write the three questions in the order that would kill the slide fastest: what is the paid-through date, how many lines are in each month's denominator, and did the reason-code mix move.
  4. State how the restated number is presented, with the most recent month marked provisional and restated at each refresh.

Expected result: A restated specification reading: numerator is claim lines whose first adjudication returns claim_status denied; denominator is lines with received_date in month M counted at claim_version 1; the month is held until 60 days of adjudication runout have passed; the rate is broken out by denial_reason_code group; and both the denominator count and the paid-through date appear on the slide, with the most recent month marked provisional. The three questions are the paid-through date, the denominator size, and whether reason-code mix shifted.

Verify it yourself

  • The restated denominator must be smaller than the original, since version-1 filtering can only remove rows. If it is not, the version filter is not doing anything.
  • Both months should be recomputable at the same paid-through age; comparing a mature month to a two-week-old month is the defect the hold is meant to prevent.
  • The reason-code breakout should sum back to the blended rate, which is the check that the mix decomposition is complete.

Follow-up

  • The rate is still down after version dedup and a 60-day hold. What is the next thing you check before anyone celebrates?
  • Which of these problems would also affect a paid-amount series computed the same way?
  • How would you present the restated number so the audience understands the prior month is provisional?

Pooled persistency fell but no one lost coverage

Medium · interval sql · mix shift · cohort seasonality

Pooled 12-month enrollment persistency fell from 0.78 to 0.66 for the cohort anchored in a single month. member_enrollment carries span_id, member_id, plan_id, product_type, effective_date, termination_date, termination_reason_code, dual_eligible_flag and region_code. The metric allows at most one gap of 45 days or fewer. Product mix in that anchor month was not stable. Determine how much of the fall is real disenrollment rather than span bookkeeping or mix, and deliver a persistency series a contract owner could act on.

Approach

  1. Inspect the query's unit of analysis first. Spans are not members, and a benefit or plan change closes one span and opens the next on the same day, so any logic that treats a non-null termination_date as loss of coverage counts an administrative event as churn.
  2. Rebuild member-level coverage by unioning all spans per member and merging adjacent or overlapping ones, then apply the 45-day gap rule to the merged intervals rather than per span.
  3. Cut the terminations by termination_reason_code. plan_change and eligibility_redetermination have entirely different meanings for a contract, and death is not churn you can address.
  4. Split by product_type and stop pooling. Commercial open enrollment and managed medicaid redetermination produce structurally different churn on different calendars, and the pooled average of the two is a number that describes nobody.
  5. Compare the anchor month to the same month in the prior year rather than to the adjacent month, because enrollment is strongly seasonal and an anchor month sitting on an open-enrollment or redetermination boundary is not comparable to its neighbour.
  6. Apply the Kitagawa decomposition across product_type to separate how much of the 12 point fall is mix, meaning the anchor cohort simply contained more of a high-churn product, from how much is within-product deterioration.

Follow-up

  • Why 45 days, and what would change if the allowable gap were 30 or 60?
  • A member terminates and re-enrolls 90 days later under a different plan_id. Are they retained? Defend your answer against the opposite one.
  • What guardrail would catch someone improving persistency by declining to enroll members likely to churn?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

  • Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
  • Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
  • State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

  • Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
  • Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
  • Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

  • Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
  • Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
  • Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

  • Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
  • Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
  • Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

  • Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
  • Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
  • List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

  • Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
  • Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
  • Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

  • Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
  • Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
  • Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Tell me about a time you faced a disagreement with a team member and how you resolved it.

Medium · behavioural and stakeholder questions

Approach

  1. Close with what you would do differently, concretely.
  2. Pick a story where you drove the decision, not one where you observed it.
  3. State the situation in two sentences and spend the rest on your reasoning.

Follow-up

  • How did you know the outcome was caused by your change?
  • What would you do differently if you ran that project again?

Discuss a time when you had to adapt to a significant change in a project.

Medium · behavioural and stakeholder questions

Approach

  1. Pick a story where you drove the decision, not one where you observed it.
  2. Quantify the outcome, including what you would not claim credit for.
  3. State the situation in two sentences and spend the rest on your reasoning.

Follow-up

  • How did you know the outcome was caused by your change?
  • What would you do differently if you ran that project again?

Defend a null result against a programme sponsor

Medium · regression to the mean · difference-in-differences · stakeholder management

A care management programme enrolled the top 1 percent of members by prior-year allowed spend. The sponsor's deck shows allowed PMPM for enrollees falling 34 percent from the year before enrollment to the year after, and asks for budget to triple the programme. You rebuild the evaluation with a concurrent comparison group selected by the identical spend rule in the same period. The difference-in-differences estimate is a 3 percent reduction with a confidence interval spanning zero. Deliverable: how you present this, to whom, in what order, and what you propose next.

Approach

  1. Name the probe: whether you can deliver a finding that costs somebody their programme without softening it into uselessness or creating an adversary who routes around you next time.
  2. Lead with the mechanism, not the verdict. Show the comparison group's own unadjusted drop, which will be large, because a cohort selected on an extreme of the outcome regresses toward the mean whether or not anyone intervenes. The sponsor's 34 percent is mostly that, and it is a property of the selection rule rather than a criticism of their clinicians.
  3. Give the sponsor the finding privately before it appears in any deck their leadership sees. Being surprised in a room is what turns a methods disagreement into a political one.
  4. State the estimate with its interval and say what it rules out as well as what it fails to establish. A 3 percent point estimate whose interval crosses zero is not evidence of no effect; it is insufficient power to separate a modest effect from none, and those are different claims.
  5. Arrive with a design rather than only an objection. Propose a regression discontinuity at the enrollment threshold if the rule is applied sharply, or a randomised rollout across the next wave of eligible members, and state the sample size needed to detect the effect size the sponsor believes in.

Follow-up

  • The sponsor says withholding the programme from a comparison group is unethical. What do you propose instead?
  • Leadership wants one number for the board next week. What do you give them?
  • What result would change your mind and make you believe the 34 percent?