Interview guides · Health Catalyst

Health Catalyst Data Scientist Interview Questions & Guide 2026

Causal inference drills for Health Catalyst data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

A Data Scientist at Health Catalyst plays a pivotal role in transforming healthcare data into actionable insights that drive improved outcomes and efficiencies. This position is crucial not just for the refinement of products but also for enhancing user experiences and supporting strategic business objectives. You will engage with complex datasets, applying statistical analyses and machine learning techniques to solve real-world health challenges. Your work will directly impact health systems, providers, and patients by optimizing decision-making processes and promoting data-driven solutions.

When randomisation is off the table, the skill being checked is naming an identification strategy together with the assumption it rests on: parallel trends for difference-in-differences, relevance and exclusion for an instrument, overlap and conditional ignorability for matching. Say the assumption out loud and say how you would try to break it.

Focus areas

Reconstruct drug exposure intervals from dispensing records
Correct for claims runout before reporting recent months
Minimise protected health information in every extract

Overview

In your role, you will collaborate with multidisciplinary teams, including engineers, product managers, and operations staff, to develop and implement analytical frameworks. You might work on projects that involve predictive modeling, health analytics, and data visualization, contributing to innovative solutions that align with our mission of empowering healthcare organizations through data. Expect to engage with real-world problems where your analytical skills will not only be tested but celebrated as you influence healthcare delivery.

The Health Catalyst interview process

Health Catalyst candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessments

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Interviews

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
The counterfactual: what would have been done without your analysis, and why that default was worse
How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Final Interviews

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Where candidates lose points

Rates built on member counts rather than exposure

Members join and leave mid-period, so dividing events by distinct members mixes a person covered for 30 days with one covered for 365. New joiners also have artificially low observed utilisation because their claims have not arrived yet and because care takes time to initiate. Denominators must be member-months or member-years, and comparative quality measures usually need a continuous-enrolment requirement with an explicit allowable gap, stated in days.

Reading the most recent months of a claims-based series as real

Claims incur before they are reported and paid, so recent incurred months are systematically undercounted until runout completes. The lag is not uniform: pharmacy adjudicates in days, professional claims in weeks, inpatient facility claims in months. That means recent data is both too low and mix-shifted toward cheap services, which reads as a cost improvement and a utilisation drop at once. The fix is to hold the last three incurred months back or apply completion factors, and to state the paid-through date on every chart.

Solving silently instead of narrating the reasoning

Say which branch you are taking and why you chose it over the alternative, for example checking the denominator first because it changes what the comparison means. A correct answer that arrives with no visible path scores below a rigorous one that needed a hint.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

Can you explain the concept of overfitting and how to prevent it?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

How do you evaluate the performance of a predictive model?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

How would you optimize a given algorithm for speed or memory usage?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Simulate how ignoring clustering inflates the false positive rate

Medium · simulation · design effect · type i error

Simulate a provider-level comparison with no true effect. 40 providers, 60 members each, a continuous outcome with total variance 1 and intraclass correlation 0.02. Randomise providers 20 to 20, not members. Over 5,000 replications, report the share of replications where a two-sample t-test run on all 2,400 individual observations, ignoring provider, returns p below 0.05. Report the same share when the unit of analysis is the 40 provider means. Then state how the first number follows from the design effect 1 + (m - 1) x ICC.

Approach

Decompose the variance explicitly rather than tuning it: provider variance = ICC = 0.02, residual variance = 1 - ICC = 0.98. Draw u_j once per provider and e_ij per member, so the outcome is u_j + e_ij and the correlation between two members of one provider is 0.02 by construction.
Randomise at the provider level. That is the entire mechanism: the treatment indicator is constant within a provider, so in any single replicate the provider intercepts are confounded with the arm, and the naive test reads that confounding as signal.
Vectorise over replications with a (reps, providers, members) array. 5,000 x 2,400 normals is trivial in numpy, and a Python loop is what tempts people to cut replications to 500 and report a number with Monte Carlo noise of plus or minus 0.017.
Predict the answer before running it: design effect = 1 + (60 - 1) x 0.02 = 2.18, naive standard errors are too small by sqrt(2.18) = 1.476, so the rejection rate at nominal 0.05 is about 2 x (1 - Phi(1.96 / 1.476)) = 0.18. Agreement between prediction and simulation is what makes the result a demonstration rather than an anecdote.
Run the provider-mean analysis as the control. Recovering 0.05 there proves the generator is correct and isolates the failure to the analysis unit.

Worked solution (30 min)

Draw u with shape (reps, 40) from N(0, sqrt(0.02)) and e with shape (reps, 40, 60) from N(0, sqrt(0.98)); y = u[:, :, None] + e.
Assign providers 0-19 to one arm and 20-39 to the other; because there is no true effect, a fixed split is valid and removes one source of Monte Carlo noise.
Naive test: flatten each arm to 1,200 observations and compute a two-sample t statistic per replication with vectorised means and pooled variance.
Cluster test: average within provider to 40 values, then run the same two-sample t on 20 versus 20.
Report both rejection shares and compare the naive one against 2 x (1 - Phi(1.96 / sqrt(2.18))).

Expected result: The naive individual-level test rejects in roughly 18 percent of replications, against a nominal 5 percent. Monte Carlo standard error at 5,000 replications is about 0.005, so anything in 0.17 to 0.19 is consistent. The provider-mean analysis rejects in about 5 percent, within roughly 0.003.

Verify it yourself

Set ICC to 0 and both analyses return about 0.05. If the naive one does not, the data generator is wrong, not the test.
Raise members per provider from 60 to 120 with ICC fixed and the naive rate rises, because the design effect grows linearly in cluster size.
The provider-mean analysis stays near 0.05 at every ICC you try, which is the definition of a valid test.
The simulated naive rate lands within about 0.01 of 2 x (1 - Phi(1.96 / sqrt(design effect))).

Follow-up

How many providers would you need for 80 percent power on a 0.2 SD difference with 60 members each and this ICC?
Cluster sizes are unequal in reality. The design effect approximation becomes 1 + ((1 + CV^2) x mbar - 1) x ICC. Which direction does that move your sample size and why?
The intervention cannot be withheld from any provider. Sketch a design that still yields a defensible estimate.

Write a function to calculate the area under a curve.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Turn overlapping coverage spans into fractional member-month denominators

Medium · intervals · gaps and islands · denominators

member_enrollment holds span_id, member_id, plan_id, product_type, effective_date and termination_date, which is NULL while the span is active. A member can hold several spans in a year, and back-dated plan changes make spans overlap. Return one row per calendar month of 2025 with total member-months, where a member contributes covered days in the month divided by that month's length, and a day covered by two overlapping spans counts once. Both span endpoints are inclusive. Collapse overlaps with window functions before joining a month calendar, and do not assume one span per member.

Approach

Normalise the open end first. COALESCE(termination_date, DATE '9999-12-31') or clip to the reporting end, because a NULL termination_date makes every BETWEEN comparison unknown and drops exactly the members who are still covered.
Merge overlapping spans per member before the calendar join. Order by effective_date, carry a running MAX of prior termination_date with a frame ending one row before, and open a new island where effective_date exceeds that running max plus one day, which keeps adjacent spans contiguous rather than splitting them.
Clip each merged island to the reporting year, then cross join to a twelve-row month calendar and keep pairs that intersect.
Compute covered days as LEAST(island_end, month_end) minus GREATEST(island_start, month_start) plus one, and divide by the actual length of that month rather than a flat 30, so February and the 31-day months are not distorted.
Sum the fractions by month. Merging after the calendar join also works but forces a DISTINCT over member-days, which stops scaling at population size.

Worked solution (30 min)

CTE bounded: clip spans to 2025-01-01 through 2025-12-31 with COALESCE on termination_date, discarding spans that do not intersect the year.
CTE merged: gaps-and-islands per member using LAG and a running MAX over preceding rows to collapse overlapping and adjacent spans.
CTE months: twelve rows of month_start, month_end and days_in_month.
Join merged to months on interval intersection and compute covered_days per pair.
SUM(covered_days / days_in_month::numeric) grouped by month, ordered by month.

Expected result: Twelve rows of (month, member_months) with fractional values. Any single member contributes at most 1.0 to a month and at most 12.0 across the year. A member covered 2025-01-01 through 2025-02-20 contributes 1.0 in January and 20/28 = 0.714 in February.

Verify it yourself

For a member with spans 2025-01-01..2025-03-31 and 2025-03-15..2025-06-30, March contributes exactly 1.0, not 1.5.
Total member-months is at most 12 times COUNT(DISTINCT member_id) and at least the count of members covered the whole year.
Forcing every termination_date to NULL raises the total. If it drops to zero, the NULL handling is missing.

Follow-up

Where does a continuous-enrolment filter with an allowable 45-day gap belong, and how does that change the island boundary rule?
One human appears under two member_ids after a master person index merge. What does that do to member-months, and how would you detect it?
How do you treat a span that terminates on the day it starts, and what does that row usually represent?

How would you approach a project where the requirements are not clearly defined?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Given a dataset with various health metrics, how would you identify trends?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

What is the difference between supervised and unsupervised learning?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Decompose risk-adjusted cost per member into owned leaf metrics

Hard · metric tree · price volume mix · coding intensity · ownership

Build a metric tree, at most four levels deep, from risk-adjusted allowed PMPM down to leaves an owner can move. Data: medical_claim_line (allowed_amount, units, procedure_code, place_of_service_code, service_start_date, claim_version, frequency_code), pharmacy_claim (allowed_amount, days_supply, is_generic, formulary_tier, reversal_flag), member_enrollment (coverage spans, prospective_risk_score). Give the arithmetic identity each split rests on, the leading indicator at each leaf, and identify the one node where the arithmetic is exact but the attribution is not.

Approach

State the base identity before drawing anything. Allowed PMPM equals total allowed over member-months, and it decomposes exactly by service category c as the sum over c of (units_c / member-months) times (allowed_c / units_c). Utilization per member-month and allowed per unit, summed across categories, so category mix is inside the sum rather than a separate hand-wave.
Give the change decomposition too, because the tree will be read on deltas: the change in PMPM equals the sum over categories of (change in utilization times base price) plus (base utilization times change in price) plus (change in utilization times change in price). The third term is the interaction, it is not noise, and an analysis that drops it will not reconcile.
Build the medical branch to leaves with owners: inpatient admits per 1,000 member-months and allowed per admit (with DRG case mix as the sub-split), ambulatory visits per 1,000 and allowed per visit split by place_of_service_code, and emergency visits per 1,000 and allowed per visit. Prepare the data with version dedup to the surviving claim_version before a single dollar is summed, since frequency_code 7 replaces and 8 voids.
Build the pharmacy branch on 30-day equivalents: scripts_30 equals the sum of days_supply over 30 on unreversed fills, so pharmacy PMPM equals (scripts_30 per member-month) times (allowed per 30-day script). Split the price term exactly as g times the generic price plus (1 minus g) times the brand price, where g is the generic dispensing rate, which makes generic substitution and unit price separable leaves with different owners.
Attach one leading indicator per leaf: prior-authorization approvals and elective surgical bookings ahead of inpatient admits, contract effective dates ahead of allowed per unit, site-of-service steering ahead of the ambulatory place_of_service mix, new-to-brand starts and formulary tier placement ahead of pharmacy price.
Name the node where arithmetic is exact but attribution is not: the risk-adjustment denominator. Dividing by the population mean prospective_risk_score is exact arithmetic, but the score is built from coded diagnoses, so more complete coding raises the denominator and lowers risk-adjusted PMPM with no change in care delivered. Carry raw PMPM, the mean risk score trend and a coding-intensity measure such as diagnoses per member-year beside it, permanently.

Worked solution (45 min)

Write the base identity and the three-term change decomposition on paper first, and confirm the terms sum to the total change on a two-category toy example before touching real data.
Specify the preparation rules that precede any aggregation: dedup medical_claim_line to the surviving version per claim_id using claim_version and frequency_code, drop both rows of every reversed pharmacy pair, and build member-months from coverage spans with partial months counted fractionally.
Draw the medical branch and the pharmacy branch to leaf level, writing the identity beneath each split rather than only the metric name.
Write the generic-dispensing split as the exact price identity and confirm it reproduces the observed allowed per 30-day script.
Attach one leading indicator and one named owning function to each leaf.
Write the caveat on the risk-adjustment node, including the two companion series that must be published with it.

Expected result: A four-level tree rooted at risk-adjusted allowed PMPM, splitting by service category into utilization per member-month and allowed per unit, with an inpatient sub-split on DRG case mix, an ambulatory sub-split on place_of_service_code, and a pharmacy branch on 30-day equivalents whose price term is split exactly by generic dispensing rate; each leaf carries a named owner and a leading indicator; and the risk-adjustment denominator is flagged as exact arithmetic with unreliable attribution, published alongside raw PMPM, the mean risk score trend and a coding-intensity series.

Verify it yourself

Reconcile the leaves back to the root: the summed category contributions must equal total allowed PMPM to within rounding, and the three change terms must sum to the observed period change.
Recompute total allowed with and without version dedup. Any gap is the double-counted adjusted claims the tree would otherwise attribute to real utilization.
Verify the pharmacy price identity numerically: generic dispensing rate times generic price plus its complement times brand price should reproduce the observed blended allowed per 30-day script.
Hold care constant and simulate a 5 percent rise in mean prospective_risk_score. Risk-adjusted PMPM should fall by roughly the same proportion, which is the attribution problem made visible.

Follow-up

Utilization is flat and allowed per unit is flat, yet PMPM rose 4 percent. Name every place the increase can be hiding.
Your member-month denominator came from coverage spans that overlap across products for some members. What does that do to PMPM, and how do you decide whether to deduplicate?
Which leaf would you refuse to put on a team's scorecard, and why?

Protocol deviations fell as new sites came online

Hard · reporting lag · denominator mix · clustering

Protocol deviations per 100 completed visits across a multi-site study fell from 4.6 to 2.9 over two months while randomisation accelerated. study_subject_visit carries study_id, site_id, subject_id, visit_number, visit_name, planned_visit_date, actual_visit_date, visit_window_days, visit_status, protocol_deviation_flag, deviation_severity, data_entry_ts and site_activation_date. Operations wants to credit a retraining rollout that began in that window. Establish whether deviation behaviour changed, and deliver a version of the series that is safe to review monthly.

Approach

Treat deviation flags as late-arriving data. Build a lag triangle of data_entry_ts minus actual_visit_date, because deviations are largely identified during monitoring review weeks after the visit, so the two most recent months are structurally undercounted in exactly the way an improvement looks.
Hold back the months that are not developed, or apply development factors fitted on fully developed months, and restate. Do this before any explanation involving training.
Attack the denominator. It counts visits with visit_status 'completed', while out_of_window and missed visits are excluded even though out-of-window attendance is itself a deviation in most protocols, so a site that pushes visits out of window lowers the rate twice over.
Standardise the visit mix. Newly activated sites contribute mostly screening and baseline visits, which carry fewer procedures and fewer opportunities to deviate than later treatment visits, so compute the rate within visit_number strata and reweight to a fixed mix before comparing months.
Split major from minor deviation_severity and report them separately. A single major deviation can remove a subject from the per-protocol analysis set while ten minor ones may not, so a pooled rate can fall while the consequential series rises.
Before attributing anything to training, model at the right level: subjects are nested in sites, so use a random intercept per site or cluster-robust standard errors. With average cluster size m and intracluster correlation rho the variance is understated by the design effect 1 + (m - 1) times rho, which at m = 60 and rho = 0.02 is about 2.2, so naive intervals are roughly 1.5 times too narrow.

Follow-up

Design a defensible evaluation of the retraining rollout, given it was deployed site by site and cannot be withheld.
Your reweighted series is flat but major deviations rose. What do you escalate and to whom?
How would you decide the holdback period, and how would you communicate a provisional number without it being quoted as final?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

How do you handle feedback and criticism?

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Describe a time when you faced a significant challenge while working on a team.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Disagree with a product manager about an adherence nudge

Medium · immortal time · confounding · disagreement

A product manager proposes shipping a refill-reminder feature to the whole member base, citing an internal analysis: members with 12-month proportion of days covered at or above 0.80 had 31 percent fewer admissions than members below it. The roadmap date is three weeks out and engineering is scoped. You believe the comparison builds survival and continuous coverage into the exposure definition, and is separately confounded by the kind of member who refills on time. Deliverable: what you say in the room, what you send afterwards, and the smallest study you would accept as sufficient to ship.

Approach

The probe is whether you can be specific and non-territorial at once. A disagreement that arrives as 'the analysis is flawed' costs you the room; one that names a mechanism and offers a cheaper path keeps it.
Name the two defects separately, because they have different fixes. Classifying a member as adherent over 12 months requires them to stay alive and stay covered for those 12 months, and that guaranteed event-free interval is assigned to the adherent group. Separately, members who refill on schedule differ from those who do not in ways claims never record.
Show rather than assert. Rerun the same comparison as a landmark analysis: classify adherence over days 1 to 90, start follow-up at day 91 for everyone still enrolled and event-free, and report how much of the 31 percent survives.
Separate the association claim from the intervention claim out loud. Nothing in the analysis speaks to whether a reminder changes refill behaviour, which is the actual product question and is separately testable.
Offer the shippable path: a randomised holdout on a subset with proportion of days covered as the primary endpoint, powered on the plausible effect of a reminder rather than on the 31 percent, with a stated read date.

Follow-up

The landmark rerun still shows a 20 percent difference. Do you ship?
The product manager says a holdout delays launch. How do you cost that argument?
What primary endpoint do you choose, and why not admissions?