Interview guides · Eli Lilly

Eli Lilly Data Scientist Interview Questions & Guide 2026

Diagnose a metric move for Eli Lilly data scientist prep: split by segment, rule out a logging change, separate mix shift from behaviour.

The 60-second brief

As a Data Scientist at Eli Lilly, you are at the intersection of cutting-edge life sciences and advanced computational analytics. This role is pivotal to Eli Lilly’s mission of discovering and delivering innovative medicines that make life better for people around the world. You will work on high-impact projects that range from drug discovery and clinical trial optimization to commercial analytics and patient-centric product strategy.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

Risk-adjust outcomes before comparing providers or panels
Separate censoring from events in time-to-event models
Minimise protected health information in every extract

Overview

The role demands a unique blend of technical rigor and business acumen. You will be expected to translate complex biological or business problems into actionable, data-driven insights. Whether you are building predictive models for patient outcomes or designing experiments to measure the efficacy of new initiatives, your work directly influences the strategic direction of one of the world's leading pharmaceutical companies.

You will join a global, cross-functional team, often collaborating with clinical researchers, commercial leaders, and engineers. This is an environment where precision is non-negotiable, and your ability to communicate complex findings to non-technical stakeholders is as vital as your ability to write clean, efficient code. Success in this role requires both deep scientific curiosity and a pragmatic, results-oriented mindset.

The Eli Lilly interview process

Eli Lilly candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Panel Interviews

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Rates built on member counts rather than exposure

Members join and leave mid-period, so dividing events by distinct members mixes a person covered for 30 days with one covered for 365. New joiners also have artificially low observed utilisation because their claims have not arrived yet and because care takes time to initiate. Denominators must be member-months or member-years, and comparative quality measures usually need a continuous-enrolment requirement with an explicit allowable gap, stated in days.

Ignoring clustering by provider, facility, or site

Patients within a provider panel, and subjects within a trial site, are correlated on both case mix and practice pattern. Standard errors computed as if observations were independent are too narrow by roughly the design effect, 1 + (m - 1) * ICC, where m is the average cluster size. With 50 patients per provider and an intracluster correlation of only 0.02, variance is understated by about a factor of two, which manufactures significant provider differences out of noise. Cluster-robust errors or a random intercept per provider, and cluster-level randomisation when you design the test, are the corrections.

Over-explaining the method and under-explaining the implication

Lead with the answer and what you would do about it, then give the approach when asked. Roughly one sentence of method per three of implication is the right ratio for a stakeholder-facing answer; the interviewer already knows what a regression is.

Solving silently instead of narrating the reasoning

Say which branch you are taking and why you chose it over the alternative, for example checking the denominator first because it changes what the comparison means. A correct answer that arrives with no visible path scores below a rigorous one that needed a hint.

Questions & practice

Describe the assumptions required for linear regression and how you validate them.

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Sessionise transfer chains into episodes, then count readmissions

Hard · sessionisation · cumsum grouping · readmission

encounter has encounter_id, patient_id, facility_id, encounter_type, admit_ts, discharge_ts, admission_type, discharge_disposition, is_planned_admission, principal_diagnosis_code. Chain acute-to-acute transfers into episodes of care: an inpatient encounter discharged as transfer_acute, followed by another inpatient encounter for the same patient at a different facility admitting within 24 hours of that discharge, belongs to the same episode. Then compute the 30-day unplanned readmission rate over episodes, excluding index episodes that are planned, that end in expired, hospice or against medical advice, or that are still open. Return encounter_id to episode_id, plus the rate.

Approach

Restrict to inpatient encounters for the chaining step and sort by patient_id, admit_ts. Observation and emergency encounters are not acute inpatient stays and pooling them changes both the chain and the denominator.
Build a boolean continues flag: the previous row is the same patient, its discharge_disposition is transfer_acute, its facility_id differs from this row's, and admit_ts minus previous discharge_ts is between 0 and 24 hours. Episode id = cumsum of the negation of that flag. This is the vectorised form of a sessionisation loop and is why the pattern generalises to any event stream.
Collapse to episode grain taking first admit_ts, last discharge_ts, last discharge_disposition, and any() of is_planned_admission. The episode inherits its exit state, not its entry state, which is why a transfer chain ending at home is one home discharge rather than two transfers.
Apply index exclusions at episode grain. Dropping episodes that end in death is not optional: a dead patient cannot be readmitted, so leaving them in inflates the denominator and depresses the rate for exactly the panels caring for the sickest members.
For each surviving index episode, find the next unplanned acute inpatient episode for that patient admitting within 30 days of the index discharge_ts. Count events at episode grain, since one patient can contribute several index episodes. Report the raw rate and say plainly that it is not comparable across panels without risk adjustment, which is what the observed-over-expected form exists for.

Worked solution (45 min)

Filter to encounter_type 'inpatient', sort by patient_id and admit_ts, and shift discharge_ts, discharge_disposition, facility_id and patient_id by one row.
Compute continues as same patient AND prior disposition is transfer_acute AND facility differs AND 0 <= (admit_ts - prior discharge_ts) <= 24 hours; episode_id = (~continues).cumsum().
Aggregate to episodes: first admit_ts, last discharge_ts, last discharge_disposition, any planned flag, patient_id.
Drop index-ineligible episodes: planned, disposition in expired, hospice or ama, and null discharge_ts.
For each eligible index episode, search the same patient's later episodes for an unplanned admit_ts within 30 days of index discharge_ts; a merge_asof on patient with a forward direction and a 30-day tolerance does this without a nested loop.
Rate = index episodes with an event divided by eligible index episodes; return the encounter-to-episode map alongside it.

Expected result: Two outputs: an encounter_id to episode_id map, and one rate. Worked case: encounter A discharged transfer_acute at 10:00, encounter B admitted 14:00 the same day at a different facility and discharged home on day 5, encounter C an unplanned admission on day 20. A and B form one episode, C is a readmission of it, and the patient contributes one eligible index episode and one event, not two indexes and possibly two events.

Verify it yourself

Every closed episode ends in a disposition other than transfer_acute; an episode ending in transfer_acute means a chain was cut early.
Episode count is less than or equal to inpatient encounter count, with equality only when the sample contains no transfers.
Turning the chaining rule off raises the readmission rate. The direction of that change is the proof your chaining fired at all.
No index episode's matched readmission is an encounter already inside that same episode.

Follow-up

discharge_ts is null on an open stay in the middle of a chain. What does your flag do, and what should it do?
A patient is transferred out and then back to the originating facility within 24 hours. Should that chain, and does your facility_id condition handle it?
Is a readmission itself eligible to serve as a later index episode? Say what you chose and what the choice does to the rate.

Four data-quality rules over corrected and cancelled lab results

Easy · data quality · record supersession · unit drift

lab_result has result_id, order_id, patient_id, loinc_code, value_numeric, value_text, units, reference_low, reference_high, specimen_collected_ts, resulted_ts, result_status (preliminary, final, corrected, cancelled), supersedes_result_id. Return one row per rule with a failing count, a failing share, and one example result_id, for these rules: more than one surviving row per (order_id, loinc_code) after resolving corrections and dropping cancelled; resulted_ts earlier than specimen_collected_ts; a numeric result whose unit is not the modal unit for its loinc_code; value_numeric and value_text both null on a row with result_status 'final'.

Approach

Resolve the correction chain before anything else. Drop result_status 'cancelled', then within each (order_id, loinc_code) keep the row whose result_id appears in no other row's supersedes_result_id. That terminal node is the surviving value and the rule holds even if a correction is loaded with a backdated timestamp.
Build each rule as a boolean mask over a named frame rather than as a filtered copy, so the failing share has an explicit, per-rule denominator. Rule three is eligible only on rows with a non-null value_numeric; rule four only on final rows.
For the unit rule, derive the modal unit per loinc_code from the data instead of hardcoding an expected unit. The failure being hunted is one analyte reported in two units whose values differ by a fixed factor, which no range check catches because both sets of numbers look plausible.
Emit a tidy frame with columns rule, eligible_rows, failing_rows, failing_share, example_result_id. A printed report cannot be diffed between runs; a frame can be stored and alerted on.
Order the output by failing_share descending so the run has a lead finding rather than four equal-weight lines.

Follow-up

The modal-unit rule fires on 8 percent of one analyte. How do you decide whether that is a unit-conversion bug or a second legitimate assay?
What would you add to catch a value that is inside its reference range but physiologically impossible?
Which of these four rules should block a downstream pipeline and which should only alert, and why?

Why do you want to join Eli Lilly, and where do you see your impact in five years?

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Can you describe how to implement a complex algorithm directly within a database environment?

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

How do you use SQL window functions to calculate running totals or period-over-period growth?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Find inpatient encounters with no resulted lab during the stay

Medium · anti-join · null semantics · event matching

encounter holds encounter_id, patient_id, encounter_type, admit_ts, discharge_ts and facility_id. lab_result holds result_id, order_id, patient_id, encounter_id (NULL for outpatient standing orders), loinc_code, specimen_collected_ts, resulted_ts and result_status. Return every inpatient encounter discharged in 2025 with no final or corrected lab_result linked to it. A colleague wrote WHERE encounter_id NOT IN (SELECT encounter_id FROM lab_result) and got zero rows back. Say why, write the correct query, then say what changes when labs must instead be matched on patient_id with the specimen collected between admit_ts and discharge_ts.

Approach

Name the mechanism rather than the symptom. x NOT IN (subquery) expands to x <> y1 AND x <> y2 AND ..., and a single NULL y makes that chain UNKNOWN, never TRUE. Since lab_result.encounter_id is nullable by design for standing orders, this query can only ever return zero rows.
Rewrite as NOT EXISTS, or LEFT JOIN with an IS NULL filter. Both treat a NULL key as no match rather than as unknown. Adding IS NOT NULL inside the NOT IN subquery also works, but NOT EXISTS is the habit that survives someone making another column nullable later.
Put the result_status predicate inside the correlated subquery or the ON clause, not in an outer WHERE. Outside, it turns the anti-join back into an inner join and an encounter whose only labs were cancelled disappears instead of qualifying.
For the timestamp variant, match on patient_id with specimen_collected_ts inside the stay rather than resulted_ts. A specimen drawn an hour before discharge can result the next day, and anchoring on resulted_ts would wrongly call that stay lab-free.
Restrict to encounter_type 'inpatient' and discharge_ts in 2025, and report open encounters with a NULL discharge_ts as a separate count instead of letting the filter swallow them.

Worked solution (20 min)

Demonstrate the cause: SELECT COUNT(*) FROM lab_result WHERE encounter_id IS NULL returns a non-zero number.
Write the NOT EXISTS version with result_status IN ('final','corrected') inside the correlated subquery.
Write the LEFT JOIN version with the same predicate in the ON clause and WHERE l.result_id IS NULL, then compare counts.
Build the timestamp variant joining on patient_id with specimen_collected_ts >= admit_ts AND < discharge_ts.
Report both counts side by side and explain which encounters differ between the two definitions.

Expected result: One row per qualifying inpatient encounter with encounter_id, patient_id, facility_id, admit_ts and discharge_ts. The count is strictly below total 2025 inpatient discharges and strictly above zero, while the NOT IN version returns exactly zero rows.

Verify it yourself

NOT EXISTS and LEFT JOIN with IS NULL return identical row counts.
Adding AND encounter_id IS NOT NULL inside the NOT IN subquery reproduces the NOT EXISTS count exactly.
No returned encounter_id appears anywhere in lab_result with result_status in ('final','corrected').
The patient-and-timestamp variant returns fewer encounters than the encounter_id variant, since it also catches labs whose encounter link is missing.

Follow-up

Write the LEFT JOIN form and say exactly where the result_status predicate must sit for the two forms to agree.
How would you separate encounters whose only labs were cancelled from encounters with no lab rows at all?
Some encounters carry member_id NULL because the person index did not match. What does that do to a payer-side version of this measure?

A key performance metric drops suddenly; how do you diagnose the root cause?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a metric to measure the success of a new patient engagement program?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you manage your time and prioritize tasks when working on multiple research projects?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you ensure statistical significance in a low-sample-size clinical or commercial study?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

What are the most common experimentation pitfalls you have encountered in your previous projects?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

Power a readmission trial on a rare binary outcome

Easy · power analysis · minimum detectable effect · rare events

A care-transitions programme (pharmacist call plus a seven-day follow-up visit) will be tested on index inpatient discharges drawn from encounter, excluding planned admissions, acute-to-acute transfers, AMA discharges, and dispositions of expired or hospice. Baseline 30-day unplanned readmission on eligible stays is 15.8 percent. Leadership wants to detect a 1.5 percentage point absolute reduction, two-sided alpha 0.05, 80 percent power, 1:1 individual randomisation at discharge. Give the required index stays per arm, and the minimum detectable absolute effect if only 4,000 eligible stays per arm accrue in the study window.

Approach

Write the two-proportion sample size formula and state its preconditions: normal approximation, two-sided test, equal allocation, and the outcome measured on index stays rather than on members, since one member can contribute several index stays.
Plug in p1 = 0.158, p2 = 0.143, and (z_0.975 + z_0.80)^2 = (1.960 + 0.842)^2 = 7.849. Report the answer as index stays, and separately as the calendar time it implies given the observed monthly eligible volume.
Invert the same formula for the fixed-N case: solve for delta with n = 4,000 per arm, and say whether you used pooled or arm-specific variance because the two answers differ by about 0.1 percentage points.
Translate the MDE into the relative reduction it implies and hand it back to leadership before the study starts, so the question becomes whether that effect is worth running rather than whether the null result was a failure.
Flag the two adjustments that will make the real number larger: if a member can appear more than once, cluster on member_id, and if the programme is delivered by unit or by discharging team rather than to individuals, an individual-level calculation is the wrong model entirely.

Worked solution (20 min)

Compute (1.960 + 0.842)^2 = 7.849.
Compute the variance sum: 0.158 x 0.842 = 0.1330, 0.143 x 0.857 = 0.1226, total 0.2556.
n per arm = 7.849 x 0.2556 / 0.015^2 = 7.849 x 1,136 = 8,916, so round to about 8,920 per arm.
For the MDE at n = 4,000, solve delta = sqrt(7.849 x variance sum / 4,000); with arm-specific variance this gives 0.0221, with pooled variance at p = 0.158 it gives 0.0229.
Express the MDE as a relative effect: 0.023 / 0.158 is about a 15 percent relative reduction.

Expected result: About 8,900 index stays per arm (roughly 17,800 total) to detect 1.5 percentage points. With 4,000 per arm the MDE is about 2.2 to 2.3 percentage points absolute, which is a 14 to 15 percent relative reduction; state which variance convention you used.

Verify it yourself

Halving the target effect should roughly quadruple N: at 3.0 points the formula gives about 2,130 per arm, and 8,916 / 2,130 is about 4.2.
Confirm the normal approximation holds: expected events per arm are 0.158 x 8,916 = about 1,409, far above the usual threshold of 10.
Divide required N by observed monthly eligible index stays and check the implied study duration against the decision deadline.

Follow-up

The same programme is delivered by discharge unit, not to individual patients. What changes in the calculation?
How would you handle a member who has three eligible index stays during the study window?
Leadership proposes powering on a composite of readmission or ED revisit instead. What does that buy and what does it cost?

Adherence share collapsed the month a feed was added

Easy · pdc · interval sql · identity resolution

The share of members with proportion of days covered at or above 0.80 in one therapeutic class fell from 0.71 to 0.58 in a single month. pharmacy_claim carries fill_id, member_id, ndc_code, therapeutic_class_code, fill_date, days_supply, quantity_dispensed, refill_number, reversal_flag, reversed_fill_id and source_system_code. There is no person-level key above member_id and no enrollment table on hand, so every diagnostic has to run on pharmacy_claim alone. That month the pipeline began ingesting a second pharmacy source. Determine whether adherence actually changed, and hand back a PDC query you would defend line by line.

Approach

Compute controls that depend on neither member identity nor the PDC code: total fills in the class per month, and total days_supply dispensed in the class per month. Both are sums over rows, so neither moves when one person's fills split across two member_ids, which is exactly what a per-member control cannot promise. If both are flat while PDC falls, no dispensing behaviour changed and the computation or the keys did.
Check days_supply for nulls and zeros by source_system_code. If the new source omits days_supply and the code coalesces null to zero, covered days vanish and PDC falls even though the fills are there.
Check for member identity splitting using the columns you actually have. There is no person key above member_id, so the split has to be found by its footprint: distinct member_id in the class jumping at the break while total fills stay flat, and a rise in the share of member_ids whose earliest fill for a given ndc_code carries refill_number above zero, which is a continuation history that lost its beginning. When one person is split, each id may fall below the two-fill denominator requirement and drop out, and the ids that do survive carry artificial gaps.
Check reversals. A reversed fill is two rows, the fill and a row with reversal_flag TRUE pointing at reversed_fill_id, and both must be excluded from exposure. If the new source expresses reversals differently and reversed_fill_id does not link, phantom coverage appears, which pushes PDC up rather than down, so use this to rule the cause in or out.
Verify the denominator and window are unchanged: members with at least two fills in the class and at least 91 days of follow-up, over a stated fixed 12-month window, never a mixture of window lengths.
Write the PDC query so overlapping refills shift the next interval's start to the later of fill_date and the prior interval's end, so a day is covered at most once, and cap at one day per calendar day.

Follow-up

Why is PDC preferred to medication possession ratio when early refills are common?
You want to compare adherent and non-adherent members on hospitalisation. What is wrong with classifying on a 12-month PDC and then following from day zero?
A member switches to a different drug in the same class mid-year. How does your interval logic handle that, and is the class the right grain?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Getting an analysis acted on

Work that nobody used is a common and unflattering pattern in data careers, and interviewers probe for it. Have a story about an analysis that changed a decision, and be specific about how you got it in front of the person who could act. Also have one about work that went nowhere, with your reading of why.

Tell us about a time you faced a significant challenge in a project; what was your approach and the final result?

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Own an analysis that shipped wrong and was acted on

Easy · retrospective · ownership · data quality

Describe an analysis you delivered that was wrong, where someone acted on it before the error surfaced. Pick something with a mechanism you can name, not a typo. Cover what the number claimed, what was actually true, how much time passed before it was caught, who caught it, and what you told the people who had already acted. Deliverable: a five-minute account that ends with the specific control you added afterwards, plus one occasion since when that control has fired.

Approach

Read the probe: the interviewer is testing whether you detect your own errors and whether the correction was structural. Pick an error with a nameable mechanism (a denominator built from distinct members instead of member-months, a series read before claims runout completed, a cohort selected on the outcome) so the story has a diagnosis rather than a mood.
Open with the decision that was made on your number, not with the bug. The severity of an analytic error is measured in the action it caused: a rate filing, a staffing plan, a programme expansion.
State the counterfactual number. 'The true figure was 14.1 admissions per 1,000 member-years, not 9.8' is a fact; 'it was off by a lot' is not.
Describe the detection path honestly, including whether you found it or somebody else did. Say how long it took, because time to detection is the part you actually control next time.
Close on the control, and make it something that protects a reader who has never met you: a unit test on the denominator, a paid-through date stamped on every chart, a required exclusions block in the spec template. Name one later date it caught something.

Follow-up

What would have had to be true for you to catch this before delivery rather than after?
Did the stakeholder change how much they trust your work, and did you want them to?
Give an example of a control you added that turned out to be the wrong control.

Make an honest case for your own analytical impact

Hard · impact attribution · counterfactual · self-assessment

You are asked to account for a year of your work. Constraint: you may only use measures that existed before you knew the outcome, and you must include one project that moved nothing. The portfolio contains a measure specification that changed how a contract settles, a predictive model that shipped, a self-serve dashboard, and an evaluation that killed a proposed programme. Deliverable: how you attribute value to each, which of them you claim credit for and which you do not, and the one number you would accept being judged on next year.

Approach

The probe is whether you can reason counterfactually about your own work, which is the same discipline this domain demands of you for programmes, applied where it is uncomfortable.
State the counterfactual for each item: what would have happened without you. A measure specification that changed how a contract settles has a dollar counterfactual you can compute. A dashboard's counterfactual is analyst hours, which is real and small, and attaching a revenue number to it is where these accounts usually turn into fiction.
Treat the killed programme as your strongest item and say why. Avoided spend is real value, it is attributable because the decision turned on your evaluation, and it is the one item nobody else will count on your behalf.
Report the shipped model the way you would report anyone's model: calibration in the subgroups that matter, adoption by the people it was built for, and whether any decision changed. A model that is live but ignored has zero impact, and saying so is the credibility test in this answer.
Name the failed project with its mechanism and its cost, then say what earlier signal you would watch to kill it sooner. Time to kill is a legitimate impact measure.
Choose next year's number so that it can go down. A metric that only ratchets upward is not a commitment.

Follow-up

Two other people also claim the contract measure. How do you split it?
Your model is live and used, but the decisions it informs would have been the same without it. Is that impact?
What is the strongest argument that your year had less impact than you just described?