AbbVie Data Scientist Interview Questions & Guide 2026

The 60-second brief

This guide covers what a Data Scientist at AbbVie is expected to do and how to prepare for the interview.

Learn the economics of the product category before the loop. Marketplaces, subscription products and ad-supported products turn on different core quantities (match rate and liquidity, retention and churn, fill rate and yield) and fail in different characteristic ways.

Focus areas

• Minimise protected health information in every extract
• Build member-month denominators from overlapping coverage spans
• Separate censoring from events in time-to-event models

The AbbVie interview process

AbbVie candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

• Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
• Whether you name what you have not done instead of stretching to cover every line of the posting
• Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

• Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
• Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
• Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Hiring Manager Screen

Much of this round runs on your own history, but the manager is not collecting a project list. They are working out what it is like when something goes wrong on your watch: how late the bad news tends to arrive, and whether a number you hand over has been checked by anyone including you. That is why the strongest material is a project where you can describe the part that did not work and what it cost. A result you cannot take full responsibility for, however clean, gives them nothing to trust you with afterwards.

What to demonstrate

• Whether you volunteer the limits of a result you are proud of, or wait to be pushed onto them
• How errors surfaced in your past work, and whether you or somebody else found them
• Whether the scope you claim matches the level of detail you can still produce about it
• What you did the first time a stakeholder acted on something of yours that turned out to be wrong

How to prepare

• Rebuild one headline figure from memory down to the join and the filter, so a question about the denominator does not stall the conversation
• For each project you raise, write the sentence you would say to someone who had already acted on a number that later turned out wrong
• Mark which parts of a project were yours and which belonged to other people, and state that boundary yourself before anyone asks

Panel Interview

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

• Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
• Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
• Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
• Whether you ask what decision is actually on the table before explaining anything

How to prepare

• Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
• For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
• Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Presentation Round

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

• Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
• Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
• Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

• For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
• Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
• For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Where candidates lose points

Ignoring clustering by provider, facility, or site

Patients within a provider panel, and subjects within a trial site, are correlated on both case mix and practice pattern. Standard errors computed as if observations were independent are too narrow by roughly the design effect, 1 + (m - 1) * ICC, where m is the average cluster size. With 50 patients per provider and an intracluster correlation of only 0.02, variance is understated by about a factor of two, which manufactures significant provider differences out of noise. Cluster-robust errors or a random intercept per provider, and cluster-level randomisation when you design the test, are the corrections.

Rates built on member counts rather than exposure

Members join and leave mid-period, so dividing events by distinct members mixes a person covered for 30 days with one covered for 365. New joiners also have artificially low observed utilisation because their claims have not arrived yet and because care takes time to initiate. Denominators must be member-months or member-years, and comparative quality measures usually need a continuous-enrolment requirement with an explicit allowable gap, stated in days.

Extrapolating a first-week lift inflated by novelty effects

Plot the treatment effect by days since first exposure instead of quoting one pooled average. A lift that decays toward zero across the test window is behaviour that will not persist, and annualising it produces a forecast that misses by an order of magnitude.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

Can you explain the underlying mechanics of Bayesian inference and how it can be applied to optimize clinical trial designs?

Medium · statistics and probability

Approach

1. Sanity-check the answer against a simple bound or a simulated case.
2. Write down the assumption the method needs before you use the method.
3. Translate the result into the decision it informs, in one plain sentence.

Follow-up

• What sample size would you need to detect an effect half this size?
• How would you explain this result to someone who does not know statistics?

What are some major experimentation pitfalls you might encounter when testing an algorithm in a live operational environment, and how would you mitigate them?

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Say how the offline result would be validated online before it is trusted.

Follow-up

• How would you choose the decision threshold, and who owns that choice?
• What would you monitor after launch to know the model is still valid?

Collapse claim versions before summing allowed amounts by member

Easy · claim versioning · groupby · deduplication

You are given medical_claim_line as a pandas DataFrame: claim_line_id, claim_id, claim_version, frequency_code (1 original, 7 replacement, 8 void), member_id, service_start_date, procedure_code, allowed_amount, claim_status. Return total allowed_amount per member for service dates in one stated calendar month, counting each claim once at its surviving version and excluding any claim whose surviving version is a void. Multi-line claims must keep every line of that surviving version. Output columns: member_id, allowed_amount, sorted descending. Do not deduplicate on claim_id alone.

Approach

1. Separate the two grains out loud before writing code: versions live at the claim level, dollars live at the line level. Every bug in this exercise comes from mixing them.
2. Compute the surviving version per claim with a groupby transform of max over claim_version, then keep the lines whose claim_version equals that value. A transform keeps the frame at line grain, which a sort plus head(1) does not.
3. Drop whole claims whose surviving version carries frequency_code 8. A void is not a zero-dollar line, it retracts the claim, so filtering rows rather than claims leaves the prior version's lines behind.
4. Filter to claim_status 'paid' and state the assumption: denied, pended and reversed lines carry an allowed_amount that never became a liability, so including them overstates cost.
5. Restrict service_start_date to the month boundaries, then groupby member_id and sum. Sort descending and return.

Worked solution (20 min)

1. Build a max-version lookup per claim_id with groupby('claim_id')['claim_version'].transform('max') and keep rows where claim_version equals it.
2. From the surviving rows, collect the claim_ids whose frequency_code is 8 and drop those claim_ids entirely.
3. Filter claim_status to 'paid' and service_start_date into the month window.
4. Group by member_id, sum allowed_amount, sort descending, reset the index.

Expected result: A two-column frame member_id, allowed_amount. On a claim with versions 1, 7, 7 only the third version's lines contribute; a claim whose latest version is frequency_code 8 appears nowhere. Total allowed is strictly below a naive sum over all rows whenever any claim in the sample was adjusted, and equal to it when none was.

Verify it yourself

• The distinct claim_id count in your filtered frame equals the count of distinct claim_ids whose surviving version is not a void.
• The line count you kept for each surviving claim equals that claim's line count at its max version, not 1.
• A naive sum over every row is greater than or equal to your total, with equality only when no claim has claim_version above 1.

Follow-up

• A claim has versions 1, 7, 7 where the second replacement has fewer lines than the first. What does your code return and is that correct?
• How would you handle a claim whose lines span two calendar months at the boundary of the reporting window?
• The same logic has to run over 400 million lines in a warehouse. What changes, and what stays the same?

Write a query using SQL window functions to calculate running totals and moving averages of clinical trial enrollment rates across multiple therapeutic areas.

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• What breaks if events arrive late or out of order?
• How would you verify this result without re-running the same query?

How would you handle and clean missing or duplicate records when joining disparate clinical data tables using advanced SQL operations?

Medium · sql and data manipulation

Approach

1. Say which table is the grain you start from, and join outward from it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Reconstruct drug coverage intervals and compute proportion of days covered

Hard · gaps and islands · window functions · adherence

pharmacy_claim holds fill_id, member_id, ndc_code, therapeutic_class_code, fill_date, days_supply, reversal_flag and reversed_fill_id. Early refills overlap, so days_supply must be laid end to end: each fill's coverage starts at the later of its fill_date and the previous interval's end. Both rows of a reversed pair are excluded. For one therapeutic_class_code over 2025-01-01 to 2025-06-30, return per member the proportion of days covered, defined as covered days divided by days from first fill_date to window end, plus the share of members at or above 0.80. Require at least two fills and 91 days of follow-up.

Approach

1. Clean the fills first. Drop rows with reversal_flag TRUE and also the fills they point at through reversed_fill_id. Dropping only the reversal row leaves an original fill counted as exposure that never reached the member. Use NOT EXISTS for that second step, since reversed_fill_id is nullable and NOT IN would return nothing.
2. Shift, never stack. The shifted end is covered_end_i = GREATEST(fill_date_i, covered_end_{i-1}) + days_supply_i, which looks recursive but has a closed form: with cum_i the running sum of days_supply through fill i, covered_end_i = cum_i + MAX over j <= i of (fill_date_j - cum_{j-1}). That is SUM(days_supply) OVER (ORDER BY fill_date ROWS UNBOUNDED PRECEDING) plus a running MAX of an anchor column, so it is two window functions and no recursive CTE.
3. Derive covered_start_i as GREATEST(fill_date_i, LAG(covered_end)). The intervals are disjoint by construction, so covered days is a plain sum of lengths and never needs a distinct day grid.
4. Clip intervals to the window end before summing, and decide explicitly whether to clip at disenrollment as well. Supply that runs past the window must not inflate the numerator.
5. Use the denominator the spec names, first fill_date to window end, and hold it constant across classes. A fixed-window denominator yields a different and usually lower number, and mixing the two across classes makes the comparison meaningless.
6. Apply the inclusion rules at member level after the intervals are built, then compute the share at or above 0.80 with the qualifying member count beside it.

Worked solution (40 min)

1. Filter to the therapeutic_class_code and window, then remove reversal rows and their originals with NOT EXISTS.
2. Per member ordered by fill_date, compute cum_supply and anchor = fill_date - (cum_supply - days_supply), then covered_end = cum_supply + MAX(anchor) OVER (ORDER BY fill_date ROWS UNBOUNDED PRECEDING).
3. Compute covered_start = GREATEST(fill_date, LAG(covered_end) OVER (ORDER BY fill_date)), clip both ends to the window, and drop intervals that collapse.
4. Sum interval lengths as covered_days, compute follow_up_days from first fill_date to window end, and divide.
5. Apply the two-fill and 91-day rules, then aggregate the share at or above 0.80.

Expected result: One row per qualifying member with covered_days, follow_up_days and pdc in [0, 1], plus a summary row with qualifying members and the share at or above 0.80. A member with 30-day fills on 2025-01-01, 2025-01-20 and 2025-04-01 is covered 2025-01-01 to 2025-03-01 and 2025-04-01 to 2025-04-30, giving 60 + 30 = 90 covered days over 181 follow-up days and a PDC of 0.497.

Verify it yourself

• No pdc exceeds 1.0 before any clipping. A value above 1 proves supplies were stacked instead of shifted.
• On a twenty-member sample, the summed interval lengths equal the distinct covered day count from a slow day-grid version.
• A member with one fill is excluded by the two-fill rule rather than appearing with a PDC of 1.0.
• Re-running with reversals left in raises covered days for exactly the members who have a reversed_fill_id, and for no one else.

Follow-up

• A member is hospitalised for twelve days mid-window, when the facility supplies medication. Should those days count as covered, and what do published adherence measures do about them?
• Someone wants to classify adherence over twelve months and then compare mortality from month zero. What is wrong with that design and what fixes it?
• Two different ndc_codes in the same therapeutic class overlap. Is that stacking or switching, and how does your interval logic treat each case?

How would you design a product metric framework to measure the operational efficiency and decision quality of a newly introduced clinical trial monitoring dashboard?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Fix the population and the time window before naming any metric.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

If engagement metrics for an internal R&D analytics platform suddenly decline week-over-week, what steps would you take to investigate?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would you approach designing a metric system to assess the reduction of compliance risks using predictive analytics models?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

If we are launching a new AI-powered knowledge assistant for R&D professionals, what core engagement and utility metrics would you track to determine its success?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How do you determine the required sample size and statistical power for an experiment when working with low-frequency biomedical events?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. State the primary metric and the minimum effect worth shipping, then size the test.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

What is the difference between statistical significance and practical significance in the context of large-scale R&D data analysis?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

Plan interim looks and multiplicity control for a year-long trial

Hard · sequential testing · alpha spending · multiple comparisons · novelty effects

A 12-month cluster-randomised care-management trial has one primary endpoint (ambulatory care sensitive admissions per 1,000 member-years), three secondary endpoints, and five pre-specified subgroups. The steering group wants a read every month. At month 2 the primary is nominally significant at p = 0.03 with an effect roughly three times the powered assumption. Write the monitoring and analysis plan that should have been in place, and say precisely what you tell the steering group about month 2.

Approach

1. Quantify the peeking cost rather than asserting it. Testing accumulating data repeatedly at a fixed nominal 0.05 raises the overall type I error to roughly 0.14 at five equally spaced looks and roughly 0.19 at ten; a monthly read for a year is worse than either.
2. Replace fixed-threshold looks with a Lan-DeMets alpha-spending plan on a pre-specified number of looks. An O'Brien-Fleming spending function is right here because it spends almost nothing early (a first look needs a nominal p on the order of 1e-5) and preserves close to 0.05 for the final analysis; Pocock's constant boundary (about 0.0158 nominal at each of five looks) buys early stopping power at the cost of a stricter final look. Monitor the safety guardrail on a separate, non-binding harm boundary, because stopping for harm is a different decision from stopping for efficacy.
3. Control the endpoint family with a gatekeeping structure: test the primary at full alpha, and only on success test the three secondaries in a pre-specified fixed sequence or with Holm. Declare the five subgroups exploratory, test them as treatment-by-subgroup interactions rather than as within-subgroup significance, and report them with FDR control and that label attached.
4. Separate the three distinct reasons the month-2 effect is inflated. Alpha inflation from the unplanned look is one. Early-exposure inflation is another, since the first clusters to go live are the enthusiastic ones and their staff are running a new process attentively, which decays as it becomes routine. Claims incompleteness is the third and is specific to this outcome: inpatient facility claims run out over months, so month 2 admission counts are undercounted, and the undercount does not have to be equal across arms.
5. Tell the steering group that month 2 is not a result: p = 0.03 is nowhere near any defensible first-look boundary, the recent incurred months are incomplete at the stated paid-through date, and estimates taken at a crossing are conditionally biased upward, so even a real stop would require a bias-adjusted effect estimate rather than the raw one.

Worked solution (40 min)

1. State the fixed number of interim analyses and the information fractions at which they occur, then pick the spending function and derive nominal boundaries so the total spend equals 0.05.
2. Write the gatekeeping rule for the three secondaries and the exploratory label plus FDR rule for the five subgroups.
3. Specify that the primary is computed only at a stated paid-through date, with the three most recent incurred months either held back or completion-factored.
4. Add an exposure-time analysis: index every cluster on its own go-live date and report the effect by cluster-months since activation, so habituation and staggered rollout are visible rather than pooled away.
5. Write the month-2 statement: the boundary was not crossed, the data are incomplete, and no effect estimate will be circulated.

Expected result: A Lan-DeMets alpha-spending plan with a pre-specified look schedule and an O'Brien-Fleming shape for the primary, a separate non-binding harm boundary for the guardrail, fixed-sequence or Holm gatekeeping across the three secondaries, and the five subgroups as FDR-controlled exploratory interaction tests. Month 2 is reported as not a result on three independent grounds: boundary not crossed, claims incomplete, and early estimates conditionally inflated.

Verify it yourself

• Confirm the naive repeated-testing error rate: about 0.14 at five looks and about 0.19 at ten, against a nominal 0.05.
• Confirm the subgroup family-wise rate without correction: 1 - 0.95^5 = 0.23 under independence.
• Confirm the chosen boundaries' cumulative spend equals the nominal 0.05.
• Confirm every reported figure carries a paid-through date and that the last three incurred months are held back or completion-factored.

Follow-up

• The steering group insists on monthly visibility. What can you show them every month that does not spend alpha?
• How would you tell a genuine effect apart from an exposure-time artefact when clusters went live on a staggered schedule?
• If the trial does cross an efficacy boundary at month 8, what effect estimate do you publish and why is it not the raw one?

Adherence share collapsed the month a feed was added

Easy · pdc · interval sql · identity resolution

The share of members with proportion of days covered at or above 0.80 in one therapeutic class fell from 0.71 to 0.58 in a single month. pharmacy_claim carries fill_id, member_id, ndc_code, therapeutic_class_code, fill_date, days_supply, quantity_dispensed, refill_number, reversal_flag, reversed_fill_id and source_system_code. There is no person-level key above member_id and no enrollment table on hand, so every diagnostic has to run on pharmacy_claim alone. That month the pipeline began ingesting a second pharmacy source. Determine whether adherence actually changed, and hand back a PDC query you would defend line by line.

Approach

1. Compute controls that depend on neither member identity nor the PDC code: total fills in the class per month, and total days_supply dispensed in the class per month. Both are sums over rows, so neither moves when one person's fills split across two member_ids, which is exactly what a per-member control cannot promise. If both are flat while PDC falls, no dispensing behaviour changed and the computation or the keys did.
2. Check days_supply for nulls and zeros by source_system_code. If the new source omits days_supply and the code coalesces null to zero, covered days vanish and PDC falls even though the fills are there.
3. Check for member identity splitting using the columns you actually have. There is no person key above member_id, so the split has to be found by its footprint: distinct member_id in the class jumping at the break while total fills stay flat, and a rise in the share of member_ids whose earliest fill for a given ndc_code carries refill_number above zero, which is a continuation history that lost its beginning. When one person is split, each id may fall below the two-fill denominator requirement and drop out, and the ids that do survive carry artificial gaps.
4. Check reversals. A reversed fill is two rows, the fill and a row with reversal_flag TRUE pointing at reversed_fill_id, and both must be excluded from exposure. If the new source expresses reversals differently and reversed_fill_id does not link, phantom coverage appears, which pushes PDC up rather than down, so use this to rule the cause in or out.
5. Verify the denominator and window are unchanged: members with at least two fills in the class and at least 91 days of follow-up, over a stated fixed 12-month window, never a mixture of window lengths.
6. Write the PDC query so overlapping refills shift the next interval's start to the later of fill_date and the prior interval's end, so a day is covered at most once, and cap at one day per calendar day.

Follow-up

• Why is PDC preferred to medication possession ratio when early refills are common?
• You want to compare adherent and non-adherent members on hospitalisation. What is wrong with classifying on a 12-month PDC and then following from day zero?
• A member switches to a different drug in the same class mid-year. How does your interval logic handle that, and is the class the right grain?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Tell me about a project where your initial technical approach failed or hit a major roadblock. How did you diagnose the issue and change course?

Medium · behavioural and stakeholder questions

Approach

1. Name the disagreement or constraint, and how you resolved it with evidence.
2. Close with what you would do differently, concretely.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• What would you do differently if you ran that project again?
• What did you decide not to do, and why?

Defend a null result against a programme sponsor

Medium · regression to the mean · difference-in-differences · stakeholder management

A care management programme enrolled the top 1 percent of members by prior-year allowed spend. The sponsor's deck shows allowed PMPM for enrollees falling 34 percent from the year before enrollment to the year after, and asks for budget to triple the programme. You rebuild the evaluation with a concurrent comparison group selected by the identical spend rule in the same period. The difference-in-differences estimate is a 3 percent reduction with a confidence interval spanning zero. Deliverable: how you present this, to whom, in what order, and what you propose next.

Approach

1. Name the probe: whether you can deliver a finding that costs somebody their programme without softening it into uselessness or creating an adversary who routes around you next time.
2. Lead with the mechanism, not the verdict. Show the comparison group's own unadjusted drop, which will be large, because a cohort selected on an extreme of the outcome regresses toward the mean whether or not anyone intervenes. The sponsor's 34 percent is mostly that, and it is a property of the selection rule rather than a criticism of their clinicians.
3. Give the sponsor the finding privately before it appears in any deck their leadership sees. Being surprised in a room is what turns a methods disagreement into a political one.
4. State the estimate with its interval and say what it rules out as well as what it fails to establish. A 3 percent point estimate whose interval crosses zero is not evidence of no effect; it is insufficient power to separate a modest effect from none, and those are different claims.
5. Arrive with a design rather than only an objection. Propose a regression discontinuity at the enrollment threshold if the rule is applied sharply, or a randomised rollout across the next wave of eligible members, and state the sample size needed to detect the effect size the sponsor believes in.

Follow-up

• The sponsor says withholding the programme from a comparison group is unethical. What do you propose instead?
• Leadership wants one number for the board next week. What do you give them?
• What result would change your mind and make you believe the 34 percent?

Make an honest case for your own analytical impact

Hard · impact attribution · counterfactual · self-assessment

You are asked to account for a year of your work. Constraint: you may only use measures that existed before you knew the outcome, and you must include one project that moved nothing. The portfolio contains a measure specification that changed how a contract settles, a predictive model that shipped, a self-serve dashboard, and an evaluation that killed a proposed programme. Deliverable: how you attribute value to each, which of them you claim credit for and which you do not, and the one number you would accept being judged on next year.

Approach

1. The probe is whether you can reason counterfactually about your own work, which is the same discipline this domain demands of you for programmes, applied where it is uncomfortable.
2. State the counterfactual for each item: what would have happened without you. A measure specification that changed how a contract settles has a dollar counterfactual you can compute. A dashboard's counterfactual is analyst hours, which is real and small, and attaching a revenue number to it is where these accounts usually turn into fiction.
3. Treat the killed programme as your strongest item and say why. Avoided spend is real value, it is attributable because the decision turned on your evaluation, and it is the one item nobody else will count on your behalf.
4. Report the shipped model the way you would report anyone's model: calibration in the subgroups that matter, adoption by the people it was built for, and whether any decision changed. A model that is live but ignored has zero impact, and saying so is the credibility test in this answer.
5. Name the failed project with its mechanism and its cost, then say what earlier signal you would watch to kill it sooner. Time to kill is a legitimate impact measure.
6. Choose next year's number so that it can go down. A metric that only ratchets upward is not a commitment.

Follow-up

• Two other people also claim the contract measure. How do you split it?
• Your model is live and used, but the decisions it informs would have been the same without it. Is that impact?
• What is the strongest argument that your year had less impact than you just described?