AstraZeneca Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at AstraZeneca, you occupy a critical position at the intersection of advanced analytics, life sciences, and global healthcare innovation. Your day-to-day work directly supports life-changing research, clinical trial optimization, and operational efficiency across a globally recognized pharmaceutical enterprise. By transforming complex biological, clinical, and operational datasets into actionable insights, you enable cross-functional teams to make data-driven decisions that accelerate the delivery of medicines to patients worldwide.

The boundary with engineering varies enough to be worth asking about directly. Some seats end at the analysis and the writeup; others carry feature pipelines, scheduling and alerting, which decides whether production data plumbing belongs in your prep at all.

Focus areas

• Minimise protected health information in every extract
• Build member-month denominators from overlapping coverage spans
• Separate censoring from events in time-to-event models

Overview

The scope of this role spans multiple high-impact problem spaces, including predictive modeling for clinical outcomes, operational data strategy optimization, and biomarker discovery pipelines. You will collaborate closely with statisticians, bioinformaticians, clinical operations specialists, and software engineers to design scalable analytical solutions. Whether you are building experimentation frameworks to evaluate digital health interventions or engineering feature pipelines from disparate data sources, your work helps shape how AstraZeneca leverages data to solve critical medical challenges.

Expect a fast-paced yet deeply collaborative environment where scientific rigor meets modern product thinking. Successful candidates thrive in bridging the gap between raw data and business value while maintaining strict adherence to ethical data standards and regulatory guidelines. If you are passionate about applying cutting-edge data science to advance human health, this role offers unmatched scale, complexity, and professional impact.

The AstraZeneca interview process

AstraZeneca candidates report 8 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

• Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
• Whether you name what you have not done instead of stretching to cover every line of the posting
• Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

• Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
• Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
• Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Online Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Technical and Managerial Rounds

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Assessment Center

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

• Whether the query you write matches the plan you just described
• What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
• Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

• Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
• Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
• Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Technical Deep-Dive

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interview

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

• Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
• Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
• Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

• For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
• Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
• Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Where candidates lose points

Immortal time in adherence, treatment, and enrolment definitions

Classifying members as adherent, treated, or programme-enrolled requires them to survive and stay covered long enough to accumulate the defining events. That guaranteed event-free interval is assigned to the exposed group, so the exposure looks protective for reasons that have nothing to do with the treatment. Adherence studies are the classic case: measuring 12-month proportion of days covered and then comparing mortality builds survival into the exposure definition. Use time-varying exposure or a landmark analysis with the classification window excluded from follow-up.

Treating clinical measurements as missing at random

A lab result, a vital sign, or a screening exists because someone ordered it, and ordering tracks suspicion of disease, visit frequency, and site workflow. Imputing the mean or dropping incomplete rows biases the population estimate and can flip the sign of an association, because the untested are systematically healthier or systematically disengaged. The presence indicator is often more predictive than the value, which is a warning sign rather than a feature win: a model that learns test ordering will not transfer to a site with different protocols.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Never asking what decision the analysis will inform

Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.

Questions & practice

How do you diagnose and correct for selection bias in observational health data studies?

Medium · statistics and probability

Approach

1. Quantify uncertainty explicitly rather than reporting a point estimate alone.
2. Say what the estimate is of, and over what population it generalises.
3. Sanity-check the answer against a simple bound or a simulated case.

Follow-up

• Which assumption here is most likely to be violated in practice?
• How would you explain this result to someone who does not know statistics?

What statistical tests would you apply to evaluate the distribution shift between training datasets and real-world clinical deployment data?

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Build a claims lag triangle and complete recent months

Medium · chain ladder · runout · pivot

medical_claim_line has claim_id, claim_version, frequency_code, service_start_date, paid_date, allowed_amount, claim_status. Using 24 fully mature incurred months, build a development triangle of allowed dollars by incurred month and payment lag in whole months, derive cumulative completion factors by chain ladder, then estimate ultimate allowed for the three most recent incurred months at a stated paid-through date. Deduplicate to surviving claim versions before building the triangle. Return incurred_month, paid_to_date, completion_factor, estimated_ultimate.

Approach

1. Deduplicate to surviving claim versions and keep paid lines first. A triangle built on all versions develops on adjustment churn rather than on payment timing, and replacements arrive late, so the distortion concentrates in exactly the tail you are trying to estimate.
2. Compute lag as a whole-month difference between incurred month and paid month, not as day difference divided by 30. Ragged month lengths otherwise shuffle identical claims between lag buckets depending on which month they fall in.
3. Pivot to incurred_month by lag, cumulate along lag, then form age-to-age factors as the ratio of the column L+1 total to the column L total, summed over incurred months mature enough to have both. Volume weighting is the chain ladder; a simple mean of per-month ratios lets one low-volume month dominate the factor.
4. Chain the age-to-age factors from each lag to ultimate and invert: completion factor at lag L is the reciprocal of the product of factors from L onward. Apply as estimated_ultimate = paid_to_date / completion_factor.
5. State the assumption you have just made, which is that the development pattern is stable. It is not, after a claims-system migration, a network change or a processing backlog, and the most recent month's factor is the least reliable because it rests on the fewest observations while carrying the largest adjustment.

Worked solution (35 min)

1. Deduplicate to surviving versions, drop voids, keep claim_status 'paid', and derive incurred_month from service_start_date and paid_month from paid_date.
2. lag = (paid_month.year - incurred_month.year) x 12 + (paid_month.month - incurred_month.month); drop negative lags and investigate them separately.
3. Pivot to a triangle of summed allowed_amount, cumulate along the lag axis, and mask cells past the paid-through date so no cell contains a future payment.
4. Compute volume-weighted age-to-age factors per lag over the mature rows, then chain them into cumulative completion factors.
5. Apply the factor for each recent month's current maturity to its paid_to_date and return the four columns.

Expected result: Three rows. Completion is well below 1 at short lag and approaches 1 in the tail; for a mixed medical book a one-month-old incurred month commonly sits far from complete, while a twelve-month-old one is close. Exact values are book-specific, so the reviewable property is monotonicity and the holdout test below, not the numbers themselves.

Verify it yourself

• Cumulative completion factors are non-decreasing in lag and converge toward 1 at the tail.
• Hold out one fully mature incurred month, apply the factors at each of its intermediate maturities, and confirm each estimate lands within a few percent of its known total.
• estimated_ultimate is greater than or equal to paid_to_date for every row.
• A pharmacy-only triangle completes far faster than an inpatient-facility-only triangle, which confirms the lag computation is reading real payment timing.

Follow-up

• A processing backlog means the last two months developed slower than history. What does your estimate do, and how would you detect that before reporting?
• Pharmacy, professional and inpatient facility develop on different clocks. How do you split the triangle, and what breaks if the service mix shifts?
• What happens to a PMPM series if someone reports an unadjusted recent month, and in which direction does the error point?

Write a SQL query using SQL window functions such as ROW_NUMBER and RANK to identify the top three clinical trial sites by patient enrollment volume per quarter.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. State the window function and its partition and ordering out loud before writing it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

How would you use a rolling average via SQL window frames to smooth out weekly variability in operational supply chain telemetry data?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Report site activation to first randomization without hiding non-enrollers

Medium · grain collapse · left join · censoring

study_site holds one row per (study_id, site_id): site_activation_date, which is NULL for a site selected but never activated, and site_status. study_subject_visit holds subject_visit_id, study_id, site_id, subject_id, randomization_ts (NULL for screen failures), visit_name, visit_number, actual_visit_date and visit_status, with one row per subject per protocol visit, so randomization_ts repeats across every visit of a randomized subject and a site that has screened nobody has no rows there at all. For one study_id and a stated data cut date, return per activated site the activation date, the first randomization_ts and the days between them, with activated sites that have randomized nobody still present. Add a study-level summary giving median days to first randomization and the share of activated sites with no randomization by day 90.

Approach

1. Take the site universe from study_site, never from study_subject_visit. The visit table can only contain sites that have screened at least one subject, so a universe built from DISTINCT site_id there omits exactly the sites this metric exists to expose. Filter study_site to the study and to site_activation_date IS NOT NULL, then LEFT JOIN the per-site first randomization.
2. Collapse the grain before aggregating anything on the visit side. The table repeats randomization_ts across every visit of a randomized subject, so any count taken at row grain is inflated by visits per subject. MIN(randomization_ts) per site is safe, COUNT is not.
3. State the date boundary rule. randomization_ts is a timestamp and site_activation_date is a date, so a randomization at 23:40 local time can land on either side of a day boundary depending on the zone the timestamp is stored in. Cast once, deliberately, and say which zone.
4. Use PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY days) for the median and be explicit that it ignores NULLs, so it is a median over enrolling sites only. That is exactly why the non-enroller share is reported beside it rather than instead of it.
5. Censor the day-90 denominator against the data cut. A site activated 40 days before the cut has not had 90 days to randomize, so it cannot be scored on that bar and belongs outside that denominator.

Worked solution (25 min)

1. CTE sites: site_id and site_activation_date from study_site for the study where site_activation_date IS NOT NULL. The (study_id, site_id) key already gives one activation date per site, so assert that key rather than de-duplicating defensively.
2. CTE first_rand: site_id and MIN(randomization_ts) over study_subject_visit rows for the study where randomization_ts IS NOT NULL, grouped by site_id.
3. LEFT JOIN sites to first_rand and compute days_to_first_randomization with an explicit cast of the timestamp to a date.
4. Summary: PERCENTILE_CONT(0.5) over non-null days, plus the count and share of sites with no randomization by day 90, restricted to sites activated at least 90 days before the data cut.
5. Compare the site-level row count with COUNT(DISTINCT site_id) in study_subject_visit for the study; the difference is the activated sites that have screened nobody, which is the population the metric is for.

Expected result: One row per activated site with site_id, site_activation_date, first_randomization_ts and days_to_first_randomization, NULL in both randomization columns for sites that have randomized nobody, including sites with no study_subject_visit rows at all; plus one summary row with median_days_over_enrolling_sites, sites_activated_90d_or_more and share_with_no_randomization_by_day_90.

Verify it yourself

• Site-level row count equals the number of study_site rows for the study with a non-null site_activation_date, and is greater than or equal to COUNT(DISTINCT site_id) in study_subject_visit for that study. Below it means either an activated site is missing from study_site or a visit row carries a site_id whose activation date is NULL, both data-quality findings.
• No days_to_first_randomization is negative. A negative value is a data-quality finding to raise, not a number to clamp at zero.
• The day-90 denominator excludes sites activated within 90 days of the data cut. Including them can only add sites that have not had a full 90 days, and any of those without a randomization would be miscounted as a failure.

Follow-up

• The same site_id appears under two studies. What breaks, and what is the correct grain for this metric?
• Would you present this as a median, as a Kaplan-Meier curve of time to first randomization, or both, and what does the curve show that the median cannot?
• A site randomized one subject who withdrew at visit 2. Does that site count as enrolling here, and how does that differ from a randomized-and-retained measure?

How do you isolate seasonality from underlying product degradation when evaluating monthly active user metrics?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Fix the population and the time window before naming any metric.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Propose a metric to quantify the efficiency gains of an automated data strategy workflow in research and development.

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Walk me through your step-by-step methodology for investigating a sudden 15 percent drop in patient platform engagement.

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

How do you balance competing product metrics when optimizing a data pipeline that affects both patient retention and operational speed?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

Walk through how you would design an A/B testing framework for rolling out a new clinical decision-support dashboard.

Medium · a/b testing and experimentation

Approach

1. Name the randomisation unit first; it decides the variance and what the test can detect.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

What are the most common experimentation pitfalls you look out for when analyzing a multi-variant trial in a healthcare setting?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• How would you handle interference between treated and control units?
• What would you do if you could not randomise at all?

Set the decision rule when cost and denial guardrails conflict

Medium · guardrail conflict · denials · non-inferiority bound · audit sampling

Utilization management is asked to cut risk-adjusted allowed PMPM, and its lever is prior authorization and first-pass denial on advanced imaging and specialty drugs. The scorecard guardrail is first-pass denial rate from medical_claim_line (claim_status, denial_reason_code, claim_version, received_date) paired with the overturn rate on appealed denials. The two move in opposite directions by construction. Write the decision rule: which metric binds, at what pre-declared threshold, who adjudicates a breach, and what happens on breach. Do not resolve the conflict by weighting both into one score.

Approach

1. Say why a composite is the wrong instrument: the weights get chosen after someone has seen which way the quarter went, and any weighted sum lets a large enough cost win purchase an unbounded increase in wrong denials. A guardrail is a constraint, not a term in an objective.
2. Sharpen what the guardrail is actually protecting. A higher denial rate is not by itself a harm, since some denied claims should be denied. The harm is denials that should not have been made, so split first-pass denials by denial_reason_code into administratively correctable ones (eligibility, coding, missing documentation) and clinical-necessity ones, and put the bound only on the clinical group.
3. Fix the measurement of the guardrail before setting its threshold: count the denominator at claim_version 1 so replacements (frequency_code 7) and voids (8) are not double counted, hold the month back for 60 days of adjudication runout, and report by received_date.
4. Get the direction of the appeal-selection bias right. Appeals are filed on the denials the appellant already believes are wrong, by the members and providers with the staffing, literacy and dollar threshold to file at all, and an appeal usually arrives carrying documentation the first adjudication never had. The overturn rate on appealed denials therefore runs well above the wrong-denial rate across all denials, often by a multiple, and it moves with filing behaviour as much as with adjudication quality. The unbiased population estimate is a blind re-review of a random sample of all denied clinical-necessity lines each quarter, judged on the record as it stood at first adjudication; the appeal series is a fast indicator read against its own history between audits, not a second estimate of the same quantity.
5. Write the rule as a bound with a consequence: risk-adjusted allowed PMPM is the primary and binds only while the audited wrong-denial rate stays within a pre-declared absolute margin of its trailing four-quarter mean. Breach stops the program's expansion rather than discounting its score, and the margin is set before the quarter opens.
6. Name the adjudicator and the cadence: a clinical owner outside the utilization function, reviewing at the stated paid-through date, with the breach consequence written down before any result is known.

Worked solution (25 min)

1. Restate the guardrail metric properly: numerator is claim lines denied at first adjudication, denominator is lines received in the month counted at claim_version 1, held 60 days, broken out by denial_reason_code group.
2. Split denial_reason_code into administratively correctable and clinical-necessity groups and say which function owns each, since the two have different fixes and different harms.
3. Write the appeal-selection argument in two sentences, including which way it runs, and specify the random blind re-review: the frame it is drawn from, namely all denied clinical-necessity lines rather than appealed ones, plus sample size, who reviews, what record they are shown, blinding, and quarterly cadence.
4. State the bound as an absolute margin on the audited wrong-denial rate against its trailing four-quarter mean, with the margin fixed before the period.
5. Write the breach consequence and the named adjudicator, and note that both are recorded before results exist.

Expected result: A written rule in which risk-adjusted allowed PMPM is primary; the binding guardrail is the audited wrong-denial rate on clinical-necessity denials, estimated from a blind random re-review of all denied clinical-necessity lines, with the appeal overturn rate kept as a between-audit indicator that runs above the audited rate and never sets the bound; the bound is a pre-declared absolute margin against a trailing four-quarter mean; a clinical owner outside utilization management adjudicates; and a breach halts expansion rather than adjusting a score.

Verify it yourself

• Recount the denominator with and without version dedup. If the two differ by more than a percent or two, the incumbent number has been counting resubmissions.
• Re-run a closed month at two extract dates 60 days apart. A moving denial rate means the runout hold is too short.
• Compare the audited wrong-denial rate to the appeal overturn rate for the same quarter. The overturn rate should be the higher of the two, usually by a multiple, because appeals are filed on the denials most likely to be wrong and often add documentation the first adjudication never saw. An audit rate at or above the overturn rate is the anomaly: check that the sample was drawn from all denied clinical-necessity lines rather than only appealed ones, and that the reviewers were blinded and shown the record as it stood at first adjudication.
• Confirm the margin, the adjudicator and the consequence are all recorded with a timestamp before the period's data was available.

Follow-up

• Denials rose and the overturn rate fell. Give the benign reading and the bad reading, and the one number that separates them.
• What happens to your bound if the appeal filing rate per denial drops 30 percent in the same quarter?
• How large a random audit sample do you need to detect a 2 percentage point rise in wrong denials off a 6 percent base, and what does that cost in clinician review hours?

Glycemic control share jumped when a feed connected

Medium · informative missingness · unit mismatch · metric specification

The share of members with a controlled glycemic result moved from 0.62 to 0.81 the month a second result feed was connected. lab_result carries result_id, order_id, patient_id, encounter_id, loinc_code, analyte_name, value_numeric, units, reference_low, reference_high, abnormal_flag, result_status and supersedes_result_id. The metric is members whose latest result in the year is below the control threshold, divided by members with any result in the year. Decide whether control actually improved, and specify the metric so results from both feeds can be pooled.

Approach

1. Split numerator and denominator. The denominator is members with any result, so a feed that tests more people changes the metric without changing anyone's health. Count denominator members before and after and see how much of the 19 point move is denominator growth.
2. Recompute on a fixed cohort: members who had a qualifying result in both the before and after periods. If the share is flat on the fixed cohort, the movement is who got tested, not what their results were.
3. Deduplicate to the latest non-cancelled row per (order_id, loinc_code), respecting supersedes_result_id, because a corrected result arrives as a new row and counting both gives one person two values.
4. Audit units and loinc_code mix by feed. Two unit conventions coexist for this analyte, percent and mmol/mol, related by mmol/mol equal to 10.929 times (percent minus 2.15), so 7.0 percent is 53 mmol/mol. Comparing mmol/mol values to a percent threshold would classify them all as uncontrolled and depress the share, so a rise cannot be explained by that mismatch, which rules it out rather than in.
5. State plainly that the tested population is not the covered population. Testing is ordered when someone suspects a problem or when a site runs standing orders, so the mean among tested members is not the population mean and cannot be made into one by imputation.
6. Respecify: numerator and denominator both restricted to members with a stated continuous-enrolment requirement, one deduplication rule, one unit after conversion, one threshold, and a separately reported testing rate so the denominator's movement is visible.

Follow-up

• Someone proposes imputing the mean result for untested members. What happens to the estimate and why?
• A model you are building gains AUC from a feature that marks whether the test exists. Why is that a warning rather than a win?
• How would you report this metric so a clinical committee can see improvement separately from coverage of testing?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Your part, not the team's

Most data work is done by groups, so an interviewer has to work out which piece was yours. An answer that runs on 'we' for several minutes gets interrupted with a question about what you personally did, and by then the answer sounds defensive even when it is true. Mark your own contribution as you go, and name the parts that belonged to someone else instead of leaving them ambiguous. Keep a few specifics back as well, like the name of the metric or who actually objected, so a probe can be answered with something you had not already said.

Describe a situation where your analysis conflicted with a senior stakeholder's intuition. How did you resolve the disagreement?

Medium · behavioural and stakeholder questions

Approach

1. Close with what you would do differently, concretely.
2. Name the disagreement or constraint, and how you resolved it with evidence.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that project again?

Make an honest case for your own analytical impact

Hard · impact attribution · counterfactual · self-assessment

You are asked to account for a year of your work. Constraint: you may only use measures that existed before you knew the outcome, and you must include one project that moved nothing. The portfolio contains a measure specification that changed how a contract settles, a predictive model that shipped, a self-serve dashboard, and an evaluation that killed a proposed programme. Deliverable: how you attribute value to each, which of them you claim credit for and which you do not, and the one number you would accept being judged on next year.

Approach

1. The probe is whether you can reason counterfactually about your own work, which is the same discipline this domain demands of you for programmes, applied where it is uncomfortable.
2. State the counterfactual for each item: what would have happened without you. A measure specification that changed how a contract settles has a dollar counterfactual you can compute. A dashboard's counterfactual is analyst hours, which is real and small, and attaching a revenue number to it is where these accounts usually turn into fiction.
3. Treat the killed programme as your strongest item and say why. Avoided spend is real value, it is attributable because the decision turned on your evaluation, and it is the one item nobody else will count on your behalf.
4. Report the shipped model the way you would report anyone's model: calibration in the subgroups that matter, adoption by the people it was built for, and whether any decision changed. A model that is live but ignored has zero impact, and saying so is the credibility test in this answer.
5. Name the failed project with its mechanism and its cost, then say what earlier signal you would watch to kill it sooner. Time to kill is a legitimate impact measure.
6. Choose next year's number so that it can go down. A metric that only ratchets upward is not a commitment.

Follow-up

• Two other people also claim the contract measure. How do you split it?
• Your model is live and used, but the decisions it informs would have been the same without it. Is that impact?
• What is the strongest argument that your year had less impact than you just described?

Push back on an over-broad extract without blocking the work

Hard · minimum necessary · de-identification · governance

A researcher asks for a one-off extract covering 12,000 members: member_id, date of birth, five-digit residential postal code, full lab_result rows including value_text, and clinical note text, held on their laptop for a retrospective analysis of one lab analyte over time. They describe it as de-identified because names are excluded. A flat refusal is not an acceptable answer. Deliverable: the column list you will actually provide with a justification per retained field, the sentence explaining why free text defeats an enumerated-identifier approach, your small-cell rule for the published stratification, and what you deliver this week.

Approach

1. The probe is whether privacy is a design skill for you or a compliance reflex. Flat refusal and unquestioned fulfilment both fail, for opposite reasons.
2. Reconstruct the analysis from the research question rather than from the request. A longitudinal trend in one analyte needs a stable pseudonymous subject key, the analyte's loinc_code, value_numeric, units, the reference bounds, and a date. It does not need names, notes, or a precise birth date.
3. Replace each identifier with the least precise version that still answers the question: a salted per-study pseudonym instead of member_id, age in years or an age band instead of date of birth, a consistent per-subject date offset if only intervals matter, and a coarser geography than five-digit postal code.
4. Explain the free-text problem in one sentence. Removing an enumerated list of identifier fields does not de-identify a note, because notes carry names, relationships, employers, dates and rare clinical detail in prose, so only a reviewed extraction into structured fields or a formal statistical determination addresses it.
5. Set the publication rule before the analysis runs. Suppress cells below a stated threshold and suppress complements too, because a reader holding a row total and all but one cell recovers the suppressed count by subtraction.
6. Keep them moving. Deliver the reduced extract into a controlled environment this week rather than onto a laptop, and offer to derive any note-based variable yourself as a structured flag so they never need the prose.

Follow-up

• They say the age band loses resolution their model needs. What do you do?
• How do you set the suppression threshold, and who signs off on it?
• The analysis later needs linking to a second data source. What changes?