cswork / libraryOpen practice workspace

Interview guides · Children's Hospital of Philadelphia

Children's Hospital of Philadelphia Data Scientist Interview Questions & Guide 2026

Children's Hospital of Philadelphia data scientist prep: probability, sampling distributions, and stating the assumption behind every test you run.

The 60-second brief

As a Data Scientist at Children's Hospital of Philadelphia (CHOP), you are stepping into a role where your technical expertise directly impacts pediatric healthcare, clinical research, and operational excellence. CHOP is a premier pediatric research hospital, which means our data teams do not just optimize metrics; they uncover insights that can save lives, improve patient outcomes, and drive forward groundbreaking medical research.

How much statistics you need depends on the flavour of the seat. Experiment-facing work wants you deep enough to notice that repeated looks at accumulating data inflate the false positive rate of a fixed-sample test; modelling-facing work wants estimation and honest uncertainty intervals.

Focus areas

  • Correct for claims runout before reporting recent months
  • Minimise protected health information in every extract
  • Build member-month denominators from overlapping coverage spans

Overview

In this position, you will operate at the intersection of advanced analytics, machine learning, and clinical application. You will work closely with a diverse group of stakeholders, including world-renowned clinical faculty, medical researchers, and hospital administration staff. Your work will span everything from predictive modeling for patient deterioration to optimizing hospital resource allocation and supporting large-scale genomic or epidemiological studies.

What makes this role uniquely challenging and rewarding is the complexity of the data and the audience you serve. You are not just building models in a vacuum; you are translating complex, often messy clinical data into actionable insights for medical professionals. You must be as comfortable presenting your research to a room of doctors as you are writing efficient and code to clean electronic health records. Expect a highly collaborative, mission-driven environment where rigor, accuracy, and clear communication are paramount.

The Children's Hospital of Philadelphia interview process

Children's Hospital of Philadelphia candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

  • Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
  • Whether your reason for wanting the role points at the work itself rather than the company's reputation
  • Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

  • Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
  • Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
  • Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Conversational Interview

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

  • Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
  • Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
  • Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

  • For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
  • Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
  • For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Technical Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

  • Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
  • Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
  • Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

  • Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
  • Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
  • If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Final Interview

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

  • Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
  • Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
  • Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

  • Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
  • Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
  • Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
  • Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Where candidates lose points

Rates built on member counts rather than exposure

Members join and leave mid-period, so dividing events by distinct members mixes a person covered for 30 days with one covered for 365. New joiners also have artificially low observed utilisation because their claims have not arrived yet and because care takes time to initiate. Denominators must be member-months or member-years, and comparative quality measures usually need a continuous-enrolment requirement with an explicit allowable gap, stated in days.

Pre-post evaluation on a high-cost or high-risk cohort

Cohorts selected on an extreme value of the outcome regress toward the mean on their own. Members identified as the top 1 percent of spend in one year spend far less the next, whether or not anyone intervenes, because the selecting year captured both chronic severity and one-off events. A pre-post design on such a cohort will report savings every time. A concurrent comparison group selected by the same rule in the same period, or a regression discontinuity at the selection threshold, is the minimum credible design.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Questions & practice

Explain the bias-variance tradeoff as if you were speaking to a hospital administrator.

Medium · statistics and probability

Approach

  1. Say what the estimate is of, and over what population it generalises.
  2. Write down the assumption the method needs before you use the method.
  3. Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

  • What sample size would you need to detect an effect half this size?
  • How would you explain this result to someone who does not know statistics?

In your previous research, how did you decide which metrics to use to evaluate your model's success?

Medium · machine learning and modelling

Approach

  1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
  2. Say how the offline result would be validated online before it is trusted.
  3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

  • How would you choose the decision threshold, and who owns that choice?
  • Where could label leakage enter this setup?

If you are building a model to predict a rare pediatric disease, how would you handle the extreme class imbalance in your training data?

Medium · machine learning and modelling

Approach

  1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
  2. Pick an evaluation metric that matches the cost of each error type, not a default.
  3. Say how the offline result would be validated online before it is trusted.

Follow-up

  • Where could label leakage enter this setup?
  • How would you choose the decision threshold, and who owns that choice?

Simulate how ignoring clustering inflates the false positive rate

Medium · simulation · design effect · type i error

Simulate a provider-level comparison with no true effect. 40 providers, 60 members each, a continuous outcome with total variance 1 and intraclass correlation 0.02. Randomise providers 20 to 20, not members. Over 5,000 replications, report the share of replications where a two-sample t-test run on all 2,400 individual observations, ignoring provider, returns p below 0.05. Report the same share when the unit of analysis is the 40 provider means. Then state how the first number follows from the design effect 1 + (m - 1) x ICC.

Approach

  1. Decompose the variance explicitly rather than tuning it: provider variance = ICC = 0.02, residual variance = 1 - ICC = 0.98. Draw u_j once per provider and e_ij per member, so the outcome is u_j + e_ij and the correlation between two members of one provider is 0.02 by construction.
  2. Randomise at the provider level. That is the entire mechanism: the treatment indicator is constant within a provider, so in any single replicate the provider intercepts are confounded with the arm, and the naive test reads that confounding as signal.
  3. Vectorise over replications with a (reps, providers, members) array. 5,000 x 2,400 normals is trivial in numpy, and a Python loop is what tempts people to cut replications to 500 and report a number with Monte Carlo noise of plus or minus 0.017.
  4. Predict the answer before running it: design effect = 1 + (60 - 1) x 0.02 = 2.18, naive standard errors are too small by sqrt(2.18) = 1.476, so the rejection rate at nominal 0.05 is about 2 x (1 - Phi(1.96 / 1.476)) = 0.18. Agreement between prediction and simulation is what makes the result a demonstration rather than an anecdote.
  5. Run the provider-mean analysis as the control. Recovering 0.05 there proves the generator is correct and isolates the failure to the analysis unit.

Worked solution (30 min)

  1. Draw u with shape (reps, 40) from N(0, sqrt(0.02)) and e with shape (reps, 40, 60) from N(0, sqrt(0.98)); y = u[:, :, None] + e.
  2. Assign providers 0-19 to one arm and 20-39 to the other; because there is no true effect, a fixed split is valid and removes one source of Monte Carlo noise.
  3. Naive test: flatten each arm to 1,200 observations and compute a two-sample t statistic per replication with vectorised means and pooled variance.
  4. Cluster test: average within provider to 40 values, then run the same two-sample t on 20 versus 20.
  5. Report both rejection shares and compare the naive one against 2 x (1 - Phi(1.96 / sqrt(2.18))).

Expected result: The naive individual-level test rejects in roughly 18 percent of replications, against a nominal 5 percent. Monte Carlo standard error at 5,000 replications is about 0.005, so anything in 0.17 to 0.19 is consistent. The provider-mean analysis rejects in about 5 percent, within roughly 0.003.

Verify it yourself

  • Set ICC to 0 and both analyses return about 0.05. If the naive one does not, the data generator is wrong, not the test.
  • Raise members per provider from 60 to 120 with ICC fixed and the naive rate rises, because the design effect grows linearly in cluster size.
  • The provider-mean analysis stays near 0.05 at every ICC you try, which is the definition of a valid test.
  • The simulated naive rate lands within about 0.01 of 2 x (1 - Phi(1.96 / sqrt(design effect))).

Follow-up

  • How many providers would you need for 80 percent power on a 0.2 SD difference with 60 members each and this ICC?
  • Cluster sizes are unequal in reality. The design effect approximation becomes 1 + ((1 + CV^2) x mbar - 1) x ICC. Which direction does that move your sample size and why?
  • The intervention cannot be withheld from any provider. Sketch a design that still yields a defensible estimate.

Write a SQL query to identify the top 5 diagnoses for patients admitted in the last year.

Medium · sql and data manipulation

Approach

  1. State the window function and its partition and ordering out loud before writing it.
  2. Compute rates by summing numerator and denominator separately, never by averaging rates.
  3. Say which table is the grain you start from, and join outward from it.

Follow-up

  • How does the query change if the join becomes one-to-many?
  • What breaks if events arrive late or out of order?

Describe how you would use Python to handle missing values in a dataset where 30% of the entries for a key vital sign are blank.

Medium · sql and data manipulation

Approach

  1. Compute rates by summing numerator and denominator separately, never by averaging rates.
  2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
  3. Say which table is the grain you start from, and join outward from it.

Follow-up

  • What breaks if events arrive late or out of order?
  • How does the query change if the join becomes one-to-many?

How do you optimize a SQL query that is running too slowly on a massive patient database?

Medium · sql and data manipulation

Approach

  1. Say which table is the grain you start from, and join outward from it.
  2. Check whether any join is one-to-many before aggregating, or the sums inflate.
  3. State the window function and its partition and ordering out loud before writing it.

Follow-up

  • How does the query change if the join becomes one-to-many?
  • How would you verify this result without re-running the same query?

Compute allowed PMPM by incurred month without span fan-out

Hard · fan-out · many-to-many · runout

Report allowed PMPM by incurred month and product_type for 2025 from medical_claim_line (member_id, claim_id, claim_version, frequency_code, service_start_date, allowed_amount, claim_status), pharmacy_claim (member_id, fill_date, allowed_amount, reversal_flag, reversed_fill_id), member_enrollment (span_id, member_id, product_type, effective_date, termination_date) and member_risk_score (member_id, score_year, prospective_risk_score), which holds at most one row per member per score year. The numerator is allowed_amount on surviving claim versions plus unreversed fills, attributed to the product_type in force on the service or fill date. The denominator is member-months. Also return a risk-adjusted PMPM for each cell: allowed PMPM divided by the member-month-weighted mean prospective_risk_score for score_year 2025 over the members making up that cell's denominator. A colleague joined claim lines to member_enrollment on member_id alone and the annual total came back 1.4 times the finance figure. Diagnose that, then produce the correct monthly series.

Approach

  1. Name the defect precisely. member_enrollment is one row per span, so joining on member_id alone is one-to-many and duplicates every claim line once per span that member ever held. The inflation factor is the spend-weighted mean span count per member, which is why 1.4 is plausible and why it is not a constant you can divide back out.
  2. Make the join functional: member_id AND service date between effective_date and COALESCE(termination_date, DATE '9999-12-31'). That yields at most one span per line unless spans genuinely overlap, in which case pick a documented precedence rule, most recently effective wins being the usual one, and prove uniqueness with a duplicate check rather than asserting it.
  3. Aggregate numerator and denominator separately and join them on (month, product_type). Computing both sides across one join is the second route to inflation, because the member-month denominator fans out over claim lines just as easily as the dollars fan out over spans.
  4. Clean each numerator source on its own terms: surviving claim versions with voids removed and claim_status 'paid' on the medical side, reversal rows and their originals removed on the pharmacy side.
  5. Apply completion before anyone reads a trend. State the paid-through date and either withhold the three most recent incurred months or apply completion factors, since the lag differs by service type and the incomplete tail reads as a cost improvement and a utilisation drop at once.
  6. Risk-adjust last, and weight the score the way the denominator is weighted. The divisor for a cell is SUM(member_months * prospective_risk_score) / SUM(member_months) over that cell's members, not a plain AVG over distinct members: a member covered two months must not count the same as one covered twelve. member_risk_score is member-level, so join it after the member-month grain is fixed, or it becomes a second fan-out.
  7. State the rule for members with no 2025 member_risk_score row: keep them in member_months, leave them out of the weighted mean, and publish the scored share of member-months beside the adjusted series. A thin score table moves the adjusted number while the raw one sits still, and without that share nobody can tell the two apart.

Worked solution (45 min)

  1. CTE med: surviving claim version per claim_id, voids dropped, claim_status 'paid', service_start_date in 2025.
  2. CTE rx: fills with reversal_flag FALSE and not referenced by any reversed_fill_id, via NOT EXISTS, fill_date in 2025.
  3. Attribute each numerator row to one span with the date-bounded join, after asserting that row counts before and after the join are equal.
  4. CTE denom: member-months by month, product_type and member_id from spans alone, with fractional partial months and overlapping spans collapsed first.
  5. Join numerator to denominator aggregated on (incurred_month, product_type) and divide for allowed_pmpm.
  6. Join denom to member_risk_score on member_id for score_year 2025, compute the cell divisor as SUM(member_months * prospective_risk_score) / SUM(member_months) over scored members, carry the scored share of member-months, and divide allowed_pmpm by the divisor for risk_adjusted_pmpm.
  7. Withhold the three most recent incurred months and label the paid-through date on every row.

Expected result: One row per (incurred_month, product_type) with allowed_medical, allowed_pharmacy, member_months, allowed_pmpm, mean_risk_score, scored_member_month_share and risk_adjusted_pmpm, covering January through September when the paid-through date is year end and three months are withheld. Row count of the claim-line set is identical before and after the enrollment join.

Verify it yourself

  • Claim-line row count before and after the span join is unchanged. An increase is fan-out; a decrease is a service date with no covering span, which is a real finding about eligibility gaps.
  • SUM(allowed_amount) across product_types for a month equals the ungrouped monthly total computed without touching member_enrollment at all.
  • member_months for a month never exceeds the count of distinct members covered at any point in that month.
  • mean_risk_score for a cell lies between the minimum and maximum prospective_risk_score among its scored members, and risk_adjusted_pmpm equals allowed_pmpm exactly when that mean is 1.0. If the weighted mean equals a plain AVG over distinct scored members in every cell, the member-month weights were never applied.
  • A downward step in the last reported months means the runout holdback was not applied; the withheld tail should not be visible in the series at all.

Follow-up

  • A member holds two overlapping spans under different product_types on the service date. Which one gets the dollar, and how do you keep the member-month denominator consistent with that choice?
  • Write the check that proves the span join is one-to-one rather than assuming it.
  • Finance reports on paid_date and you reported on service_start_date. Reconcile one month between the two bases.

How do you prioritize your time when working on multiple research initiatives with competing deadlines?

Medium · product sense and metrics

Approach

  1. Name one primary metric, then the guardrail that stops it being gamed.
  2. Restate the decision this analysis has to support, and who acts on the answer.
  3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

  • Which segment would you cut first, and what would that rule out?
  • What would you do if the primary metric and the guardrail moved in opposite directions?

Walk me through a recent data science project you led from conception to deployment.

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the decision backwards to the evidence you would need.
  3. Clarify what is being asked and what a complete answer would contain.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Evaluate a book-wide benefit change with no untreated members

Hard · difference-in-differences · synthetic control · parallel trends · seasonality confounding

On 1 January a therapeutic class moves from formulary tier 2 to tier 3, raising member cost share, for the entire commercial group book at once. No untreated subset of that book exists. You have 36 months of pharmacy_claim and member_enrollment covering that book plus the Medicare Advantage and Medicaid managed books, which are unaffected. Outcomes are class fills per 1,000 member-months and the share of members with PDC at or above 0.80. Give the design, the identifying assumption, and how you would test it.

Approach

  1. Rule out the single-series interrupted time series first and say why. The change lands on 1 January, which is also when deductibles reset, benefit designs change, and members switch plans, so a level shift at the new year is confounded with the entire annual benefit cycle. Any design that cannot separate those two is not usable here.
  2. Build an event-study difference-in-differences with the unaffected product books as controls, estimating month-by-month coefficients relative to December. The lead coefficients are the test of the identifying assumption, not decoration: they should be indistinguishable from zero, and you should say in advance what magnitude of lead violation would make you abandon the design.
  3. State what the control books are doing for you and what they are not. They experience the same January reset, which is precisely why they are useful, but they differ in population age, cost-sharing structure, and eligibility churn, so parallel counterfactual trends is a real assumption and residual confounding should be named rather than waved at.
  4. Add a synthetic control as the robustness check: construct a weighted combination of donor series (other product books crossed with other therapeutic classes) that tracks the treated series through the pre-period, and do inference by permutation, assigning the intervention to each donor and comparing post-to-pre RMSPE ratios. Require that the treated pre-period sits inside the convex hull of the donors, and abandon the approach if pre-period fit is poor.
  5. Fix the population before estimating anything. If the tier move pushes members out of the book, the post-period population differs from the pre-period one, so restrict to members continuously enrolled across the window with a stated allowable gap, and report sensitivity to that gap. Note that pharmacy claims adjudicate in days, so unlike a medical-cost series this outcome is essentially complete within weeks.

Worked solution (45 min)

  1. Build a monthly panel of fills per 1,000 member-months and PDC share by product book, restricted to members continuously enrolled across the window with a 45-day allowable gap.
  2. Plot the treated and control series for 24 pre-period months and inspect visually before fitting anything.
  3. Estimate an event-study DiD with month-relative-to-January coefficients, product fixed effects, and calendar-month fixed effects, clustering standard errors at the product-book level or higher.
  4. Run the placebo: re-estimate as if the change happened on the prior 1 January, which isolates the annual reset from the tier move.
  5. Fit a synthetic control on donor product-class series, check pre-period RMSPE, and run the permutation inference across donors.

Expected result: An event-study difference-in-differences of the affected commercial book against unaffected product books on continuously enrolled members, with a synthetic control and permutation inference as the robustness check. The identifying assumption is parallel counterfactual trends, tested by null lead coefficients and by a prior-January placebo that separates the tier change from the deductible reset.

Verify it yourself

  • Lead coefficients in the event study should be within noise of zero across the pre-period.
  • The prior-January placebo should return an effect near zero; if it does not, the January reset is contaminating the estimate.
  • Synthetic control pre-period RMSPE should be small, and the treated unit's post-to-pre RMSPE ratio should be extreme relative to the donor distribution.
  • The estimate should be stable when the continuous-enrolment gap allowance moves from 0 to 45 days.

Follow-up

  • The lead coefficients are not flat: the treated book was already drifting down for six months. What do you do without abandoning the question?
  • Therapeutic substitution is expected. How do you tell a true reduction in therapy from members moving to a cheaper drug in the same class?
  • How would you report the result so a clinical committee can act on it, given that fills per 1,000 member-months is not an outcome anyone treats?

Protocol deviations fell as new sites came online

Hard · reporting lag · denominator mix · clustering

Protocol deviations per 100 completed visits across a multi-site study fell from 4.6 to 2.9 over two months while randomisation accelerated. study_subject_visit carries study_id, site_id, subject_id, visit_number, visit_name, planned_visit_date, actual_visit_date, visit_window_days, visit_status, protocol_deviation_flag, deviation_severity, data_entry_ts and site_activation_date. Operations wants to credit a retraining rollout that began in that window. Establish whether deviation behaviour changed, and deliver a version of the series that is safe to review monthly.

Approach

  1. Treat deviation flags as late-arriving data. Build a lag triangle of data_entry_ts minus actual_visit_date, because deviations are largely identified during monitoring review weeks after the visit, so the two most recent months are structurally undercounted in exactly the way an improvement looks.
  2. Hold back the months that are not developed, or apply development factors fitted on fully developed months, and restate. Do this before any explanation involving training.
  3. Attack the denominator. It counts visits with visit_status 'completed', while out_of_window and missed visits are excluded even though out-of-window attendance is itself a deviation in most protocols, so a site that pushes visits out of window lowers the rate twice over.
  4. Standardise the visit mix. Newly activated sites contribute mostly screening and baseline visits, which carry fewer procedures and fewer opportunities to deviate than later treatment visits, so compute the rate within visit_number strata and reweight to a fixed mix before comparing months.
  5. Split major from minor deviation_severity and report them separately. A single major deviation can remove a subject from the per-protocol analysis set while ten minor ones may not, so a pooled rate can fall while the consequential series rises.
  6. Before attributing anything to training, model at the right level: subjects are nested in sites, so use a random intercept per site or cluster-robust standard errors. With average cluster size m and intracluster correlation rho the variance is understated by the design effect 1 + (m - 1) times rho, which at m = 60 and rho = 0.02 is about 2.2, so naive intervals are roughly 1.5 times too narrow.

Follow-up

  • Design a defensible evaluation of the retraining rollout, given it was deployed site by site and cannot be withheld.
  • Your reweighted series is flat but major deviations rose. What do you escalate and to whom?
  • How would you decide the holdback period, and how would you communicate a provisional number without it being quoted as final?

Suggested practice checklist

Roughly 90 minutes a night on weekdays with one longer weekend block. The plan deliberately cuts scope rather than compressing everything, on the assumption that finishing one thing a night beats half-starting four.

Day 1 — Fix the scope and set a baseline

  • Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list for everything else and keep it visible all week.
  • Take one 20-minute SQL prompt and one 10-minute metric question cold, and write the single sentence that says what blocked each attempt, since that sentence is what decides which two topics get the most evenings.
  • Set the week's one rule: one problem finished to completion every night, including the night you only have 40 minutes.

Deliverable: A one-page scope with an explicit not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One query pattern, written three times

  • Choose the single pattern most likely to appear (a cohort retention grid, or a funnel counted by user) and write it three times from a blank file rather than editing the previous attempt.
  • On the third attempt, write the grain of every CTE as a comment before writing its body.
  • Stop at 90 minutes even if the third version is imperfect, and write the one thing you would fix with another hour.

Deliverable: Three independent versions of the same query plus a note on what changed between them.

Day 3 — Only the statistics you will be asked to defend

  • Write, in under 200 words, how you would decide whether a difference between two groups is real: the test, its assumptions, and what you would switch to when an assumption fails.
  • Compute a 95 percent confidence interval for a difference in proportions by hand on realistic numbers, then write in one sentence what changes if the two samples are paired rather than independent.
  • Write your answer to "what does a p-value mean", check it against a definition, and delete the version that describes it as the probability the hypothesis is true.

Deliverable: A 200-word written answer and one hand-computed interval you can reproduce under pressure.

Day 4 — One case, and the assumptions holding it up

  • Answer one product case aloud in 20 minutes with a recording running, then listen back with a pen and mark every claim you asserted without saying what it rested on: an assumed user behaviour, an assumed data source, an assumed baseline rate, an assumed grain.
  • Pick the three assumptions the recommendation actually depends on, write how you would check each one against data, and say which one being wrong would flip the recommendation rather than merely weaken it.
  • Write the four-step structure you used onto a card small enough to hold in working memory when you are nervous.

Deliverable: One recording, three load-bearing assumptions each with a written check, and a four-step structure card.

Day 5 — Your own work, timed

  • Write a 90-second version and a four-minute version of your main project, and time both out loud rather than reading them.
  • Prepare answers to the two follow-ups that always come: what you would do differently, and how you knew it worked.
  • Put one number in the first sentence and be able to say exactly where that number came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in a longer weekend block

  • Run a 60-minute mock covering query work, a case and a behavioural question in a single sitting with no breaks, because sustained attention is the thing evenings have not trained.
  • Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
  • Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

  • Write the 20-minute warm-up you will actually do on the morning of the interview: one query you can already write from a blank file, one metric you can define out loud, and nothing you have never seen before.
  • Re-read only your own notes from this week, and open no new material.
  • Write down the logistics: the tool you will be asked to work in, whether lookups are allowed, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the case structure, the project numbers, and the logistics.

Stories where someone disagreed with you

Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.

Describe a time when you had to explain a complex machine learning concept to a non-technical stakeholder. How did you ensure they understood?

Medium · behavioural and stakeholder questions

Approach

  1. State the situation in two sentences and spend the rest on your reasoning.
  2. Pick a story where you drove the decision, not one where you observed it.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • How did you know the outcome was caused by your change?

Disagree with a product manager about an adherence nudge

Medium · immortal time · confounding · disagreement

A product manager proposes shipping a refill-reminder feature to the whole member base, citing an internal analysis: members with 12-month proportion of days covered at or above 0.80 had 31 percent fewer admissions than members below it. The roadmap date is three weeks out and engineering is scoped. You believe the comparison builds survival and continuous coverage into the exposure definition, and is separately confounded by the kind of member who refills on time. Deliverable: what you say in the room, what you send afterwards, and the smallest study you would accept as sufficient to ship.

Approach

  1. The probe is whether you can be specific and non-territorial at once. A disagreement that arrives as 'the analysis is flawed' costs you the room; one that names a mechanism and offers a cheaper path keeps it.
  2. Name the two defects separately, because they have different fixes. Classifying a member as adherent over 12 months requires them to stay alive and stay covered for those 12 months, and that guaranteed event-free interval is assigned to the adherent group. Separately, members who refill on schedule differ from those who do not in ways claims never record.
  3. Show rather than assert. Rerun the same comparison as a landmark analysis: classify adherence over days 1 to 90, start follow-up at day 91 for everyone still enrolled and event-free, and report how much of the 31 percent survives.
  4. Separate the association claim from the intervention claim out loud. Nothing in the analysis speaks to whether a reminder changes refill behaviour, which is the actual product question and is separately testable.
  5. Offer the shippable path: a randomised holdout on a subset with proportion of days covered as the primary endpoint, powered on the plausible effect of a reminder rather than on the 31 percent, with a stated read date.

Follow-up

  • The landmark rerun still shows a 20 percent difference. Do you ship?
  • The product manager says a holdout delays launch. How do you cost that argument?
  • What primary endpoint do you choose, and why not admissions?

Explain an incomplete cost series to a non-technical executive

Medium · claims runout · uncertainty · executive communication

An executive opens a dashboard showing risk-adjusted allowed PMPM down 6 percent across the last three incurred months and asks whether the cost initiative is working. You know pharmacy adjudicates within days, professional claims within weeks, and inpatient facility claims over months, so those three months are both undercounted and mix-shifted toward cheap services. There is also a genuine movement of roughly 1.5 percent in the older, complete months. Deliverable: a two-minute spoken answer, the change you make to the chart, and what you commit to telling them and when.

Approach

  1. The probe is whether you can be honest about uncertainty without sounding evasive. Executives read hedging as not knowing, so the answer has to end in a commitment.
  2. Separate the two claims immediately. The 6 percent is an artefact of incomplete data, and there is a smaller real movement in the months that are complete. Delivering only the debunk leaves them holding nothing.
  3. Explain the lag in their terms: the bills for the most expensive care arrive last, so an incomplete month always looks cheap and always looks like it is improving. Keep the phrase completion factor out of the first pass.
  4. Fix the chart instead of explaining around it. Shade or omit the incomplete months, print the paid-through date on the axis, and overlay the same series as it looked at equivalent maturity a year earlier so the shape is comparable rather than merely lower.
  5. Close with a date and a magnitude. Commit to a restated figure once runout matures, say roughly how far you expect the 6 percent to shrink, and name what you would need to see to call the initiative working.

Follow-up

  • They ask for your best guess today, knowing it is provisional. What do you say?
  • The restated number comes back at 1 percent. How do you handle having flagged the 6 percent at all?
  • How do you stop this dashboard producing the same conversation next quarter?