Interview guides · Chubb

Chubb Data Scientist Interview Questions & Guide 2026

Diagnose a metric move for Chubb data scientist prep: split by segment, rule out a logging change, separate mix shift from behaviour.

The 60-second brief

As a Data Scientist at Chubb, you sit at the intersection of quantitative modeling, insurance analytics, and core business strategy. You will develop innovative data-driven solutions, build predictive and explanatory models, and collaborate with cross-functional teams across actuarial, product, and IT domains. Your work directly influences pricing strategies, risk assessment, and operational decision-making across global property and casualty insurance portfolios.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Read vintage curves, not blended portfolio averages
Report only matured cohorts for loss metrics
Separate authorization, settlement and dispute outcomes cleanly

Overview

The role requires you to transform complex, noisy datasets into actionable business insights and production-ready machine learning pipelines. Whether you are building Generalized Linear Models (GLMs), deploying ensemble algorithms like Random Forest and XGBoost, or designing rigorous experiments, your impact scales across multiple markets. You will tackle unique challenges in risk quantification while translating highly technical statistical concepts into intuitive recommendations for non-technical stakeholders.

Expect a collaborative environment where intellectual curiosity and rigorous methodology are highly valued. While interview loops can move quickly depending on your region and team alignment, interviewers look for a balanced blend of solid foundational coding, statistical depth, and pragmatic problem-solving. Success at demands both technical excellence and the ability to articulate how your models drive measurable business value.

The Chubb interview process

Chubb candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Interviews

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Interaction with Managers

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Behavioral Interviews

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

Whether you can state the other side's argument accurately before you explain why you disagreed
What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Where candidates lose points

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

Reading the most recent months of fraud and dispute rates as final

Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, and several reason codes run considerably longer, so the disputes belonging to a recent transaction month have simply not been filed yet. Any chart attributed by transaction date therefore slopes down at the right edge regardless of what is happening. The fix is to report only matured cohorts, or to apply development factors estimated from completed months and to show the estimate as an estimate.

Generalising beyond the population the sample actually supports

State the frame the sample was drawn from and where it diverges from the population you want to talk about: time window, platform, geography, opt-in. If a group is excluded from the frame, either weight to known margins or narrow the claim rather than quietly extending it.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Questions & practice

Write a script to simulate a particular scenario and calculate relevant summary statistics.

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Write down the assumption the method needs before you use the method.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

How would you explain this result to someone who does not know statistics?
Which assumption here is most likely to be violated in practice?

Explain how a Random Forest or XGBoost algorithm works under the hood and how you handle hyperparameter tuning.

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Implement accident-quarter loss ratio at twelve months development

Medium · loss ratio · development age · earned premium

fct_policy_period_monthly arrives as a stack of month-end snapshots: each row carries valuation_month alongside as_of_month, policy_id, product_line, written_premium_minor, earned_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor. Compute the accident-quarter loss ratio at exactly 12 months of development: incurred losses over earned premium, both taken from rows whose as_of_month falls in the accident quarter, read from the snapshot 12 months after that quarter closes. Report quarters that cannot reach that age as incomplete rather than dropping them.

Approach

Derive accident_quarter from as_of_month, then define the evaluation snapshot per quarter as valuation_month equal to the quarter's final month plus twelve months. Every figure in the ratio comes from that one snapshot, not from whichever snapshot happens to be newest.
Numerator is paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor over the accident quarter's rows in that snapshot. Loss adjustment expense may be included or not, but the choice applies to every quarter and is named in an output column.
Denominator is earned_premium_minor over the same rows. Written premium is booked in full at inception, so in a growing book it runs ahead of earned premium and drags the ratio down, with the error reversing when the book shrinks.
Left-join the full quarter list against available valuation months so a quarter with no 12-month snapshot yields status incomplete and a null ratio, instead of disappearing and shortening the series without saying so.
Split by product_line, since both the loss ratio level and the speed of development differ by line, and a blended series moves with mix as much as with experience.

Worked solution (30 min)

Add accident_quarter and a target_valuation column equal to the quarter end plus twelve months.
Filter rows to those where valuation_month equals the row's target_valuation, then assert each accident_quarter has exactly one distinct valuation_month left.
Aggregate incurred and earned premium by accident_quarter and product_line and take the ratio.
Reindex against the full list of accident quarters and product lines, marking rows with no matching snapshot as incomplete with a null ratio.
Recompute one quarter by hand on a five-policy subset and confirm it matches.

Expected result: One row per accident_quarter and product_line with incurred, earned_premium, loss_ratio, an lae_included flag and status in complete or incomplete. Complete quarters end twelve months before the latest valuation_month, so the four or five most recent quarters carry null ratios.

Verify it yourself

Assert exactly one distinct valuation_month per accident quarter after filtering; more than one means the snapshot join fanned out.
For a growing book, earned premium in a quarter is below written premium booked in that quarter; if it is above, either the book is shrinking or the accident-quarter derivation is wrong.
Incomplete quarters carry null ratios and never appear on the same line as complete ones in any chart.
Swapping the denominator to written premium shifts a quarter by L(W - E)/(EW), so the sign of the change is the sign of written minus earned premium in that quarter: the ratio falls where the book grew and rises where it shrank. Verify the swap per quarter against sign(W - E) rather than expecting a uniform move; a series that moves in one direction everywhere only means the book grew in every quarter.

Follow-up

The most recent complete quarter came in four points better than the one before. What do you check before calling it an improvement?
How would you estimate the 12-month figure for a quarter that is only 6 months developed, and how would you label the estimate?
Why can an expense ratio legitimately use a different denominator from the loss ratio in the same presentation?

Write a Python function to solve the two sum problem from standard coding challenges.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

What is object-oriented programming, and how do you apply its principles when writing modular data pipelines?

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Vintage ninety-plus rate at twelve months on book

Medium · vintage analysis · window functions · right censoring

From fct_loan_performance_monthly, build a vintage table keyed on origination_month: the share of each cohort that ever reached days_past_due of 90 or more, or charge_off_flag = true, at or before months_on_book = 12. Restructuring resets days_past_due, so for any loan with restructured_flag true, evaluate only the month ends strictly before its first restructured month. Return origination_month, loans_funded, bad_loans and bad_rate. Exclude any cohort that does not yet have a months_on_book = 12 observation for every loan still on book.

Approach

Establish the cohort denominator from the first month end each loan appears at, months_on_book = 0, so a loan is counted once in its origination_month rather than once per monthly row.
Find each loan's first restructured month with MIN(as_of_month_end) FILTER (WHERE restructured_flag) OVER (PARTITION BY loan_id), or the equivalent grouped subquery, and keep it null for loans never restructured.
Flag a loan bad if any row with months_on_book <= 12 and as_of_month_end earlier than that first restructured month has days_past_due >= 90 or charge_off_flag = true, which is what 'pre-restructure worst state' means in practice.
Gate maturity by requiring the cohort's newest month end to be at least 12 months after origination_month, and report immature cohorts as incomplete rather than letting them appear at a flattering low rate.
Aggregate to one row per origination_month and read the column downward, not across calendar time, because the whole point is comparing cohorts at equal age.

Worked solution (35 min)

CTE 1: per loan_id, derive origination_month, the first restructured month end, and the maximum months_on_book observed.
CTE 2: join back to the monthly rows, filter to months_on_book <= 12 and to month ends before any restructure, then flag bad with a BOOL_OR.
CTE 3: aggregate to origination_month with COUNT() as loans_funded and COUNT() FILTER (WHERE is_bad) as bad_loans.
Apply the maturity gate and compute bad_rate with a numeric cast.

Expected result: One row per matured origination_month with loans_funded, bad_loans and a bad_rate between 0 and 1. The most recent immature cohorts are absent from the output entirely rather than present with a small bad_rate.

Verify it yourself

SUM(loans_funded) across reported cohorts equals COUNT(DISTINCT loan_id) restricted to those same origination months.
Recomputing without the months_on_book cap raises bad_rate monotonically for every cohort, and raises it most for the oldest, which is the seasoning effect made visible.
No cohort in the output has a maximum months_on_book below 12.

Follow-up

Should a restructure inside 12 months count as bad in its own right? Argue both sides and say what you would actually ship.
A loan that prepaid in full at month 4 never had a chance to go 90 days past due. In or out of the denominator, and why?
The 2025-11 cohort is two points worse at month 12 than its neighbours. What three queries do you run before you call it a credit-quality change?

How would you design a metric to measure the long-term value of a new insurance product cohort?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Why do you want to work for Chubb, and how does your background align with our analytics objectives?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

What is it about the insurance industry that attracts you to this data science role?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Imagine a key business conversion metric drops by fifteen percent over the weekend. How would you systematically diagnose the root cause?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

What are the most common experimentation pitfalls you encounter when running tests on highly seasonal insurance portfolios?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

How do you test for statistical significance when dealing with skewed financial loss distributions?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

Monitor a rollout daily without inflating false positives

Medium · peeking · sequential testing · alpha spending

A risk-rule change is ramped to 50 percent and the team reads the dashboard every morning for twenty business days, intending to stop the first time the two-sided p-value on the count-weighted approval rate falls below 0.05. Quantify how much that inflates the false positive rate, propose a monitoring scheme that still permits an early stop for harm, and explain what dispute maturity does to the matured fraud basis points guardrail when it is read on day twenty.

Approach

Quantify rather than assert. Repeatedly applying a fixed-horizon test at nominal two-sided 0.05 inflates the family-wise type I error to roughly 8 percent at 2 looks, 14 percent at 5, 19 percent at 10 and about 25 percent at 20 equally spaced looks. Under continuous monitoring with no stopping rule the probability of crossing at some point tends to 1.
Pick a scheme matched to how the team actually behaves. If looks are on a fixed schedule, use a group-sequential design with an alpha-spending function: O'Brien-Fleming spends almost nothing early so the final boundary stays near nominal, which suits a team that mostly wants to ship at the end; Pocock spends evenly and buys genuine early-stopping power at the cost of a stricter final boundary. If looks are truly continuous and ad hoc, use an always-valid confidence sequence such as a mixture sequential probability ratio test, which is valid at every moment in exchange for a larger fixed-horizon sample at equal power.
Treat the harm stop as a separate, asymmetric decision. Stopping a change because it looks harmful costs one abandoned experiment; failing to stop costs real loss every day it runs. Run the guardrail as a one-sided monitor at a looser alpha with its own pre-registered stop rule, and do not spend the primary metric's alpha budget on it.
Fix the schedule before launch: the number of looks, their timing, the boundaries, and the maximum sample. Boundaries computed after the fact from however many times someone opened the dashboard are not a correction.
Separate peeking from immaturity on the fraud guardrail. Disputes on a transaction can be filed for roughly 120 days, and some reason codes run longer, so a day-twenty read covers transactions with at most twenty days of dispute exposure. The number is not low, it is incomplete. Either report only matured transaction months, or apply development factors estimated from completed months and label the result an estimate with its interval.

Worked solution (25 min)

Quote the inflation for the stated plan: twenty looks at nominal two-sided 0.05 gives a family-wise type I error of about 25 percent, so one rule change in four would appear significant with no true effect.
Specify the replacement: five pre-scheduled looks at 20, 40, 60, 80 and 100 percent of planned sample under an O'Brien-Fleming spending function, with z boundaries of approximately 4.56, 3.23, 2.63, 2.28 and 2.04. For comparison, a Pocock design at five looks uses a constant boundary of about 2.41.
Add a one-sided harm monitor on the fraud guardrail and the decline-rate guardrail with its own alpha and its own pre-registered rule, documented before launch.
For the fraud read, restrict to transaction months with at least 120 days of maturity; if none exist yet, present a development-factor estimate from completed months with its uncertainty and mark the recent months incomplete on the chart rather than plotting them as low.
If the boundary is crossed early, report a bias-adjusted effect estimate, because the estimate at a stopping boundary is systematically larger in magnitude than the truth.

Expected result: Twenty daily looks at nominal 0.05 give a family-wise type I error of roughly 25 percent. The recommendation is a pre-registered five-look O'Brien-Fleming group-sequential design with a final z boundary near 2.04, or an always-valid confidence sequence if looks cannot be scheduled, plus a separate one-sided harm monitor. The day-twenty fraud guardrail is reported as incomplete, not as a pass.

Verify it yourself

Simulate the stated stopping rule under the null a few thousand times; the crossing rate should reproduce the quoted inflation, and under the group-sequential boundaries it should land near 5 percent.
The alpha spent across the planned looks under the chosen spending function must sum to the total alpha; if it does not, the boundaries were not computed from a spending function.
Plot the fraud rate by transaction month against dispute maturity. If the curve slopes down at the right edge on historical data where nothing changed, that slope is maturity, not effect.
Compare the effect estimate at the crossing look with the estimate at the planned maximum sample on historical tests; the gap is the stopping bias you owe the sponsor.

Follow-up

Under an O'Brien-Fleming boundary you cross on day three. What do you say about the effect size, and why is the naive point estimate biased?
How would you set the stopping rule for the fraud guardrail given that its true value is not observable inside the test window?
The team argues that they are only looking, not deciding, so peeking is harmless. Under what precise condition is that true, and how would you verify it?

Approval rate rose in every band yet fell overall

Medium · mix shift · simpsons paradox · segmentation

Monthly application approval rate on fct_loan_application fell from 62 to 57 percent. Cut by bureau_score band, the rate rose in every band, including the null-bureau band. Columns: application_id, channel, submitted_at, requested_amount_minor, declared_annual_income_minor, bureau_score, model_pd_12m, model_version, policy_rule_hits, decision, decided_by, decision_at. The denominator is decision in ('approve','decline'). Explain the arithmetic, quantify how much of the five-point fall is mix versus within-band movement, and say what you would tell the team that owns acquisition.

Approach

Confirm the paradox is real rather than a banding artefact. Rebuild the bands on fixed cutpoints taken from the earlier period, because quantile bands re-cut each month move with the population and can manufacture this pattern on their own.
Compute the exact decomposition rather than describing it: within = sum of w_i0 * (r_i1 - r_i0), mix = sum of r_i0 * (w_i1 - w_i0), interaction = sum of (w_i1 - w_i0) * (r_i1 - r_i0). The three terms sum identically to the change in the blended rate, so the report can state the split.
Attribute the weight change by cutting the same fixed bands by channel. Keep null bureau_score as its own band; a thin file is a population signal, not missing data to be imputed away.
Check whether the new arrivals also changed the population inside a band, by comparing requested_amount_minor and declared_annual_income_minor distributions within one band across the two months.
Deliver two numbers rather than one story: policy is looser in every band, and the funnel is being fed a different population. Those have different owners and different fixes.

Follow-up

If the new channel is profitable at its own approval rate, is the blended fall a problem at all?
How would you present this so that nobody reads the blended series unaccompanied again?
What breaks if you fix the mix by reweighting to a frozen band distribution every month?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Tell me more about your previous work history, specific modeling projects, and your role in deploying them to production.

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
Name the disagreement or constraint, and how you resolved it with evidence.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Defend a vintage finding that contradicts the portfolio dashboard

Medium · vintage analysis · mix shift · stakeholder pushback

The lending dashboard shows blended 90-plus days-past-due falling for four consecutive quarters while originations grew 60 percent. Using fct_loan_performance_monthly, you build a vintage view keyed on origination_month by months_on_book and find the three most recent vintages are worse than their predecessors at the same age. The business lead presents that dashboard weekly and pushes back hard, suggesting you picked favourable cohorts. You get one meeting and the vintage table. Present the finding so it survives the cherry-picking objection and ends in a decision.

Approach

Reconcile before you contradict: show that aggregating your vintage table along the calendar diagonal reproduces the published blended series, so the disagreement is about age mix rather than about data quality.
Make the mechanism arithmetic rather than rhetorical: a loan cannot reach 90 days past due before it is 90 days old, so rapid origination growth shifts weight onto young months-on-book where the rate is structurally near zero.
Show every vintage rather than a selected pair, all indexed at months_on_book equal to 12, with cohort sizes printed beside each curve so nobody can claim the divergence rests on a thin cohort.
Handle restructuring explicitly, because restructured_flag resets days_past_due: count each loan on its worst pre-restructure state, or recent vintages will look better than they are.
Close on the decision rather than the chart: state what the divergence implies for the cutoff or the channel mix, and state in advance what evidence would make you withdraw the claim.

Follow-up

Two cohorts differ at month 12. How do you separate a seasoning effect from a genuine credit-quality effect?
Someone argues the recent vintages are simply a broker-channel mix shift. How do you test that, and what would confirm it?

Explain an incomplete dispute chart to a non-technical executive

Easy · dispute maturity · stakeholder communication · right-censoring

A finance lead is looking at first-chargeback rate by transaction month, built from fct_card_dispute joined to fct_payment_authorization on auth_id and attributed to requested_at. The last three months slope sharply down and the lead wants to announce a fraud improvement at tomorrow's review. Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, so those months are not complete. In five minutes, with no statistics vocabulary, explain why the decline is not yet evidence and say exactly what you would put on the slide instead.

Approach

Lead with the mechanism in the listener's own terms, not with the statistical name for it: a dispute is attributed to the month the transaction happened, but it can be filed up to roughly 120 days later, so recent months contain only the disputes filed so far.
Show completeness rather than arguing about the rate: for each transaction month, plot the share of its eventual disputes already filed, estimated from months that are fully matured. The last three months will sit visibly below 100 percent.
Replace the chart with two artefacts: a matured series that stops 120 days back and is labelled final, and a development-factor estimate for the immature months drawn as a dashed range and labelled an estimate.
Hand over one sentence the executive can repeat without you in the room: the recent months look better because the disputes have not arrived yet, not because fewer will arrive.
Offer a weekly signal they can watch instead, such as the risk-score mix of approved volume or the decline-rule hit rate, and state up front what it does and does not predict.

Follow-up

The deck ships tomorrow regardless. What exactly goes on the slide, and what wording do you insist on?
How would you estimate the development factors, and how would you notice if they had shifted?