Interview guides · AIA Group

AIA Group Data Scientist Interview Questions & Guide 2026

A two-week prep plan for AIA Group data scientist candidates: write your answer before reading the solution, revisit misses three days later.

The 60-second brief

A Data Scientist at AIA Group serves as a strategic bridge between complex data ecosystems and actionable business outcomes. Operating within one of the world’s largest insurance providers, this role is tasked with transforming massive, multi-dimensional datasets into insights that inform customer behavior, risk assessment, and operational efficiency. You will not merely build models; you will solve high-impact problems that directly influence how millions of people manage their health and financial security.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Decompose expected loss into PD, LGD, EAD
Set fraud thresholds by expected cost
Read vintage curves, not blended portfolio averages

Overview

The work is characterized by its scale and its intersection with the evolving landscape of insurtech. Whether you are optimizing product metrics, deploying machine learning solutions, or designing rigorous experimentation frameworks, your output directly impacts the bottom line of a global organization. The environment is collaborative yet rigorous, requiring you to communicate complex statistical findings to non-technical stakeholders while maintaining the highest standards of data integrity and product-sense.

The AIA Group interview process

AIA Group candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Deep-Dive

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Final Round Discussion

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Averaging delinquency across a book that is growing

A loan three months old cannot be 90 days past due, so a portfolio with many recent originations reports a low blended 90+ rate purely from age mix. The blended rate falls fastest exactly when originations grow fastest, which is precisely when credit quality most needs watching, so the metric moves in the reassuring direction during the riskiest period. Only comparisons at equal months on book are valid, which is what a vintage or roll-rate view enforces.

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Solving silently instead of narrating the reasoning

Say which branch you are taking and why you chose it over the alternative, for example checking the denominator first because it changes what the comparison means. A correct answer that arrives with no visible path scores below a rigorous one that needed a hint.

Questions & practice

Collapse retry chains and compute a dollar-weighted approval rate

Medium · sessionisation · window functions · dollar-weighted rates

fct_payment_authorization gives auth_id, card_token_id, merchant_id, amount_minor, transaction_currency, requested_at, auth_result, is_reversal, channel and issuer_country. Two reference frames give the minor-unit exponent per currency and a daily rate to one reporting currency. Collapse retry chains first: attempts sharing card_token_id, merchant_id and amount_minor whose consecutive gaps are under 15 minutes form a single attempt, whose outcome is its last row. Exclude reversals and zero-amount verifications. Return a 7-day rolling dollar-weighted approval rate by channel and issuer_country.

Approach

Filter before grouping: drop is_reversal rows and zero-amount verifications, since neither is a purchase attempt and both would otherwise sit in the denominator.
Sort by card_token_id, merchant_id, amount_minor and requested_at, take the gap to the previous row within that key, mark a chain start where the gap exceeds 15 minutes or the key changes, and label chains with a cumulative sum of that flag. This is a gap rule between consecutive attempts, not a fixed clock bucket, so a chain may span more than 15 minutes in total.
Keep each chain's terminal row by requested_at. If a retry was approved, the purchase was approved; keeping the first row reports the decline that caused the retry as the outcome.
Convert amounts exactly once: amount_minor divided by 10 to the power of the currency exponent, multiplied by the reference rate for the authorization date. Do not reach for settlement_fx_rate, which is null on precisely the declined rows the denominator needs.
Build the rolling window as a ratio of two rolling sums, approved value over total value, per channel and issuer_country. A rolling mean of daily ratios weights a quiet Sunday the same as a busy Friday.

Worked solution (35 min)

Filter out reversals and zero-amount rows, then sort by the chain key and requested_at.
Compute the within-key time difference, derive the chain start flag and the chain id, and take the last row per chain with groupby(chain_id).tail(1) after sorting.
Join the exponent and daily rate tables, compute value_reporting, and assert no nulls remain after the join.
Aggregate approved value and total value to a daily grain by channel and issuer_country, reindex to a complete date range per group so missing days are zero rather than absent.
Take 7-day rolling sums of both columns and divide, then confirm one hand-picked group-day against a direct filter.

Expected result: A DataFrame keyed by date, channel and issuer_country with approved_value, total_value and approval_rate. The collapsed attempt count is materially below the raw row count, with the gap concentrated in declined ecommerce rows, and ecommerce sits below card_present.

Verify it yourself

Within every chain, all consecutive requested_at gaps are under 15 minutes; across chains with the same key, at least one gap is not.
Chain count is strictly below raw attempt count, and the difference is concentrated in declined rows rather than spread evenly.
Total converted value on a single-currency single-day subset equals the raw amount_minor sum divided by that currency's exponent power, times that day's rate.
The rolling rate on one chosen day matches a direct filter-and-divide over the preceding seven days.

Follow-up

The count-weighted rate is flat while the dollar-weighted rate falls 80 basis points. What do you look at first?
How would you choose the 15-minute window rather than inheriting it?
A merchant moves from two retries to five. Which of your two rates moves, and is that a real change in approval quality?

Estimate a delinquency roll-rate matrix and project twelve months

Hard · roll rates · markov chain · survivorship

fct_loan_performance_monthly gives loan_id, as_of_month_end, months_on_book, delinquency_bucket, charge_off_flag, prepaid_in_full_flag and restructured_flag. Build a month-to-month transition matrix over the five delinquency buckets plus absorbing charged_off and prepaid states. Loans that stop appearing must be routed to an absorbing state rather than dropped. Project the current book forward 12 months by repeated matrix multiplication and report the projected share reaching charge-off. Handle restructured_flag explicitly, and name one place the Markov assumption fails on this data.

Approach

Build consecutive month pairs per loan by shifting as_of_month_end within loan_id, then verify the shifted value is exactly one month later. A gap is not a transition, it is an exit you have not resolved yet.
Resolve exits before counting anything. A loan whose last row carries charge_off_flag moves to charged_off, one carrying prepaid_in_full_flag moves to prepaid, and one that disappears with neither is a data question to raise rather than silently discard, because discarding it is survivorship that inflates every cure rate.
Count pairs into a 7 by 7 matrix and row-normalise. Assert every row sums to one and the two absorbing rows are the identity; a row that does not sum to one means exits were dropped.
Decide and state the restructure rule. Restructuring resets days_past_due, so a dpd_60_89 to current move on a restructured loan is not a cure. Either give restructured loans their own state or carry the pre-restructure bucket, but do not let that move land in the cure cell.
Project by taking the current bucket distribution as a row vector and multiplying by the matrix twelve times. Report the charged_off entry, and report it again from an all-current starting vector so the reader can see how much of the projection comes from loans that are already delinquent today.
State the homogeneity failure plainly: transition rates depend strongly on months_on_book, so one pooled matrix applied to a book with a young mix understates early-life delinquency. If the mix is moving, estimate separate matrices by seasoning band.

Follow-up

How would you validate the projection against what actually happened, and over what window?
The cure rate out of dpd_30_59 rose five points last quarter. What are the candidate explanations and how would you separate them?
When would you prefer a vintage curve to a roll-rate projection, and why?

Measure calibration of a twelve-month default probability from scratch

Hard · calibration · brier score · binning

fct_loan_application gives application_id, model_pd_12m, model_version, decision, funded_at and loan_id. fct_loan_performance_monthly gives loan_id, months_on_book, days_past_due and charge_off_flag. Define the outcome as ever 90 or more days past due, or charged off, by months_on_book = 12. Without sklearn or scipy, build an equal-count binned reliability table, the expected calibration error, the Brier score and its reliability, resolution and uncertainty components, and report the residual the binned identity leaves behind. Restrict to cohorts that have actually reached 12 months on book.

Approach

Build the label first and name the population it covers out loud: only funded loans have outcomes, so this measures calibration on the approved population. The declined region is unmeasured, and no binning scheme repairs that.
Restrict to applications whose loans have reached months_on_book = 12. A cohort observed at 8 months has a mechanically lower default rate and will read as systematic over-prediction that is really just immaturity.
Bin by equal count, deciles of model_pd_12m through a rank-based cut, not equal width. The PD distribution is heavily right-skewed, so equal-width bins put most of the mass in the first bin and leave the risky bins with single-digit counts whose observed rates mean nothing.
Per bin compute n, mean predicted, observed rate, and the binomial standard error sqrt(o(1-o)/n) so a gap can be read against noise. ECE is the count-weighted mean absolute gap between mean predicted and observed.
Compute Brier directly as the mean squared error, then reliability = sum of n_k (pbar_k - obar_k)^2 over N, resolution = sum of n_k (obar_k - obar)^2 over N, uncertainty = obar(1 - obar). Report residual = Brier - (reliability - resolution + uncertainty). That identity is exact only for discrete forecasts, so with binned continuous scores the residual is the within-bin spread of the score; a large one means the bins are too wide to support the decomposition.
Split by model_version. A mixed-version population can look well calibrated in aggregate while each version is biased in opposite directions.

Follow-up

AUC is unchanged after a population shift but the reliability curve has moved. What happened, and what do you do about it?
How would you recalibrate without retraining, and what would you check afterwards?
The top decile shows observed default well above predicted. Is that a calibration problem or a policy problem?

How do you ensure data quality when merging datasets from disparate legacy systems?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Write a query using SQL window functions to calculate a rolling average of customer claims over the last 90 days.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Explain how you would optimize a query that joins multiple large tables in a production environment.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Collapse merchant retry chains before measuring authorization approval

Medium · window functions · gaps and islands · sessionisation

Merchant retry logic turns one declined purchase into several rows in fct_payment_authorization. Collapse attempts into chains: inside a partition of (card_token_id, merchant_id, amount_minor), consecutive rows whose requested_at gap is 15 minutes or less belong to the same chain. Return one row per chain with chain_started_at, attempt_count, amount_minor, channel, and chain_outcome set to 'approved' when any attempt in the chain was approved. Then report a 7-day rolling dollar-weighted approval rate by channel over chains rather than attempts. Exclude is_reversal rows.

Approach

In a CTE, take LAG(requested_at) OVER (PARTITION BY card_token_id, merchant_id, amount_minor ORDER BY requested_at) and set is_new_chain when the previous value is null or the gap exceeds 15 minutes.
Turn the flag into a chain identifier with SUM(is_new_chain::int) OVER (same partition, same order, rows unbounded preceding to current row), which is the standard gaps-and-islands construction.
Aggregate to one row per chain: MIN(requested_at), COUNT(*), and BOOL_OR(auth_result = 'approved') as the chain outcome, because the purchase succeeded if any attempt in the chain did.
Roll the chains to a daily grain per channel, then apply the rolling window as PARTITION BY channel ORDER BY chain_date RANGE BETWEEN INTERVAL '6 days' PRECEDING AND CURRENT ROW. An interval-offset RANGE frame is value-based: it admits every row in the partition whose chain_date falls in the closed window [D - 6 days, D], so a channel with no chains on some dates still gets the right seven-day sums — sparsity is exactly what this construct tolerates. ROWS BETWEEN 6 PRECEDING AND CURRENT ROW is the one that needs a dense grain, because it counts rows and on a sparse grain reaches back further than seven days.
Left-join a dense date spine only if the output needs a row on days with no chains. That is a presentation requirement, not a correctness one, and it leaves every rolling sum unchanged. The RANGE frame's preconditions are that chain_date is a date or timestamp (an interval offset needs an ordering column it can be added to) and that the aggregation left one row per (channel, chain_date).
Divide approved chain value by total chain value inside the rolling frame, never the average of daily rates, since averaging rates weights a quiet day the same as a heavy one.

Worked solution (40 min)

CTE 1: filter out is_reversal rows, then compute prev_requested_at with LAG and the is_new_chain flag.
CTE 2: build chain_id with a running SUM of the flag over the same partition and order.
CTE 3: group by partition keys plus chain_id to emit chain_started_at, attempt_count, amount_minor, channel and chain_outcome.
CTE 4: aggregate chains to (chain_date, channel) with approved value and total value. Add the date spine here only if the report must show empty days.
Final SELECT: two rolling SUMs over PARTITION BY channel ORDER BY chain_date RANGE BETWEEN INTERVAL '6 days' PRECEDING AND CURRENT ROW, divided at the end.

Expected result: A rolling series with one row per date and channel, where the chain count is strictly below the attempt count on ecommerce and recurring, and the chain-level approval rate is at or above the attempt-level rate for every channel, since collapsing removes failed attempts that later succeeded.

Verify it yourself

SUM(attempt_count) over all chains equals the filtered attempt row count exactly; if it does not, the chain identifier is not partitioned the same way as the LAG.
Every chain has exactly one row whose is_new_chain flag is true, so COUNT(DISTINCT chain_id) equals the number of flagged rows.
A chain whose attempt_count is 1 must have chain_outcome equal to that single row's approved status.
The rolling denominator on day D equals the sum of the daily denominators over every date present between D-6 and D inclusive — seven terms on a dense grain, fewer on a sparse one, and the same total either way.
Adding or removing the date spine leaves every rolling sum identical, because spine rows contribute zero. Swapping RANGE for ROWS changes them on any channel whose grain has a gap, which is the direct test of which construct tolerates sparsity.

Follow-up

This definition chains off the previous attempt, so a chain can span an hour if attempts arrive every 14 minutes. When is that right, and when would you anchor the window to the first attempt instead?
Two genuine purchases of the same amount at the same merchant, four minutes apart, collapse into one chain. How large is that error and how would you bound it?
A channel goes dark for three days mid-window. Which of the two frame types would silently widen, and what would the resulting rate look like?
What changes if the merchant retries with a slightly different amount to dodge a velocity rule?

If a key engagement metric suddenly drops by 10%, how would you conduct a diagnostic investigation?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you define the success metrics for a new digital insurance product?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you balance long-term customer lifetime value against short-term conversion rates?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Design an experiment to test the effectiveness of a new personalized recommendation engine.

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
Name the guardrails that would stop a launch even on a positive primary result.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

What are the risks of "peeking" at your A/B test results before the experiment concludes?

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

Read a segment-sliced result with a fading first-week lift

Medium · multiple comparisons · novelty effects · mix shift

A three-week test of an in-app credit-limit-increase offer had one pre-registered primary metric. The readout also slices 24 secondary segment-by-metric cells, of which three are significant at 0.05 with p-values 0.011, 0.028 and 0.041. The treatment effect on the primary was plus 8.1 percent in calendar week one, plus 3.4 percent in week two and plus 1.5 percent in week three. Say which claims survive, what the decay does and does not prove, and what you tell the sponsor.

Approach

Compute what chance alone produces before interpreting any cell. With 24 tests at 0.05 the expected number of false positives is 1.2, the probability of at least one is 1 - 0.95^24 = 0.708, and the probability of three or more under an independence approximation is about 0.12. Three hits is an unremarkable outcome, not a signal.
Apply a stated correction rather than a vibe, and be precise about which one the reported numbers can settle. Holm and Bonferroni both begin at 0.05 / 24 = 0.00208, so the smallest p-value alone decides them. Benjamini-Hochberg sorts all m p-values ascending and finds the largest k with p(k) <= k * q / m; at q = 0.10 and m = 24 the rank-k threshold k / 240 starts at 0.00417 and rises to 0.10 at k = 24, so it is a step-up test over the whole family and its verdict cannot be read off the three cells that happened to fall below 0.05. Ask for all 24 p-values, report the surviving set, which may be empty, and demote the rest to hypotheses for a future test.
Refuse to call the decay novelty until the competing explanations are ruled out. Three things produce a falling weekly effect: a genuine novelty response from existing users that reverts, a mix shift because week three's exposed population is not week one's, and an effect that is correctly one-shot because a limit increase is taken once per customer.
Separate them with concrete cuts. Plot the effect against time since first exposure instead of calendar week, which removes the mix shift. Compare users onboarded during the test, who never saw the previous experience and therefore cannot show novelty, against existing users. Hold back a small fraction of traffic for a long-run read.
Fix the estimand to match the mechanism. For a one-shot offer the meaningful quantity is cumulative take-up per exposed user over a fixed horizon, not a weekly rate, and under that estimand a declining weekly rate is exactly what a working feature looks like.
Give the sponsor one number with its interval, name the decision it supports, and list the exploratory cells separately under a heading that says they are not results.

Worked solution (25 min)

Compute the chance baseline: 0.95^24 = 0.292, so P(at least one false positive) = 0.708 and the expected count is 1.2.
Settle the corrections the reported numbers can settle. Holm and Bonferroni both require the smallest p-value to clear 0.05 / 24 = 0.00208; 0.011 does not, so neither rejects anything, and that conclusion needs only the one number. Benjamini-Hochberg is different: with m = 24 and q = 0.10 the rank-k threshold is k / 240, which runs 0.00417, 0.00833, 0.0125 for the three reported cells (all three fail their own thresholds) but passes 0.05 at k = 13 (0.05417) and reaches 0.10 at k = 24. Because BH rejects the k smallest hypotheses whenever any p(k) clears its threshold, one of the 21 unreported p-values sitting between 0.05 and its own rank threshold would make BH reject every cell down to rank 1, the three included. Demand the full ordered vector of 24 p-values before stating a BH verdict.
Rebuild the weekly series on time since first exposure and overlay new versus existing users, then state which of novelty, mix shift and one-shot mechanics the pattern is consistent with.
Recompute the primary as cumulative take-up per exposed user at a fixed 21-day horizon and report that with its confidence interval as the headline.
Write the sponsor summary: one confirmatory number, the exploratory cells listed as non-results pending the full p-value vector, and a proposed follow-up test for the single most plausible segment hypothesis.

Expected result: Holm and Bonferroni reject nothing, and that follows from the reported numbers alone: the smallest p-value, 0.011, exceeds 0.05 / 24 = 0.00208. Benjamini-Hochberg at q = 0.10 is not decidable from three p-values, because it is a step-up test over all 24 whose rank-k threshold k / 240 passes 0.05 at k = 13, so the correct move is to demand the full ordered vector and name the condition that would flip it rather than assert that nothing survives. Until that vector arrives the three cells are exploratory, not findings. Only the pre-registered primary is confirmatory, restated as cumulative take-up over a fixed horizon. The 8.1 to 1.5 percent decay is not established as novelty; the correct answer names mix shift and the one-shot nature of a limit increase as competing explanations and gives the cuts that separate them.

Verify it yourself

Binomial sanity check: P(3 or more hits out of 24 at 0.05) is about 0.12 under independence, so three hits is unremarkable; correlation among the cells makes the true probability higher still.
BH self-check: the rank-k thresholds k * 0.10 / 24 must run from 0.00417 at k = 1 to 0.10 at k = 24 and cross 0.05 between k = 12 and k = 13. Any BH conclusion that uses only the p-values below 0.05 is ignoring the ranks where the threshold is loosest.
The cumulative take-up curve per exposed user should be monotone increasing and flatten, while the weekly rate falls. Both are the same data, which is the point.
New users during the test should show no decay if the decay is novelty. If they decay the same way, novelty is not the explanation.
Re-run the segment slice on an A/A period: if it also produces two or three significant cells, the slicing procedure itself is the finding.

Follow-up

The three significant cells are all in the same segment. Does that change your reading, and how would you test whether it is a real interaction rather than three correlated slices of one population?
Design the long-term holdback: what size, for how long, and what does it cost you in foregone treatment?
If cumulative take-up is the estimand, what is the right horizon and how do you keep it from being chosen after seeing the data?

Approval rate fell but approved value did not

Easy · approval rate · retry chains · denominators

Over ten days the count-weighted 7-day approval rate on fct_payment_authorization fell from 91 to 86 percent, while captured value in the reporting currency is flat. Available columns: auth_id, card_token_id, merchant_id, mcc, channel, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, risk_score, is_reversal, parent_auth_id, captured_at, captured_amount_minor, issuer_country. In twenty minutes, decide whether approved value is actually at risk, and hand back a corrected rate together with its denominator and every exclusion written down.

Approach

Restate the denominator before querying anything. The current one counts every row, so reversals (is_reversal = true), incremental authorizations (parent_auth_id not null) and zero-amount verification attempts (amount_minor = 0) are all sitting in it.
Chart numerator and denominator separately by day. If approved counts are flat and total attempts rose, the rate moved because the denominator grew, which is a different investigation from a rule change and points at a different owner.
Group declines by decline_reason_code and merchant_id. Retry-driven inflation concentrates in a few soft decline codes at a few merchants; a genuine policy change spreads across merchants within one code family.
Collapse retry chains: partition by (card_token_id, merchant_id, amount_minor), keep one attempt per 15-minute window taking the best outcome, and recompute both count-weighted and dollar-weighted rates on the collapsed set.
Convert amount_minor to one reporting currency using each currency's ISO 4217 exponent before any dollar weighting, then cut by channel and issuer_country to confirm nothing is hiding underneath a flat total.
Report both rates side by side with the exclusion list attached, and state which definition the alert should have been built on.

Follow-up

How would you pick the retry-collapsing window when merchants retry on different schedules?
Flat captured value could itself be hiding a mix shift. How do you rule that out?
What monitor would have caught denominator inflation on the day it started, rather than ten days later?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

Describe a situation where you had to lead a project with ambiguous requirements; how did you define success?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Recommend a decision whose true outcome matures a year later

Hard · decision under censoring · leading indicators · staged rollout

An underwriting rule change must be decided in six weeks. Its real outcome, the vintage 90-plus rate at months_on_book 12 in fct_loan_performance_monthly, matures in a year. The executive wants a yes or no, not a range. Randomising the credit decision across the whole population is not available. Name the leading indicator you would accept, state its bias and the direction of that bias, define the decision rule and stopping condition before any rollout starts, and say what reading would make you recommend reversing the change.

Approach

Fix the readout before the rollout, because a readout chosen after the data arrives is a story rather than a decision rule: indicator, window, threshold and reversal condition all go in writing first.
Choose the leading indicator on its measured relationship to the matured outcome in historical vintages rather than on availability. Early delinquency, typically the share reaching dpd_1_29 or missing a first scheduled payment by months_on_book 3, is the usual candidate, and you quantify how well it predicted the 12-month rate across past cohorts.
State the bias and its direction plainly: early delinquency under-represents default that emerges later and is contaminated by servicing and payment-date effects, so treat it as a floor on risk rather than an estimate of it.
Buy identification where full randomisation is unavailable: a narrow randomised approval band around the cutoff, or a staged rollout by channel or region read as a difference-in-differences, with the parallel-trends assumption stated and checked in the pre-period rather than assumed.
Give the executive the binary they asked for with the trigger attached in the same sentence: yes, conditional on the month-3 indicator staying inside a stated band, with an automatic hold if it breaches.

Follow-up

How would you validate that the month-3 indicator predicts the 12-month outcome, and what evidence would invalidate it mid-rollout?
Compliance refuses a randomised band. What is your next-best identification strategy, and what precision do you lose by taking it?

Explain an incomplete dispute chart to a non-technical executive

Easy · dispute maturity · stakeholder communication · right-censoring

A finance lead is looking at first-chargeback rate by transaction month, built from fct_card_dispute joined to fct_payment_authorization on auth_id and attributed to requested_at. The last three months slope sharply down and the lead wants to announce a fraud improvement at tomorrow's review. Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, so those months are not complete. In five minutes, with no statistics vocabulary, explain why the decline is not yet evidence and say exactly what you would put on the slide instead.

Approach

Lead with the mechanism in the listener's own terms, not with the statistical name for it: a dispute is attributed to the month the transaction happened, but it can be filed up to roughly 120 days later, so recent months contain only the disputes filed so far.
Show completeness rather than arguing about the rate: for each transaction month, plot the share of its eventual disputes already filed, estimated from months that are fully matured. The last three months will sit visibly below 100 percent.
Replace the chart with two artefacts: a matured series that stops 120 days back and is labelled final, and a development-factor estimate for the immature months drawn as a dashed range and labelled an estimate.
Hand over one sentence the executive can repeat without you in the room: the recent months look better because the disputes have not arrived yet, not because fewer will arrive.
Offer a weekly signal they can watch instead, such as the risk-score mix of approved volume or the decline-rule hit rate, and state up front what it does and does not predict.

Follow-up

The deck ships tomorrow regardless. What exactly goes on the slide, and what wording do you insist on?
How would you estimate the development factors, and how would you notice if they had shifted?