Interview guides · AMS

AMS Data Scientist Interview Questions & Guide 2026

Data scientist prep for AMS: practice defining a payments and lending metric, picking its guardrails and defending the tradeoff.

The 60-second brief

A Data Scientist at AMS operates at the intersection of advanced analytics, investigative research, and mission-critical support. This role is pivotal in transforming vast, complex datasets—ranging from structured internal databases to unstructured intelligence—into actionable insights that support audit, investigation, and federal agency goals. You are not just building models; you are building the analytical foundation that helps detect fraud, waste, and abuse.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

Decompose expected loss into PD, LGD, EAD
Report only matured cohorts for loss metrics
Reconcile amounts in minor units and currency

Overview

The work is high-stakes and highly visible. You will collaborate with cross-functional teams, including investigators and auditors, to translate complex technical findings into clear, narrative-driven intelligence. Whether you are developing predictive models for risk assessment or engineering scalable data pipelines in cloud environments like Azure or AWS, your contributions directly impact the operational efficiency and integrity of the agency’s mission.

Success in this role requires a balance of technical rigor and communication finesse. You will be expected to work independently to design experiments and validate data, while also serving as a bridge between technical infrastructure and non-technical stakeholders. It is a challenging, intellectually demanding position that requires a commitment to precision, ethical data stewardship, and the ability to thrive in a structured, long-term project environment.

The AMS interview process

AMS candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Resume Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Interviews with Hiring Managers

Underneath the questions about your past work sits a resourcing question. Given four things worth doing and one of you, which gets done and what happens to the rest? Managers ask because that is the daily texture of the job, and because the answer shows whether you rank work by effort or by what it changes. The weak version sorts by personal interest or by whoever asked most insistently. The strong version ties each candidate piece of work to a decision somebody downstream is waiting on, and then names the one you would drop and who you would tell.

What to demonstrate

Whether you rank work by the decision it unblocks or by how interesting the method is
How you describe a request you declined, and whether you can say who you said it to
Whether your sense of how long something takes survives one follow-up question about the messy part
How you decide something is good enough to hand over unfinished

How to prepare

Write out your current queue and, next to each item, the decision that stays stalled until it lands. Anything with no waiting decision becomes your example of work you would cut
Rehearse turning down a plausible stakeholder request out loud, including the smaller alternative you offered instead
Have one case where you shipped a rough answer early and one where you refused to, with the reason that separated them

Peer Interviews

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Final Round with Leadership

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Recalibrating an underwriting cutoff on approved and funded applicants only

Rejected applicants have no repayment outcome, and they were rejected because the incumbent model scored them badly, so the missingness depends directly on the outcome being modelled. Reject inference by augmentation or parcelling fills the gap using the incumbent model's own assumptions, which means it can confirm those assumptions but cannot test them. The only genuinely new information about the reject region comes from bureau performance on rejects who borrowed elsewhere, or from a deliberately randomised approval band around the cutoff.

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Questions & practice

Write integrity checks for the authorization and settlement lifecycle

Easy · data quality · minor units · fx reconciliation

You are given fct_payment_authorization as a pandas DataFrame with auth_id, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, is_reversal, parent_auth_id, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency and settlement_fx_rate. Write a function returning one row per integrity check with the check name, failing row count, failing share and up to five example auth_id values. Cover at least six checks, one of which reconciles captured_amount_minor against settlement_amount_minor through settlement_fx_rate. Partial capture, zero-amount verification and a decline with no capture are all legitimate and must not be flagged.

Approach

Separate contract violations from observations before writing any code: an approved row carrying a decline_reason_code is structurally impossible, while a capture two days after requested_at is merely slow and belongs in a different severity tier.
Express each check as a boolean mask over the whole frame and collect the masks in a dict, so the summary table is one comprehension over mask.sum() rather than a row loop.
For the reconciliation, leave minor units before comparing: expected = captured_amount_minor / 10exponent[transaction_currency] * settlement_fx_rate * 10exponent[settlement_currency]. Build the exponent table covering zero-decimal and three-decimal currencies instead of assuming two everywhere.
Guard the legitimate cases explicitly so each mask fires only on the genuine contradiction: captured_amount_minor below amount_minor is partial capture, amount_minor of zero on an approved row is account verification, a null captured_at on a declined row is correct.
Sort the output by failing share times a stated severity weight, because a check firing on 0.01 percent of rows can still be the one that breaks a ledger reconciliation.

Follow-up

Which of these would you run as a blocking pipeline assertion and which as a monitored metric, and why?
The FX check fails on 3 percent of rows, all in one settlement currency. How do you decide between a data bug and a rounding convention?
How would you detect that a currency's minor-unit exponent is wrong in your reference table, using only the transaction data?

Build a vintage delinquency table without pivot or unstack

Easy · vintage analysis · cohorts · groupby · cumulative max

fct_loan_performance_monthly gives loan_id, origination_month, months_on_book, days_past_due, charge_off_flag and restructured_flag. Produce a DataFrame with one row per origination_month and columns for months_on_book 0 through 12, each cell holding the share of that vintage's funded loans that had ever reached 90 or more days past due, or charge-off, by that age. You may not use pivot, pivot_table, crosstab or unstack. Cells for ages a cohort has not yet reached must be NaN rather than zero.

Approach

Define the per-row indicator as days_past_due >= 90 or charge_off_flag, then take a cumulative maximum of it per loan ordered by months_on_book, because the metric is reached-by-age-m, not in-that-state-at-age-m.
Deal with restructuring before the cumulative max. Restructuring resets days_past_due, so a restructured loan re-enters at current and, without the cumulative maximum carrying its pre-restructure worst state, reads as a cure.
Fix the denominator once as the count of distinct loan_id per origination_month across the whole cohort. Prepaid and charged-off loans stop producing rows, so a denominator recomputed at each age silently shrinks exactly where losses land.
Aggregate with groupby(['origination_month','months_on_book'])['ever_90'].sum(), then pre-build the output frame indexed by sorted origination months with integer columns 0 to 12 and assign from the grouped Series by .loc on its index.
Mask cells beyond each cohort's maximum observed months_on_book so an immature cell reads NaN instead of an artificially low rate.

Worked solution (30 min)

Sort by loan_id and months_on_book, build the ever_90 indicator, then apply groupby('loan_id')['ever_90'].cummax().
Compute cohort_size as the distinct loan_id count per origination_month, before any filtering on age.
Group the cumulative indicator by origination_month and months_on_book and sum it to get the numerator per cell.
Create the output frame with the sorted origination months as index and range(0, 13) as columns, fill it from the grouped Series, and divide each row by its cohort_size.
Compute each cohort's maximum observed months_on_book and set every cell to the right of it to NaN.

Expected result: A DataFrame with one row per origination_month and columns 0 to 12, values non-decreasing left to right within every row, near zero in column 0, and NaN forming an upper-right triangle where cohorts have not yet aged.

Verify it yourself

Every row is non-decreasing across columns; a decrease means the cumulative maximum was not applied or a restructure reset leaked through.
The denominator per row equals the cohort's distinct loan count and does not change across columns.
Column 0 is at or near zero for every vintage, since a loan cannot be 90 days past due at its first month end.
origination_month plus months_on_book is constant along an anti-diagonal, so one anti-diagonal should reproduce a direct calendar-month filter.

Follow-up

Two adjacent vintages diverge at months_on_book 6. How would you separate seasoning, mix shift and a genuine credit-quality change?
The three most recent vintages look best on this table. What do you check before saying so?
How does the table change if charge-off policy moved from 180 to 120 days past due partway through the series?

Implement accident-quarter loss ratio at twelve months development

Medium · loss ratio · development age · earned premium

fct_policy_period_monthly arrives as a stack of month-end snapshots: each row carries valuation_month alongside as_of_month, policy_id, product_line, written_premium_minor, earned_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor. Compute the accident-quarter loss ratio at exactly 12 months of development: incurred losses over earned premium, both taken from rows whose as_of_month falls in the accident quarter, read from the snapshot 12 months after that quarter closes. Report quarters that cannot reach that age as incomplete rather than dropping them.

Approach

Derive accident_quarter from as_of_month, then define the evaluation snapshot per quarter as valuation_month equal to the quarter's final month plus twelve months. Every figure in the ratio comes from that one snapshot, not from whichever snapshot happens to be newest.
Numerator is paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor over the accident quarter's rows in that snapshot. Loss adjustment expense may be included or not, but the choice applies to every quarter and is named in an output column.
Denominator is earned_premium_minor over the same rows. Written premium is booked in full at inception, so in a growing book it runs ahead of earned premium and drags the ratio down, with the error reversing when the book shrinks.
Left-join the full quarter list against available valuation months so a quarter with no 12-month snapshot yields status incomplete and a null ratio, instead of disappearing and shortening the series without saying so.
Split by product_line, since both the loss ratio level and the speed of development differ by line, and a blended series moves with mix as much as with experience.

Follow-up

The most recent complete quarter came in four points better than the one before. What do you check before calling it an improvement?
How would you estimate the 12-month figure for a quarter that is only 6 months developed, and how would you label the estimate?
Why can an expense ratio legitimately use a different denominator from the loss ratio in the same presentation?

How do you optimize a query that is performing poorly on a large-scale data lakehouse?

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Describe your process for cleaning and normalizing a dataset that contains significant missing values or inconsistencies.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Walk me through your approach to ETL when moving data from a legacy system to a cloud-based environment.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Accident-quarter loss ratio on earned rather than written premium

Hard · loss ratio · earned vs written · development age

From fct_policy_period_monthly, compute the accident-quarter loss ratio by product_line: incurred losses, being paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor, over earned_premium_minor for the same accident quarter. State explicitly whether loss_adjustment_expense_minor is included and apply that choice consistently. Also output the same ratio computed on written_premium_minor so the two can be compared. The table holds current values with no valuation-date snapshot. Say in one line which comparison this schema cannot support and what you would need to support it.

Approach

Derive the accident quarter from as_of_month with date_trunc, and note that the table already attributes losses to the month of the loss event while earning premium pro rata into the same month, which is what makes the two sides comparable at all.
Aggregate earned_premium_minor, written_premium_minor and the three loss components to product_line and accident quarter in one pass, keeping loss adjustment expense as its own column so the inclusion choice is a final-select decision rather than something buried in a CTE.
Compute both ratios side by side and a third column for their difference, because the size and sign of that difference is a direct read on whether the book grew or shrank in the quarter.
State the limitation plainly: every row carries today's reserve estimate, so each accident quarter is observed at a different development age and a cross-quarter comparison mixes development with underwriting. A fixed development age needs a valuation-date dimension, that is one row per accident period per valuation, which this table does not have.
Guard against the mirror-image error on the numerator by confirming ibnr_reserve_minor is non-zero on recent quarters; if it is null or zero there, the recent periods are understated twice over and the series is not usable.

Worked solution (40 min)

CTE quarterly: group fct_policy_period_monthly by product_line and date_trunc('quarter', as_of_month), summing earned_premium_minor, written_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor.
Final SELECT: build incurred_minor as the three loss components plus the LAE column, with the LAE inclusion written as a named expression so the choice is visible on the page.
Emit loss_ratio_earned and loss_ratio_written, both cast to numeric, plus their difference and the written-to-earned premium ratio.
Order by product_line and accident quarter, and append the one-line note about the missing valuation dimension to the query as a comment.

Expected result: One row per product_line and accident quarter with earned and written premium, incurred losses, both ratios and their difference. In quarters where written premium exceeds earned, loss_ratio_written is strictly below loss_ratio_earned, and the gap widens with the growth rate.

Verify it yourself

For a mature product line with flat volume, summed earned premium over a full policy term is close to summed written premium, so the two ratios converge there.
The sign of loss_ratio_earned minus loss_ratio_written matches the sign of written premium minus earned premium in every quarter, which is an arithmetic identity and not a data finding.
Recent accident quarters have non-zero ibnr_reserve_minor; a zero there means the numerator is missing its tail and the ratio is understated.
Within a quarter, loss_ratio_earned divided by loss_ratio_written equals written_premium_minor divided by earned_premium_minor exactly, because both ratios share one numerator. Any deviation means the LAE term was built into one ratio and not the other, which is the real test that the inclusion choice was applied consistently.
Recomputing with LAE excluded scales each quarter's ratio by incurred_minor / (incurred_minor + lae_minor). That factor is not constant across quarters unless LAE happens to be booked as a fixed fraction of incurred losses, so a uniform shift is evidence about how LAE is loaded, not evidence that the query is self-consistent — use the previous check for that.

Follow-up

Written premium exceeds earned premium by 18 percent this quarter and by 3 percent two years ago. What happened to the book, and what does it do to each ratio?
How would you build a development triangle from a valuation-dated version of this table, and what would you use the chain-ladder factors for?
Statutory presentation conventionally takes the expense ratio on written premium while the loss ratio uses earned. How do you avoid a combined ratio that quietly mixes the two bases?

If you noticed a sudden metric drop in a dashboard you manage, what is your step-by-step diagnostic process?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize your work when you have multiple competing requests from different project teams?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you design an A/B test to measure the impact of a new feature on user engagement?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

What are some common experimentation pitfalls you have encountered, and how do you mitigate them during the design phase?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

Monitor a rollout daily without inflating false positives

Medium · peeking · sequential testing · alpha spending

A risk-rule change is ramped to 50 percent and the team reads the dashboard every morning for twenty business days, intending to stop the first time the two-sided p-value on the count-weighted approval rate falls below 0.05. Quantify how much that inflates the false positive rate, propose a monitoring scheme that still permits an early stop for harm, and explain what dispute maturity does to the matured fraud basis points guardrail when it is read on day twenty.

Approach

Quantify rather than assert. Repeatedly applying a fixed-horizon test at nominal two-sided 0.05 inflates the family-wise type I error to roughly 8 percent at 2 looks, 14 percent at 5, 19 percent at 10 and about 25 percent at 20 equally spaced looks. Under continuous monitoring with no stopping rule the probability of crossing at some point tends to 1.
Pick a scheme matched to how the team actually behaves. If looks are on a fixed schedule, use a group-sequential design with an alpha-spending function: O'Brien-Fleming spends almost nothing early so the final boundary stays near nominal, which suits a team that mostly wants to ship at the end; Pocock spends evenly and buys genuine early-stopping power at the cost of a stricter final boundary. If looks are truly continuous and ad hoc, use an always-valid confidence sequence such as a mixture sequential probability ratio test, which is valid at every moment in exchange for a larger fixed-horizon sample at equal power.
Treat the harm stop as a separate, asymmetric decision. Stopping a change because it looks harmful costs one abandoned experiment; failing to stop costs real loss every day it runs. Run the guardrail as a one-sided monitor at a looser alpha with its own pre-registered stop rule, and do not spend the primary metric's alpha budget on it.
Fix the schedule before launch: the number of looks, their timing, the boundaries, and the maximum sample. Boundaries computed after the fact from however many times someone opened the dashboard are not a correction.
Separate peeking from immaturity on the fraud guardrail. Disputes on a transaction can be filed for roughly 120 days, and some reason codes run longer, so a day-twenty read covers transactions with at most twenty days of dispute exposure. The number is not low, it is incomplete. Either report only matured transaction months, or apply development factors estimated from completed months and label the result an estimate with its interval.

Worked solution (25 min)

Quote the inflation for the stated plan: twenty looks at nominal two-sided 0.05 gives a family-wise type I error of about 25 percent, so one rule change in four would appear significant with no true effect.
Specify the replacement: five pre-scheduled looks at 20, 40, 60, 80 and 100 percent of planned sample under an O'Brien-Fleming spending function, with z boundaries of approximately 4.56, 3.23, 2.63, 2.28 and 2.04. For comparison, a Pocock design at five looks uses a constant boundary of about 2.41.
Add a one-sided harm monitor on the fraud guardrail and the decline-rate guardrail with its own alpha and its own pre-registered rule, documented before launch.
For the fraud read, restrict to transaction months with at least 120 days of maturity; if none exist yet, present a development-factor estimate from completed months with its uncertainty and mark the recent months incomplete on the chart rather than plotting them as low.
If the boundary is crossed early, report a bias-adjusted effect estimate, because the estimate at a stopping boundary is systematically larger in magnitude than the truth.

Expected result: Twenty daily looks at nominal 0.05 give a family-wise type I error of roughly 25 percent. The recommendation is a pre-registered five-look O'Brien-Fleming group-sequential design with a final z boundary near 2.04, or an always-valid confidence sequence if looks cannot be scheduled, plus a separate one-sided harm monitor. The day-twenty fraud guardrail is reported as incomplete, not as a pass.

Verify it yourself

Simulate the stated stopping rule under the null a few thousand times; the crossing rate should reproduce the quoted inflation, and under the group-sequential boundaries it should land near 5 percent.
The alpha spent across the planned looks under the chosen spending function must sum to the total alpha; if it does not, the boundaries were not computed from a spending function.
Plot the fraud rate by transaction month against dispute maturity. If the curve slopes down at the right edge on historical data where nothing changed, that slope is maturity, not effect.
Compare the effect estimate at the crossing look with the estimate at the planned maximum sample on historical tests; the gap is the stopping bias you owe the sponsor.

Follow-up

Under an O'Brien-Fleming boundary you cross on day three. What do you say about the effect size, and why is the naive point estimate biased?
How would you set the stopping rule for the fraud guardrail given that its true value is not observable inside the test window?
The team argues that they are only looking, not deciding, so peeking is harmless. Under what precise condition is that true, and how would you verify it?

Settled volume jumps after a new acceptance corridor launches

Medium · minor units · currency conversion · reconciliation

Settled payment volume per active customer rose 12 percent in the month a new acceptance corridor went live. The pipeline sums settlement_amount_minor from fct_payment_authorization and divides by 100 to get major units. Columns available: amount_minor, transaction_currency, captured_amount_minor, settlement_amount_minor, settlement_currency, settlement_fx_rate, settled_at, is_reversal, issuer_country, acquirer_country. Confirm or refute the 12 percent, and specify exactly the conversion the metric should be using.

Approach

Split the month-over-month increase by settlement_currency. If one currency carries nearly all of it while its transaction count barely moved, the problem is arithmetic rather than demand, and that is a two-minute check.
Look up each currency's ISO 4217 exponent. Dividing a three-decimal currency by 100 instead of 1000 overstates it tenfold; dividing a zero-decimal currency by 100 understates it hundredfold. Neither is a rounding issue.
Rewrite the conversion to scale settlement_amount_minor by ten to the power of that currency's exponent, then convert to the reporting currency. Do not reuse settlement_fx_rate for this; it is the rate applied between transaction and settlement currency at settlement time, not a reporting-currency rate.
Check the other half of the series too. settlement_amount_minor is denominated in settlement_currency while amount_minor is in transaction_currency, so any series that mixes the two is uninterpretable regardless of the exponent fix.
Exclude reversals and refunds, recompute, then reconcile the corrected total to the settlement ledger for the same period and show the tie-out in the write-up.
Recompute the per-customer denominator on the corrected month, since a new corridor adds customers as well as volume and the ratio can move either way once the numerator is right.

Follow-up

Where else in the warehouse does a hardcoded divide-by-100 appear, and how would you find every instance?
What test would fail the build the next time a currency with a different exponent is onboarded?
The reporting-currency rate is itself a choice. Which rate, on which date, and why does finance care about the answer?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Getting an analysis acted on

Work that nobody used is a common and unflattering pattern in data careers, and interviewers probe for it. Have a story about an analysis that changed a decision, and be specific about how you got it in front of the person who could act. Also have one about work that went nowhere, with your reading of why.

Tell me about a time you identified a flaw in a model or analysis; how did you handle it and what was the outcome?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Disagree with a product manager over an approval-rate target

Medium · metric design · denominators · influence without authority

A product manager proposes a quarterly goal of raising card authorization approval rate by 150 basis points, measured as approved authorizations divided by all authorizations in fct_payment_authorization. You believe that metric can be hit with no customer benefit, because merchant retry chains, zero-amount verification authorizations, incremental authorizations and reversals all sit in the denominator, and declines skew toward high-value cross-border ecommerce. You support the underlying goal. In one working session, change the metric without killing the initiative, and name the guardrail you would accept.

Approach

Separate the goal from the metric out loud and agree with the goal first, so the disagreement stays narrow and technical rather than becoming positional.
Demonstrate the failure rather than asserting it: compute the proposed metric and the dollar-weighted collapsed version over the same recent window, and find a period where they moved in opposite directions.
Propose the replacement precisely: sum of approved amount_minor over sum of attempted amount_minor, after collapsing retries to one attempt per card_token_id, merchant_id and amount_minor within a 15-minute window, excluding is_reversal rows and zero-amount verifications, with everything converted to one reporting currency before summing.
Attach the guardrail that makes the target honest: matured first-chargeback rate and net fraud loss in basis points of settled volume, read only on transaction months carrying at least 120 days of maturity.
Give the product manager something back: the replacement metric cuts cleanly by channel and issuer_country, which makes a roadmap of merchant-specific and authentication fixes legible in a way the blended rate never was.

Follow-up

How do you identify a retry chain when the merchant varies the amount slightly between attempts?
The product manager wants a weekly read on the guardrail. What is the earliest defensible signal, and how do you label it?

Explain an incomplete dispute chart to a non-technical executive

Easy · dispute maturity · stakeholder communication · right-censoring

A finance lead is looking at first-chargeback rate by transaction month, built from fct_card_dispute joined to fct_payment_authorization on auth_id and attributed to requested_at. The last three months slope sharply down and the lead wants to announce a fraud improvement at tomorrow's review. Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, so those months are not complete. In five minutes, with no statistics vocabulary, explain why the decline is not yet evidence and say exactly what you would put on the slide instead.

Approach

Lead with the mechanism in the listener's own terms, not with the statistical name for it: a dispute is attributed to the month the transaction happened, but it can be filed up to roughly 120 days later, so recent months contain only the disputes filed so far.
Show completeness rather than arguing about the rate: for each transaction month, plot the share of its eventual disputes already filed, estimated from months that are fully matured. The last three months will sit visibly below 100 percent.
Replace the chart with two artefacts: a matured series that stops 120 days back and is labelled final, and a development-factor estimate for the immature months drawn as a dashed range and labelled an estimate.
Hand over one sentence the executive can repeat without you in the room: the recent months look better because the disputes have not arrived yet, not because fewer will arrive.
Offer a weekly signal they can watch instead, such as the risk-score mix of approved volume or the decline-rule hit rate, and state up front what it does and does not predict.

Follow-up

The deck ships tomorrow regardless. What exactly goes on the slide, and what wording do you insist on?
How would you estimate the development factors, and how would you notice if they had shifted?