Interview guides · ING

ING Data Scientist Interview Questions & Guide 2026

Practice explaining analysis before a ING data scientist interview: the recommendation, the uncertainty, the follow-ups.

The 60-second brief

At ING, the Data Scientist role sits at the intersection of advanced analytics, banking innovation, and customer-centric product development. You will be joining an organization that operates as a fully digital bank, where technology and data are not just support functions but the core ingredients of the business. Your work directly influences how millions of customers manage their finances, from credit risk modeling and churn prediction to personalized recommendation systems.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Reconcile amounts in minor units and currency
Set fraud thresholds by expected cost
Read vintage curves, not blended portfolio averages

Overview

You will operate within an Agile environment, often working in cross-functional, international teams alongside data analysts, engineers, and business stakeholders. The role is highly impact-driven; you are expected to take end-to-end ownership of Machine Learning (ML) products, ensuring they are robust, scalable, and provide tangible value. If you thrive on solving complex, real-world problems and want to see your analytical models deployed in a high-stakes financial setting, this role offers a unique platform for influence.

The ING interview process

ING candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Technical Interviews

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Discussions with Hiring Manager

Underneath the questions about your past work sits a resourcing question. Given four things worth doing and one of you, which gets done and what happens to the rest? Managers ask because that is the daily texture of the job, and because the answer shows whether you rank work by effort or by what it changes. The weak version sorts by personal interest or by whoever asked most insistently. The strong version ties each candidate piece of work to a decision somebody downstream is waiting on, and then names the one you would drop and who you would tell.

What to demonstrate

Whether you rank work by the decision it unblocks or by how interesting the method is
How you describe a request you declined, and whether you can say who you said it to
Whether your sense of how long something takes survives one follow-up question about the messy part
How you decide something is good enough to hand over unfinished

How to prepare

Write out your current queue and, next to each item, the decision that stays stalled until it lands. Anything with no waiting decision becomes your example of work you would cut
Rehearse turning down a plausible stakeholder request out loud, including the smaller alternative you offered instead
Have one case where you shipped a rough answer early and one where you refused to, with the reason that separated them

Where candidates lose points

Reading the most recent months of fraud and dispute rates as final

Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, and several reason codes run considerably longer, so the disputes belonging to a recent transaction month have simply not been filed yet. Any chart attributed by transaction date therefore slopes down at the right edge regardless of what is happening. The fix is to report only matured cohorts, or to apply development factors estimated from completed months and to show the estimate as an estimate.

Averaging delinquency across a book that is growing

A loan three months old cannot be 90 days past due, so a portfolio with many recent originations reports a low blended 90+ rate purely from age mix. The blended rate falls fastest exactly when originations grow fastest, which is precisely when credit quality most needs watching, so the metric moves in the reassuring direction during the riskiest period. Only comparisons at equal months on book are valid, which is what a vintage or roll-rate view enforces.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Questions & practice

Explain how you would handle missing data or outliers in a financial dataset before training a model.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Check what information would not exist at prediction time, and exclude it.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Bootstrap a fraud loss rate that clusters within merchant

Medium · bootstrap · clustered resampling · heavy tails

You have a per-transaction frame with auth_id, merchant_id, settled_amount_reporting and net_loss_reporting, both already in one reporting currency. Most rows carry zero loss, a few carry large ones, and losses cluster within merchant. Using only numpy's random generator and no resampling helper from any library, write a bootstrap that returns a 95 percent interval for net fraud loss in basis points of settled volume, resampling merchants with replacement and taking all rows belonging to each drawn merchant. Also produce the naive row-level interval and state which you would report.

Approach

State the estimator before writing it: total net loss divided by total settled volume, times 10,000. It is a ratio of sums, so each replicate recomputes both sums. Averaging per-transaction loss rates instead would weight a five-unit transaction like a five-thousand-unit one.
Pre-aggregate loss and volume to merchant level once. For a ratio of sums, drawing merchants and taking all their rows is arithmetically identical to drawing merchant-level (loss_sum, volume_sum) pairs, so a replicate becomes one integer draw plus two vectorised sums rather than a groupby inside the loop.
Draw B replicates of M merchant indices with replacement, where M is the observed merchant count, compute the ratio per replicate, and take the 2.5th and 97.5th percentiles. Say explicitly that this is a percentile interval and that BCa would correct the skew-induced bias if the decision is close.
Repeat with independent row draws for the naive interval and compare widths on the same replicate count.
Report the clustered interval. Rows within a merchant share an acceptance profile, a category code and a fraud exposure, so they are not independent, and the row-level interval understates variance by roughly the design effect.

Follow-up

Your clustered interval is three times wider. How do you explain that to someone who wanted a tighter number?
One merchant accounts for 40 percent of losses. What does that do to the interval, and what would you do about it?
How does this change if the question is whether two months differ rather than what this month's rate is?

Build a vintage delinquency table without pivot or unstack

Easy · vintage analysis · cohorts · groupby · cumulative max

fct_loan_performance_monthly gives loan_id, origination_month, months_on_book, days_past_due, charge_off_flag and restructured_flag. Produce a DataFrame with one row per origination_month and columns for months_on_book 0 through 12, each cell holding the share of that vintage's funded loans that had ever reached 90 or more days past due, or charge-off, by that age. You may not use pivot, pivot_table, crosstab or unstack. Cells for ages a cohort has not yet reached must be NaN rather than zero.

Approach

Define the per-row indicator as days_past_due >= 90 or charge_off_flag, then take a cumulative maximum of it per loan ordered by months_on_book, because the metric is reached-by-age-m, not in-that-state-at-age-m.
Deal with restructuring before the cumulative max. Restructuring resets days_past_due, so a restructured loan re-enters at current and, without the cumulative maximum carrying its pre-restructure worst state, reads as a cure.
Fix the denominator once as the count of distinct loan_id per origination_month across the whole cohort. Prepaid and charged-off loans stop producing rows, so a denominator recomputed at each age silently shrinks exactly where losses land.
Aggregate with groupby(['origination_month','months_on_book'])['ever_90'].sum(), then pre-build the output frame indexed by sorted origination months with integer columns 0 to 12 and assign from the grouped Series by .loc on its index.
Mask cells beyond each cohort's maximum observed months_on_book so an immature cell reads NaN instead of an artificially low rate.

Worked solution (30 min)

Sort by loan_id and months_on_book, build the ever_90 indicator, then apply groupby('loan_id')['ever_90'].cummax().
Compute cohort_size as the distinct loan_id count per origination_month, before any filtering on age.
Group the cumulative indicator by origination_month and months_on_book and sum it to get the numerator per cell.
Create the output frame with the sorted origination months as index and range(0, 13) as columns, fill it from the grouped Series, and divide each row by its cohort_size.
Compute each cohort's maximum observed months_on_book and set every cell to the right of it to NaN.

Expected result: A DataFrame with one row per origination_month and columns 0 to 12, values non-decreasing left to right within every row, near zero in column 0, and NaN forming an upper-right triangle where cohorts have not yet aged.

Verify it yourself

Every row is non-decreasing across columns; a decrease means the cumulative maximum was not applied or a restructure reset leaked through.
The denominator per row equals the cohort's distinct loan count and does not change across columns.
Column 0 is at or near zero for every vintage, since a loan cannot be 90 days past due at its first month end.
origination_month plus months_on_book is constant along an anti-diagonal, so one anti-diagonal should reproduce a direct calendar-month filter.

Follow-up

Two adjacent vintages diverge at months_on_book 6. How would you separate seasoning, mix shift and a genuine credit-quality change?
The three most recent vintages look best on this table. What do you check before saying so?
How does the table change if charge-off policy moved from 180 to 120 days past due partway through the series?

How would you use SQL window functions to calculate a rolling average of customer transaction volumes over the last 30 days?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Given two tables (users and transactions), write a query to identify the top 3 spending categories for users who have been active in the last quarter.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

How do you optimize a query that is performing poorly on a large dataset?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Reconcile captured authorizations against the daily settlement total

Easy · reconciliation · currency · rounding

fct_payment_authorization holds captured_amount_minor in transaction_currency, and settlement_amount_minor in settlement_currency with settlement_fx_rate applied at settlement rather than at authorization. The rate is quoted in major units of settlement_currency per major unit of transaction_currency, and dim_currency.minor_unit_exponent carries the ISO 4217 exponent for each code (0, 2 or 3 depending on the currency). Produce a daily reconciliation: for each settled_at date and settlement_currency, return settled_count, total settlement_amount_minor, and the sum of captured_amount_minor converted into settlement minor units. Flag any date and currency pair whose two totals differ by more than one minor unit per settled authorization. Do not sum amounts across currencies anywhere in the output.

Approach

Restrict to rows that actually settled: settled_at is not null and settlement_amount_minor is not null, which is a smaller population than captured rows because a capture can still be in flight.
Truncate settled_at to a date with an explicit time zone so the cut matches the ledger's cut, since settled_at is timestamptz and date_trunc on timestamptz silently uses the session time zone.
Join dim_currency twice, once on transaction_currency and once on settlement_currency, so both exponents are on the row. Minor units are not a common scale: a bare captured_amount_minor * settlement_fx_rate is correct only when the two exponents are equal, and a zero-decimal currency settling into a two-decimal one is wrong by a factor of 100.
Convert per row as ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle)) — minor units to major in the transaction currency, apply the major-per-major rate, then back to minor units in the settlement currency. The collapsed form ROUND(captured_amount_minor::numeric * settlement_fx_rate * POWER(10::numeric, exp_settle - exp_txn)) is the same expression. Round per row and then sum, not SUM(...) * an average rate, because the rate varies row by row and rounding per row is what the settlement file did.
Group by the settlement date and settlement_currency together, never by date alone, and carry the currency into every output column name or row.
Compare the two totals with a tolerance scaled by settled_count, since per-row rounding accumulates linearly in the number of rows rather than being a fixed constant.

Worked solution (25 min)

Filter to settled rows and derive settlement_date from settled_at with an explicit time zone.
Join dim_currency on transaction_currency and again on settlement_currency to pick up exp_txn and exp_settle; fail the run if either is null rather than defaulting to 2.
Aggregate by settlement_date and settlement_currency: COUNT(*), SUM(settlement_amount_minor), and SUM(ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle))).
Add a derived difference column and a boolean flag where ABS(difference) > settled_count.
Order by the flag first and then by settlement_date so the exceptions surface at the top.

Expected result: One row per settlement_date and settlement_currency, carrying settled_count, settlement_total_minor, converted_capture_total_minor, their signed difference, and a boolean exception flag. On clean days the difference is small relative to settled_count and has no consistent sign. Pairs with unequal exponents reconcile exactly as well as equal-exponent pairs; if they do not, the exponent terms are missing rather than the FX rate being wrong.

Verify it yourself

The difference divided by settled_count stays well under one minor unit on unflagged days, which is the expected size of per-row rounding.
Restricted to rows where exp_txn = exp_settle, the exponent-aware conversion and a bare captured_amount_minor * settlement_fx_rate agree exactly; restricted to rows where the exponents differ by 2, the bare form is smaller by exactly a factor of 100. That is the direct test that the dim_currency join is load-bearing.
Re-running with SUM over all currencies at once produces a number that changes when a zero-decimal currency's volume moves, which demonstrates why the cross-currency total is meaningless.
settled_count summed over all dates equals COUNT(*) on the same settled-row filter with no grouping.

Follow-up

A partial capture means captured_amount_minor is less than amount_minor. Where does that show up in this reconciliation, and where does it not?
On one currency pair the converted total is consistently about one hundredth of the settlement total, on every date, while the other pairs reconcile. Which two columns do you inspect first, and what single change fixes it?
The rate is documented as major-per-major. If a feed started publishing it minor-per-minor instead, which pairs would still reconcile and which would break?
How would you present a total across currencies to a finance partner who has asked for one number?

How do you decide which features to prioritize when building a recommendation system for retail banking customers?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

If a key product metric suddenly drops by 10% overnight, how would you approach the metric drop diagnosis?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How would you design the success metrics for a new personalized savings feature in the ING mobile app?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What are the most common experimentation pitfalls you have encountered, and how do you avoid them?

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

How do you determine the required sample size to reach statistical significance?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Measure an underwriting automation programme without hiding cost

Medium · metric gaming · swap set · operations metrics

An operations programme aims to cut manual underwriting. The proposed metric is manual review rate: decided_by = 'manual' over applications with decision in ('approve','decline') in fct_loan_application. The team controls the referral rules recorded in policy_rule_hits. Explain how they can halve that metric in one release while making the book worse, then design the readout that catches it: primary metric, the cuts it must always be reported in, and the guardrails. Say what evidence you would require before declaring a batch of referrals safe to auto-decide.

Approach

Name the exploit precisely. Referrals can be converted to automatic declines by a rule change, which removes them from the numerator at no operational cost while they stay in the denominator, so the rate falls and applicants a human would have approved are rejected instead. Auto-approving the same population moves the metric identically with the mirror-image failure mode.
Refuse the bare rate as a primary. Make it conditional on outcomes: manual review rate measured at a held-constant approval rate within each model_pd_12m band, so a reduction is creditable only when the decisions themselves did not move.
Require fixed cuts on every report: by model_pd_12m band and by the rule identifier inside policy_rule_hits. A single blended operations number cannot be decomposed after the fact, and the decomposition is the entire diagnostic.
Set guardrails on the things the exploit disturbs: approval rate within each score band, the override rate where manual decisions disagree with the automated recommendation, and the vintage 90+ rate at months_on_book = 12 for the newly auto-decided population against the referrals of the prior regime.
Fix the evidence bar for retiring a referral rule: a shadow period where the rule fires and the case is still worked manually, recording what the automation would have decided, what the human decided, and, for the disagreement group alone, the realised performance. Agreement on the easy majority establishes nothing about the disputed minority.
State the conflict honestly rather than designing around it. The referral population exists because the model is least certain there, so automating it trades loss for cost by construction, and the readout's job is to price that trade rather than to make it disappear.

Worked solution (30 min)

Compute manual review rate for a recent month overall, then by model_pd_12m band, then by rule identifier extracted from policy_rule_hits.
For each rule, compute the approve share and decline share of the applications it touches, so the rules whose removal would actually change decisions are identified before anyone proposes removing them.
Simulate the exploit: reclassify one high-volume referral rule's applications as automatic declines, then recompute the blended rate and the band-level approval rates side by side.
Write the readout spec: the primary with its conditioning, the two mandatory cuts, the three guardrails, and the shadow-period evidence bar.

Expected result: A table in which the simulated rule change lowers the blended manual review rate materially while approval rate in the affected model_pd_12m bands falls at the same time, together with a readout spec that makes it impossible to publish the first effect without the second.

Verify it yourself

The simulated change lowers the manual review rate and the band-level approval rate together. If band approval rate is unchanged, that rule was not binding and its removal is genuinely free.
Rule-level counts reconcile to the blended rate once applications hitting several rules are counted once. A jsonb array of rule identifiers is the usual place this double-counts.
The newly auto-decided population and the prior manual referrals are compared at equal months on book, not at equal calendar dates.

Follow-up

The override rate falls to near zero after the change. Is that good?
How long must the shadow period run, given the outcome matures at twelve months on book?
The rule fires on 200 applications a month. What can and cannot be established at that volume?

Portfolio delinquency improving while the loan book doubles

Easy · vintage analysis · age mix · right censoring

The blended 90-plus days-past-due rate across fct_loan_performance_monthly fell from 3.1 to 2.2 percent over two quarters while monthly funded volume roughly doubled. Credit leadership wants to know whether underwriting improved. Columns: loan_id, as_of_month_end, origination_month, months_on_book, original_principal_minor, principal_balance_minor, days_past_due, delinquency_bucket, restructured_flag, charge_off_flag, charge_off_date. Produce the view that answers the question honestly, and state in one sentence what the blended rate can and cannot tell you.

Approach

Name the mechanical floor first. A first instalment falls due roughly a month after funding, so a loan cannot reach dpd_90_plus until around its fourth month on book. Every recent origination therefore enters the denominator with a numerator that is structurally zero.
Build a vintage table: rows origination_month, columns months_on_book, cell equal to the share of that cohort whose worst days_past_due reached 90 or more, or whose charge_off_flag became true, at or before that age.
Use each loan's worst state to date rather than its current bucket, and take the pre-restructure worst state, because restructuring resets days_past_due and would otherwise read as a cure.
Compare cohorts only at equal months_on_book, and render cells beyond a cohort's current maturity as absent rather than zero, so the table cannot be misread left to right.
Decompose the blended move into an age-mix component and a within-age component, so the write-up states how much of the 0.9 point improvement is arithmetic rather than asserting it.

Follow-up

What does the diagonal of a vintage table represent, and when is reading it the right thing to do?
How would a change in charge-off timing policy show up in this table, and how would you separate it from credit quality?
Which single chart goes in front of the credit committee, and what do you say when someone asks for the blended series anyway?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Stories where someone disagreed with you

Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.

Describe a time you had to explain a complex technical model to a non-technical stakeholder. How did you ensure they understood the limitations?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

State honestly what your cutoff change actually contributed

Hard · impact attribution · swap set · counterfactual

Six months ago your recommendation moved a credit cutoff, using fct_loan_application and fct_loan_performance_monthly. Since then approval rate rose four points and the 12-month vintage 90-plus rate on affected cohorts is flat. In the same window the bureau changed a score attribute, marketing shifted channel mix toward broker, and the internal funding rate moved. Your performance review asks for impact in currency terms. Give the number you would stand behind, the counterfactual it rests on, and the part of the observed movement you would not claim.

Approach

Define the counterfactual before computing anything: the claim is not what happened after the change, it is what would have happened had the old cutoff scored the same applications, which means replaying the old threshold on the post-change population.
Build the swap set: applications the new cutoff approves that the old one declined, applications the old one approved that the new one declines, and everyone else held out as unaffected. Only the swap groups carry your effect. Note that the swap-out group has no outcome under the new rule, because those loans were never funded, so its forgone margin must be estimated from matched pre-change approvals rather than observed.
Price each swap group at months_on_book equal to 12 on the measure the cutoff was meant to move: interest and fees collected, minus net charge-offs, minus funding cost at the internal transfer rate.
Strip the confounders explicitly. Cohorts affected by the bureau attribute change are either recomputed on the old attribute or excluded; channel mix is held fixed by reweighting to the pre-change mix; funding cost is charged at the rate in force each month rather than one blended average.
State the residual you will not claim, with its size, and give a range rather than a point wherever cohorts have not yet reached 12 months on book.

Follow-up

The swap-in group is only 6 percent of applications. How does that change the way you present the number, and to whom?
What would you have needed to set up at launch to make this attribution clean, and why was a randomised band around the cutoff not used?

Retract a published number after finding a currency bug

Medium · error disclosure · minor units · process repair

Two weeks ago you published an interchange and fraud analysis that summed amount_minor across fct_payment_authorization without converting currencies. Minor units are not two decimals everywhere: some currencies carry none and some carry three, so the sum has no interpretation. A pricing decision is already in flight on the back of it. You now have corrected figures. Produce the retraction: what you send, to whom, in what order, and what you change in the process so this class of error is caught next time rather than trusted next time.

Approach

Size the error before announcing it, because saying the number is wrong without a magnitude and a direction forces every reader to assume the worst case.
Check whether the conclusion actually flips: if the ranking that drove the pricing decision is unchanged, that belongs in the first sentence beside the correction rather than buried at the end.
Tell the person acting on it first and directly, then the wider distribution, using the same text, so nobody learns about it secondhand.
Write the correction as four parts: the old number, the cause in one clause, the effect on the pending decision, and the new number. Leave out self-flagellation, which makes the reader do emotional work instead of acting.
Fix the class rather than the instance: a rule that a sum over amount_minor either groups by transaction_currency or passes through both conversion steps, exponent scaling and then a dated rate into one named reporting currency, plus a standing reconciliation of the settled subset to the settlement ledger inside each settlement_currency.

Follow-up

The corrected figures do not change the decision. Do you still send the correction, and what does that choice signal?
What automated check would have caught this, where would it live, and what would it cost in false alarms?