Fortune Brands Innovations Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at Fortune Brands Innovations plays a pivotal role in bridging the gap between raw operational data and actionable business intelligence. You will contribute to a company that manages a diverse portfolio of home and security products, meaning your work directly influences manufacturing efficiencies, supply chain logistics, and consumer-facing product innovations.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

• Decompose expected loss into PD, LGD, EAD
• Set fraud thresholds by expected cost
• Report only matured cohorts for loss metrics

Overview

This role is critical for transforming complex datasets—ranging from water flow metrics to consumer demand signals—into strategic insights that drive the company’s bottom line. You will operate at the intersection of statistical modeling and practical application, working alongside cross-functional teams to solve real-world engineering and product challenges. It is an environment where precision, technical agility, and a clear understanding of business impact are highly valued.

While the role is data-centric, the company places a high premium on candidates who demonstrate a genuine passion for their technical stack and an ability to communicate complex findings to non-technical stakeholders.

The Fortune Brands Innovations interview process

Fortune Brands Innovations candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

• Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
• Whether your reason for wanting the role points at the work itself rather than the company's reputation
• Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

• Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
• Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
• Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Interviews

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Take-Home Project

The clock is part of the test. Three to six hours is not enough to do everything the dataset supports, so the submission mostly reveals how you spend a fixed budget against an open question. A reviewer sees which paths you took and, by absence, which you abandoned. Work that runs out of time inside the analysis ships a thin conclusion, while work that cuts scope early protects the last hour for writing. The most reliable way to lose here is to leave the scoping decision implicit, so it reads as something you missed rather than something you chose.

What to demonstrate

• Whether the scope you settled on is presented as a decision with a reason, rather than left for the reader to infer from what is missing
• Whether the depth of the work is consistent with the stated time budget, instead of several half-finished directions left open
• Whether the closing section reads as something written on purpose rather than assembled from whichever cells survived

How to prepare

• Run a timed rehearsal on a public dataset with a hard stop, holding the final sixty minutes for writing no matter where the analysis has got to
• Before opening the data, list the questions it could plausibly answer, pick one, and keep the discarded ones as a short note on what you did not attempt and why
• Commit a one-line finding after each analysis step so the writeup is assembled from recorded results rather than from memory at midnight

Where candidates lose points

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Averaging per-user rates to produce a population rate

Decide which quantity you want: the mean of per-user ratios and the ratio of summed numerator to summed denominator are different estimands, and heavy users dominate one but not the other. For a ratio metric, aggregate numerator and denominator separately and use the delta method for its variance.

Questions & practice

Can you explain the concept of p-values in the context of hypothesis testing?

Medium · statistics and probability

Approach

1. Sanity-check the answer against a simple bound or a simulated case.
2. Write down the assumption the method needs before you use the method.
3. Translate the result into the decision it informs, in one plain sentence.

Follow-up

• How would you explain this result to someone who does not know statistics?
• Which assumption here is most likely to be violated in practice?

How do you determine if a model is overfitting, and what steps do you take to mitigate it?

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Set a baseline first, so any model has something honest to beat.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Explain the difference between Linear Regression and Logistic Regression.

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Frame the prediction: the label, the moment of prediction, and the action it triggers.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Write integrity checks for the authorization and settlement lifecycle

Easy · data quality · minor units · fx reconciliation

You are given fct_payment_authorization as a pandas DataFrame with auth_id, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, is_reversal, parent_auth_id, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency and settlement_fx_rate. Write a function returning one row per integrity check with the check name, failing row count, failing share and up to five example auth_id values. Cover at least six checks, one of which reconciles captured_amount_minor against settlement_amount_minor through settlement_fx_rate. Partial capture, zero-amount verification and a decline with no capture are all legitimate and must not be flagged.

Approach

1. Separate contract violations from observations before writing any code: an approved row carrying a decline_reason_code is structurally impossible, while a capture two days after requested_at is merely slow and belongs in a different severity tier.
2. Express each check as a boolean mask over the whole frame and collect the masks in a dict, so the summary table is one comprehension over mask.sum() rather than a row loop.
3. For the reconciliation, leave minor units before comparing: expected = captured_amount_minor / 10exponent[transaction_currency] * settlement_fx_rate * 10exponent[settlement_currency]. Build the exponent table covering zero-decimal and three-decimal currencies instead of assuming two everywhere.
4. Guard the legitimate cases explicitly so each mask fires only on the genuine contradiction: captured_amount_minor below amount_minor is partial capture, amount_minor of zero on an approved row is account verification, a null captured_at on a declined row is correct.
5. Sort the output by failing share times a stated severity weight, because a check firing on 0.01 percent of rows can still be the one that breaks a ledger reconciliation.

Worked solution (25 min)

1. Assert auth_id is unique, then build a currency exponent lookup that includes the zero-decimal and three-decimal currencies present in the data.
2. Define masks for: approved with non-null decline_reason_code; declined with non-null captured_at; captured_amount_minor above amount_minor with parent_auth_id null; captured_at before requested_at; settled_at before captured_at; is_reversal true with parent_auth_id null; settlement_currency differing from transaction_currency while settlement_fx_rate is null.
3. Add the exponent-aware reconciliation mask with a tolerance of one minor unit plus a small relative term.
4. Assemble a frame of check_name, n_failing, pct_failing and up to five sample auth_id values, ordered by severity then share.
5. Read five flagged rows per check by hand and confirm each is genuinely contradictory before reporting any counts.

Expected result: A DataFrame of at least eight rows, one per check, each with n_failing, pct_failing and example auth_id values. The reconciliation check should fire on a whole currency at once rather than on scattered rows, because an exponent error is systematic while an FX error is not.

Verify it yourself

• Every mask returns false on three hand-constructed legitimate rows: a partial capture, a zero-amount verification and a decline with no capture.
• Each pct_failing uses the denominator that check actually applies to, with reversals and verifications excluded where they are out of scope.
• The reconciliation residual is centred at zero with spread under one minor unit; residuals clustered at exactly 100x or 0.01x mean an exponent is wrong, not a rate.

Follow-up

• Which of these would you run as a blocking pipeline assertion and which as a monitored metric, and why?
• The FX check fails on 3 percent of rows, all in one settlement currency. How do you decide between a data bug and a rounding convention?
• How would you detect that a currency's minor-unit exponent is wrong in your reference table, using only the transaction data?

Can you walk me through your process for handling missing data in a Python-based pipeline?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

How do you optimize a complex SQL query that is running slowly on a large dataset?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

What is the difference between a LEFT JOIN and an INNER JOIN in SQL?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

Attribute chargebacks to transaction month without fanning out

Medium · fan-out · cohort attribution · maturity

Report the matured first-chargeback rate. Numerator: fct_card_dispute rows whose dispute_stage is first_chargeback, representment, pre_arbitration or arbitration, attributed to the requested_at month of the linked fct_payment_authorization row rather than to opened_at. Denominator: settled authorizations in that same transaction month, meaning settled_at is not null and is_reversal is false. One authorization can carry more than one dispute case. Report only transaction months whose last day is at least 120 days old, and mark the remainder incomplete rather than showing them as low.

Approach

1. Compute the denominator from fct_payment_authorization alone, grouped by the requested_at month, before any dispute table is mentioned, so the join cannot touch it.
2. Compute the numerator as a separate aggregate: join disputes to their authorization only to recover requested_at, then count dispute rows per transaction month with the stage filter.
3. Join the two aggregates on the month key, which keeps the grain at one row per month and makes double counting structurally impossible rather than something you have to remember.
4. Apply the maturity gate on the month's last day plus 120 days against the current date, because a transaction on the final day of the month is the least mature one in the cohort.
5. Emit an explicit status column of 'final' or 'incomplete' instead of filtering immature months away silently, so a reader cannot mistake absence for zero.

Worked solution (30 min)

1. CTE denom: SELECT date_trunc('month', requested_at) AS txn_month, COUNT(*) AS settled_txns FROM fct_payment_authorization WHERE settled_at IS NOT NULL AND NOT is_reversal GROUP BY 1.
2. CTE numer: join fct_card_dispute to fct_payment_authorization on auth_id, filter dispute_stage to the four stages at or beyond first_chargeback, group by the same txn_month expression, count dispute rows.
3. LEFT JOIN numer onto denom on txn_month so months with zero disputes appear as zero rather than disappearing.
4. Derive the rate with a numeric cast and a status column driven by (txn_month + interval '1 month - 1 day') + interval '120 days' <= current_date.

Expected result: One row per transaction month with settled_txns, chargeback_cases, a rate in basis points, and a status of 'final' or 'incomplete'. The final months form a roughly flat series; the incomplete months at the right edge are visibly and correctly labelled rather than sloping to zero.

Verify it yourself

• settled_txns for a given month is identical whether or not the dispute CTE is joined, which is the direct test that no fan-out occurred.
• COUNT(DISTINCT auth_id) in the numerator is at or below the dispute row count; if they are equal, no authorization in the data carries two cases and the fan-out risk is latent rather than absent.
• Every month flagged 'final' has its last day at least 120 days before the current date.
• Attributing by opened_at instead shifts the series later and flattens the right edge, which demonstrates why the attribution key matters.

Follow-up

• Should the numerator count dispute cases or disputed authorizations? Which one reconciles to the loss line, and which one to the operations queue?
• Some reason codes allow filing well beyond 120 days. How would you estimate the tail on a month you have decided to call final?
• How would you present the incomplete months so that a weekly reader does not read the right-hand slope as an improvement?

Monitor a rollout daily without inflating false positives

Medium · peeking · sequential testing · alpha spending

A risk-rule change is ramped to 50 percent and the team reads the dashboard every morning for twenty business days, intending to stop the first time the two-sided p-value on the count-weighted approval rate falls below 0.05. Quantify how much that inflates the false positive rate, propose a monitoring scheme that still permits an early stop for harm, and explain what dispute maturity does to the matured fraud basis points guardrail when it is read on day twenty.

Approach

1. Quantify rather than assert. Repeatedly applying a fixed-horizon test at nominal two-sided 0.05 inflates the family-wise type I error to roughly 8 percent at 2 looks, 14 percent at 5, 19 percent at 10 and about 25 percent at 20 equally spaced looks. Under continuous monitoring with no stopping rule the probability of crossing at some point tends to 1.
2. Pick a scheme matched to how the team actually behaves. If looks are on a fixed schedule, use a group-sequential design with an alpha-spending function: O'Brien-Fleming spends almost nothing early so the final boundary stays near nominal, which suits a team that mostly wants to ship at the end; Pocock spends evenly and buys genuine early-stopping power at the cost of a stricter final boundary. If looks are truly continuous and ad hoc, use an always-valid confidence sequence such as a mixture sequential probability ratio test, which is valid at every moment in exchange for a larger fixed-horizon sample at equal power.
3. Treat the harm stop as a separate, asymmetric decision. Stopping a change because it looks harmful costs one abandoned experiment; failing to stop costs real loss every day it runs. Run the guardrail as a one-sided monitor at a looser alpha with its own pre-registered stop rule, and do not spend the primary metric's alpha budget on it.
4. Fix the schedule before launch: the number of looks, their timing, the boundaries, and the maximum sample. Boundaries computed after the fact from however many times someone opened the dashboard are not a correction.
5. Separate peeking from immaturity on the fraud guardrail. Disputes on a transaction can be filed for roughly 120 days, and some reason codes run longer, so a day-twenty read covers transactions with at most twenty days of dispute exposure. The number is not low, it is incomplete. Either report only matured transaction months, or apply development factors estimated from completed months and label the result an estimate with its interval.

Worked solution (25 min)

1. Quote the inflation for the stated plan: twenty looks at nominal two-sided 0.05 gives a family-wise type I error of about 25 percent, so one rule change in four would appear significant with no true effect.
2. Specify the replacement: five pre-scheduled looks at 20, 40, 60, 80 and 100 percent of planned sample under an O'Brien-Fleming spending function, with z boundaries of approximately 4.56, 3.23, 2.63, 2.28 and 2.04. For comparison, a Pocock design at five looks uses a constant boundary of about 2.41.
3. Add a one-sided harm monitor on the fraud guardrail and the decline-rate guardrail with its own alpha and its own pre-registered rule, documented before launch.
4. For the fraud read, restrict to transaction months with at least 120 days of maturity; if none exist yet, present a development-factor estimate from completed months with its uncertainty and mark the recent months incomplete on the chart rather than plotting them as low.
5. If the boundary is crossed early, report a bias-adjusted effect estimate, because the estimate at a stopping boundary is systematically larger in magnitude than the truth.

Expected result: Twenty daily looks at nominal 0.05 give a family-wise type I error of roughly 25 percent. The recommendation is a pre-registered five-look O'Brien-Fleming group-sequential design with a final z boundary near 2.04, or an always-valid confidence sequence if looks cannot be scheduled, plus a separate one-sided harm monitor. The day-twenty fraud guardrail is reported as incomplete, not as a pass.

Verify it yourself

• Simulate the stated stopping rule under the null a few thousand times; the crossing rate should reproduce the quoted inflation, and under the group-sequential boundaries it should land near 5 percent.
• The alpha spent across the planned looks under the chosen spending function must sum to the total alpha; if it does not, the boundaries were not computed from a spending function.
• Plot the fraud rate by transaction month against dispute maturity. If the curve slopes down at the right edge on historical data where nothing changed, that slope is maturity, not effect.
• Compare the effect estimate at the crossing look with the estimate at the planned maximum sample on historical tests; the gap is the stopping bias you owe the sponsor.

Follow-up

• Under an O'Brien-Fleming boundary you cross on day three. What do you say about the effect size, and why is the naive point estimate biased?
• How would you set the stopping rule for the fraud guardrail given that its true value is not observable inside the test window?
• The team argues that they are only looking, not deciding, so peeking is harmless. Under what precise condition is that true, and how would you verify it?

Evaluate a staggered country rollout without a randomised control

Hard · difference-in-differences · synthetic control · staggered adoption

A step-up authentication rule was enabled for ecommerce traffic in three issuer_country markets on three different dates across five months; twelve comparable markets never received it. You have fct_payment_authorization and fct_card_dispute at daily grain and cannot randomise. Estimate the effect on the dollar-weighted authorization approval rate and on matured fraud basis points, name the estimator and why the obvious one is wrong here, and justify your inference given that there are only three treated clusters.

Approach

1. Rule out the default first. A two-way fixed effects regression with a single post-times-treated indicator is not valid under staggered adoption with effects that vary over time, because it constructs comparisons that use already-treated markets as controls for later-treated ones and can assign negative weights to some of those comparisons, so the coefficient need not lie inside the range of the true effects.
2. Use an estimator built for staggered timing: Callaway and Sant'Anna group-time average treatment effects, or the Sun and Abraham interaction-weighted estimator, restricting the comparison group to the twelve never-treated markets and aggregating into an event study indexed on time since adoption.
3. Defend parallel trends with evidence, not a single test. Show the pre-period leads with their intervals and state their magnitude relative to the post-period effect, because pre-trend tests are underpowered and failing to reject is not evidence of parallelism. Pre-register the maximum pre-period lead you would tolerate before abandoning the design.
4. Fix the inference problem directly. With three treated clusters, cluster-robust standard errors are severely anti-conservative. Fit a separate synthetic control for each treated market against the twelve donors, with non-negative weights summing to one fitted on pre-period outcomes and predictors, then use in-space placebo permutation and the ratio of post-period to pre-period root mean squared prediction error as the test statistic. With twelve donors the smallest attainable one-sided permutation p-value is 1 / 13, about 0.077, so say that before anyone asks for p below 0.05.
5. Split the two outcomes by maturity. The dollar-weighted approval rate is observable immediately and can be read on the full post window. Matured fraud basis points require at least 120 days of dispute maturity from the transaction month, so the fraud event study must terminate at the last matured month and the immature months must be marked incomplete rather than plotted as low.
6. Enumerate and test the confounds specific to this setting: network mandates or regulatory deadlines that landed on the same dates, merchant-side changes in the treated markets, currency mix and settlement FX, and seasonality. Exclude donors subject to a concurrent mandate, and confirm that donors and treated markets do not share an acquirer whose outage would move both.

Follow-up

• Your synthetic control for one market puts 80 percent of the weight on a single donor. What do you do, and what does the leave-one-out check tell you?
• The approval rate rises and matured fraud basis points also rise. How do you express the trade-off in a single decision-grade number, and what does that require you to assume about the value of a declined good transaction?
• Suppose a fourth market adopts mid-analysis. How does that change the estimator, the donor pool and the permutation inference?

A single day of settled volume comes in thirty percent low

Medium · pipeline failure · null rates · data freshness

Yesterday's settled volume is 30 percent below the same weekday across the previous eight weeks, while authorization counts and approved counts for the day look normal. You have fct_payment_authorization with requested_at, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency, created_at and updated_at, plus warehouse load metadata. Decide within the hour whether this is a real drop or a load problem, and say what you would post to the on-call channel in either case.

Approach

1. Check the data before the business. Compare max(updated_at) and row counts per load partition for the affected day against the prior eight same weekdays; a truncated load usually shows as a count deficit confined to a specific hour range.
2. Measure null rates per column by requested_at date. Authorization fields present while settled_at and settlement_amount_minor are null points at the settlement feed; a uniform deficit across all columns points at ingestion.
3. Separate late arrival from loss. Settlement lands after authorization by design, so re-measure the same day 24 and 48 hours later. A deficit that closes on the next load is lag, not a drop.
4. Reconcile against the independent source: tie the day's settled total to the settlement ledger or acquirer file. If the ledger has the money and the warehouse does not, the business number was never wrong.
5. Only after those four steps consider a business explanation, and then only if a segment cut produces a coherent story rather than a flat haircut spread evenly across every merchant and channel.
6. Close by splitting the alert: a freshness and completeness check on the settlement feed, separate from the volume alert, so the next occurrence is classified automatically.

Follow-up

• How would you make the daily volume alert immune to settlement lag without also hiding a genuine drop?
• What is the right way to restate a published number after a backfill lands?
• Which check catches a partial load that is uniform across the day rather than concentrated in one window?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

How do you handle data cleaning tasks when dealing with noisy or inconsistent data sources?

Medium · behavioural and stakeholder questions

Approach

1. Pick a story where you drove the decision, not one where you observed it.
2. Close with what you would do differently, concretely.
3. State the situation in two sentences and spend the rest on your reasoning.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that project again?

How would you explain a complex statistical result to a non-technical stakeholder?

Medium · behavioural and stakeholder questions

Approach

1. Name the disagreement or constraint, and how you resolved it with evidence.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Close with what you would do differently, concretely.

Follow-up

• How did you know the outcome was caused by your change?
• What did you decide not to do, and why?

Retract a published number after finding a currency bug

Medium · error disclosure · minor units · process repair

Two weeks ago you published an interchange and fraud analysis that summed amount_minor across fct_payment_authorization without converting currencies. Minor units are not two decimals everywhere: some currencies carry none and some carry three, so the sum has no interpretation. A pricing decision is already in flight on the back of it. You now have corrected figures. Produce the retraction: what you send, to whom, in what order, and what you change in the process so this class of error is caught next time rather than trusted next time.

Approach

1. Size the error before announcing it, because saying the number is wrong without a magnitude and a direction forces every reader to assume the worst case.
2. Check whether the conclusion actually flips: if the ranking that drove the pricing decision is unchanged, that belongs in the first sentence beside the correction rather than buried at the end.
3. Tell the person acting on it first and directly, then the wider distribution, using the same text, so nobody learns about it secondhand.
4. Write the correction as four parts: the old number, the cause in one clause, the effect on the pending decision, and the new number. Leave out self-flagellation, which makes the reader do emotional work instead of acting.
5. Fix the class rather than the instance: a rule that a sum over amount_minor either groups by transaction_currency or passes through both conversion steps, exponent scaling and then a dated rate into one named reporting currency, plus a standing reconciliation of the settled subset to the settlement ledger inside each settlement_currency.

Follow-up

• The corrected figures do not change the decision. Do you still send the correction, and what does that choice signal?
• What automated check would have caught this, where would it live, and what would it cost in false alarms?