Interview guides · Fullpower

Fullpower Data Scientist Interview Questions & Guide 2026

Causal inference drills for Fullpower data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

As a Data Scientist at Fullpower, you will play a pivotal role in harnessing data to drive insights and innovation across the company. This position develops algorithms that enhance Fullpower's products, improve user experiences, and ultimately increase business value. You will work within dynamic teams focused on projects that involve large-scale data analysis, predictive modeling, and machine learning, all aimed at optimizing sleep and health outcomes through technology.

Product-sense cases reward reasoning from a mechanism to a testable prediction. Reciting every metric you can name reads as pattern matching; naming the single quantity that would move if your explanation were true reads as thinking.

Focus areas

Decompose expected loss into PD, LGD, EAD
Report only matured cohorts for loss metrics
Separate authorization, settlement and dispute outcomes cleanly

Overview

Your work will significantly impact various products at Fullpower, such as sleep tracking applications and wellness platforms. You will tackle complex problems and contribute to strategic decisions that shape the future of Fullpower's offerings. Given the scale of the data Fullpower handles and the importance of analytical rigor in its solutions, the Data Scientist role is demanding and gives you an opportunity to make a meaningful difference in users' lives.

The Fullpower interview process

Fullpower candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Interview

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Problem-Solving Evaluation

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Where candidates lose points

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Reading the most recent months of fraud and dispute rates as final

Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, and several reason codes run considerably longer, so the disputes belonging to a recent transaction month have simply not been filed yet. Any chart attributed by transaction date therefore slopes down at the right edge regardless of what is happening. The fix is to report only matured cohorts, or to apply development factors estimated from completed months and to show the estimate as an estimate.

Over-explaining the method and under-explaining the implication

Lead with the answer and what you would do about it, then give the approach when asked. Roughly one sentence of method per three of implication is the right ratio for a stakeholder-facing answer; the interviewer already knows what a regression is.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Questions & practice

Write a function to calculate the mean and standard deviation of a list of numbers.

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

What are some common metrics for evaluating model performance?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Bootstrap a fraud loss rate that clusters within merchant

Medium · bootstrap · clustered resampling · heavy tails

You have a per-transaction frame with auth_id, merchant_id, settled_amount_reporting and net_loss_reporting, both already in one reporting currency. Most rows carry zero loss, a few carry large ones, and losses cluster within merchant. Using only numpy's random generator and no resampling helper from any library, write a bootstrap that returns a 95 percent interval for net fraud loss in basis points of settled volume, resampling merchants with replacement and taking all rows belonging to each drawn merchant. Also produce the naive row-level interval and state which you would report.

Approach

State the estimator before writing it: total net loss divided by total settled volume, times 10,000. It is a ratio of sums, so each replicate recomputes both sums. Averaging per-transaction loss rates instead would weight a five-unit transaction like a five-thousand-unit one.
Pre-aggregate loss and volume to merchant level once. For a ratio of sums, drawing merchants and taking all their rows is arithmetically identical to drawing merchant-level (loss_sum, volume_sum) pairs, so a replicate becomes one integer draw plus two vectorised sums rather than a groupby inside the loop.
Draw B replicates of M merchant indices with replacement, where M is the observed merchant count, compute the ratio per replicate, and take the 2.5th and 97.5th percentiles. Say explicitly that this is a percentile interval and that BCa would correct the skew-induced bias if the decision is close.
Repeat with independent row draws for the naive interval and compare widths on the same replicate count.
Report the clustered interval. Rows within a merchant share an acceptance profile, a category code and a fraud exposure, so they are not independent, and the row-level interval understates variance by roughly the design effect.

Worked solution (30 min)

Compute the point estimate directly on the full data and keep it for comparison.
Aggregate to merchant-level loss and volume arrays, record M, and set B to 2,000 with a seeded numpy Generator.
In a vectorised loop, draw integer indices of shape (B, M), index both arrays, sum along axis 1, and take the ratio times 10,000.
Repeat for the row-level version using the per-transaction arrays and N draws.
Take the 2.5 and 97.5 percentiles of each replicate array and report both intervals alongside the point estimate.

Expected result: Two intervals in basis points sharing the same point estimate. The clustered interval is materially wider, commonly two to four times, and is right-skewed because a heavy merchant can be drawn more than once in a replicate.

Verify it yourself

The point estimate equals the direct ratio on the full data and lies inside both intervals.
A synthetic input where every merchant holds exactly one transaction makes the clustered and naive intervals coincide; this is the test that the resampling unit is wired correctly.
Raising B from 2,000 to 10,000 shifts the endpoints by less than the reporting precision; if it does not, a handful of merchants is driving the tail and the estimate needs that caveat.
Replicate ratios never fall outside the range implied by drawing only the lowest-loss or only the highest-loss merchants.

Follow-up

Your clustered interval is three times wider. How do you explain that to someone who wanted a tighter number?
One merchant accounts for 40 percent of losses. What does that do to the interval, and what would you do about it?
How does this change if the question is whether two months differ rather than what this month's rate is?

What data structures would you use for efficient data retrieval?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Can you demonstrate your proficiency in Python or R with a sample code challenge?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Reconcile captured authorizations against the daily settlement total

Easy · reconciliation · currency · rounding

fct_payment_authorization holds captured_amount_minor in transaction_currency, and settlement_amount_minor in settlement_currency with settlement_fx_rate applied at settlement rather than at authorization. The rate is quoted in major units of settlement_currency per major unit of transaction_currency, and dim_currency.minor_unit_exponent carries the ISO 4217 exponent for each code (0, 2 or 3 depending on the currency). Produce a daily reconciliation: for each settled_at date and settlement_currency, return settled_count, total settlement_amount_minor, and the sum of captured_amount_minor converted into settlement minor units. Flag any date and currency pair whose two totals differ by more than one minor unit per settled authorization. Do not sum amounts across currencies anywhere in the output.

Approach

Restrict to rows that actually settled: settled_at is not null and settlement_amount_minor is not null, which is a smaller population than captured rows because a capture can still be in flight.
Truncate settled_at to a date with an explicit time zone so the cut matches the ledger's cut, since settled_at is timestamptz and date_trunc on timestamptz silently uses the session time zone.
Join dim_currency twice, once on transaction_currency and once on settlement_currency, so both exponents are on the row. Minor units are not a common scale: a bare captured_amount_minor * settlement_fx_rate is correct only when the two exponents are equal, and a zero-decimal currency settling into a two-decimal one is wrong by a factor of 100.
Convert per row as ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle)) — minor units to major in the transaction currency, apply the major-per-major rate, then back to minor units in the settlement currency. The collapsed form ROUND(captured_amount_minor::numeric * settlement_fx_rate * POWER(10::numeric, exp_settle - exp_txn)) is the same expression. Round per row and then sum, not SUM(...) * an average rate, because the rate varies row by row and rounding per row is what the settlement file did.
Group by the settlement date and settlement_currency together, never by date alone, and carry the currency into every output column name or row.
Compare the two totals with a tolerance scaled by settled_count, since per-row rounding accumulates linearly in the number of rows rather than being a fixed constant.

Worked solution (25 min)

Filter to settled rows and derive settlement_date from settled_at with an explicit time zone.
Join dim_currency on transaction_currency and again on settlement_currency to pick up exp_txn and exp_settle; fail the run if either is null rather than defaulting to 2.
Aggregate by settlement_date and settlement_currency: COUNT(*), SUM(settlement_amount_minor), and SUM(ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle))).
Add a derived difference column and a boolean flag where ABS(difference) > settled_count.
Order by the flag first and then by settlement_date so the exceptions surface at the top.

Expected result: One row per settlement_date and settlement_currency, carrying settled_count, settlement_total_minor, converted_capture_total_minor, their signed difference, and a boolean exception flag. On clean days the difference is small relative to settled_count and has no consistent sign. Pairs with unequal exponents reconcile exactly as well as equal-exponent pairs; if they do not, the exponent terms are missing rather than the FX rate being wrong.

Verify it yourself

The difference divided by settled_count stays well under one minor unit on unflagged days, which is the expected size of per-row rounding.
Restricted to rows where exp_txn = exp_settle, the exponent-aware conversion and a bare captured_amount_minor * settlement_fx_rate agree exactly; restricted to rows where the exponents differ by 2, the bare form is smaller by exactly a factor of 100. That is the direct test that the dim_currency join is load-bearing.
Re-running with SUM over all currencies at once produces a number that changes when a zero-decimal currency's volume moves, which demonstrates why the cross-currency total is meaningless.
settled_count summed over all dates equals COUNT(*) on the same settled-row filter with no grouping.

Follow-up

A partial capture means captured_amount_minor is less than amount_minor. Where does that show up in this reconciliation, and where does it not?
On one currency pair the converted total is consistently about one hundredth of the settlement total, on every date, while the other pairs reconcile. Which two columns do you inspect first, and what single change fixes it?
The rate is documented as major-per-major. If a feed started publishing it minor-per-minor instead, which pairs would still reconcile and which would break?
How would you present a total across currencies to a finance partner who has asked for one number?

How do you prioritize tasks when faced with multiple deadlines?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you present your findings to a non-technical audience?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Given a dataset, how would you approach analyzing it to derive business insights?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

What steps would you take to identify the cause of a significant drop in user retention?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How would you design an experiment to test a new feature in one of our apps?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Decide the analysis before seeing data, including how long it runs and when you look.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer would contain.
Work from the decision backwards to the evidence you would need.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Design an authorization approval rate a team cannot game

Easy · approval rate · metric gaming · mix shift

A payments team is given authorization approval rate as its quarterly target, defined as approved authorizations over all authorizations in fct_payment_authorization. Using auth_id, card_token_id, merchant_id, amount_minor, transaction_currency, auth_result, is_reversal, parent_auth_id, channel, issuer_country and requested_at, list three ways the team can raise that number without a single additional customer purchase succeeding. Then write the definition you would put on the dashboard instead, with its exclusions, and name the one guardrail you would always read beside it.

Approach

Enumerate the rows in the table that are not purchases: zero-amount account verifications, reversals (is_reversal = true), incremental authorizations (parent_auth_id not null) and merchant retries of a declined attempt. Each is a lever on the numerator or the denominator that no customer would ever notice.
Name the mix levers. Recurring and card-present traffic approve at higher rates than ecommerce and cross-border, so routing changes or a marketing push into an easy channel move the blended rate with no change in decisioning quality at all.
Work the retry arithmetic rather than asserting it. A declined purchase that is retried three times and succeeds on the fourth contributes one approval and four denominator rows, which is below a healthy blended rate, so suppressing retries raises the published number while some purchases that would have succeeded on a retry now simply fail.
Replace the definition: dollar-weighted, retry chains collapsed to a single attempt per (card_token_id, merchant_id, amount_minor) within a 15-minute window, excluding reversals, zero-amount verifications and incremental authorizations, with all amounts converted into one reporting currency before summing because amount_minor is in the minor unit of transaction_currency and that unit is not always two decimals. Cut by channel and issuer_country so mix shift shows up as mix shift.
Guardrail: net fraud loss in basis points of settled volume, read only on transaction months with at least 120 days of dispute maturity. The fastest way to approve more is to decline less risk, and that bill arrives two to four months later.

Worked solution (25 min)

Compute the naive rate for one week: approved rows over all rows.
Apply the exclusions one at a time (zero-amount, is_reversal, parent_auth_id not null), recording the rate after each, so you can say what each row type was worth to the headline number.
Add a retry-collapsing CTE keyed on (card_token_id, merchant_id, amount_minor) over a 15-minute window, keeping the last attempt in each chain, and recompute.
Recompute dollar-weighted in a single reporting currency, then split by channel and issuer_country.
Write the final one-line definition with its exclusions, plus the guardrail sentence.

Expected result: Four rates for the same week (naive count, count after exclusions, count after retry collapsing, dollar-weighted after both), with the largest single gap attributed to a named row type, and a published definition that states every exclusion and the currency it reports in.

Verify it yourself

Excluding zero-amount verifications should lower the rate, since those rows approve at a higher rate than purchases. If it raises the rate, confirm that the amount_minor = 0 filter is not also catching rows with a null amount.
Collapsing retry chains raises the rate and shrinks the denominator. The numerator falls only by the number of chains that contained more than one approval.
Declines skew toward higher-value, cross-border and card-not-present transactions, so expect the dollar-weighted rate to sit below the count-weighted one. If it sits above, check for amounts summed across currencies before believing it.

Follow-up

The blended rate is flat but ecommerce cross-border fell 200 basis points. How do you present that?
Why weight by amount rather than by count, and when does the weighted version mislead?
Your 15-minute retry window merges two genuine purchases of the same amount at the same merchant. How would you bound that error?

Approval rate rose in every band yet fell overall

Medium · mix shift · simpsons paradox · segmentation

Monthly application approval rate on fct_loan_application fell from 62 to 57 percent. Cut by bureau_score band, the rate rose in every band, including the null-bureau band. Columns: application_id, channel, submitted_at, requested_amount_minor, declared_annual_income_minor, bureau_score, model_pd_12m, model_version, policy_rule_hits, decision, decided_by, decision_at. The denominator is decision in ('approve','decline'). Explain the arithmetic, quantify how much of the five-point fall is mix versus within-band movement, and say what you would tell the team that owns acquisition.

Approach

Confirm the paradox is real rather than a banding artefact. Rebuild the bands on fixed cutpoints taken from the earlier period, because quantile bands re-cut each month move with the population and can manufacture this pattern on their own.
Compute the exact decomposition rather than describing it: within = sum of w_i0 * (r_i1 - r_i0), mix = sum of r_i0 * (w_i1 - w_i0), interaction = sum of (w_i1 - w_i0) * (r_i1 - r_i0). The three terms sum identically to the change in the blended rate, so the report can state the split.
Attribute the weight change by cutting the same fixed bands by channel. Keep null bureau_score as its own band; a thin file is a population signal, not missing data to be imputed away.
Check whether the new arrivals also changed the population inside a band, by comparing requested_amount_minor and declared_annual_income_minor distributions within one band across the two months.
Deliver two numbers rather than one story: policy is looser in every band, and the funnel is being fed a different population. Those have different owners and different fixes.

Follow-up

If the new channel is profitable at its own approval rate, is the blended fall a problem at all?
How would you present this so that nobody reads the blended series unaccompanied again?
What breaks if you fix the mix by reweighting to a frozen band distribution every month?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

How do you handle constructive criticism?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Turn a one-line fraud-number request into a scoped brief

Easy · scoping · metric definition · denominators

A stakeholder messages: what is our fraud rate, and is it going up? You have fct_payment_authorization, fct_card_dispute and dim_customer. At least four defensible answers exist: count-weighted or value-weighted, attributed to the transaction month or to the dispute filing month, and gross or net of recoveries and successful representments. You get one reply before someone else produces an uncaveated number. Write that reply: the clarifying questions you ask, the single default you will produce if nobody answers, and what the default excludes.

Approach

Establish the decision behind the question first, because a risk-rule change, a board number and a merchant contract negotiation need different denominators, and asking which one is not stalling.
Offer a short menu rather than an open question: a stakeholder can choose between two named options but cannot specify a denominator from scratch.
Commit to a default so the reply is useful even if nobody answers, for example net fraud loss in basis points of settled volume, attributed to the requested_at month, matured months only.
State the exclusions in the same breath as the default: non-fraud dispute categories, transaction months with less than 120 days of maturity, and first-party abuse that arrives coded as consumer_dispute.
Give a delivery time for the default and a longer one for the fuller cut, so the choice between them carries a visible cost.

Follow-up

They come back wanting it by merchant for a contract negotiation. What changes in the definition and in the maturity rule?
How would you separate first-party abuse from third-party fraud in this data, and what would you refuse to conclude from the split?

Disagree with a product manager over an approval-rate target

Medium · metric design · denominators · influence without authority

A product manager proposes a quarterly goal of raising card authorization approval rate by 150 basis points, measured as approved authorizations divided by all authorizations in fct_payment_authorization. You believe that metric can be hit with no customer benefit, because merchant retry chains, zero-amount verification authorizations, incremental authorizations and reversals all sit in the denominator, and declines skew toward high-value cross-border ecommerce. You support the underlying goal. In one working session, change the metric without killing the initiative, and name the guardrail you would accept.

Approach

Separate the goal from the metric out loud and agree with the goal first, so the disagreement stays narrow and technical rather than becoming positional.
Demonstrate the failure rather than asserting it: compute the proposed metric and the dollar-weighted collapsed version over the same recent window, and find a period where they moved in opposite directions.
Propose the replacement precisely: sum of approved amount_minor over sum of attempted amount_minor, after collapsing retries to one attempt per card_token_id, merchant_id and amount_minor within a 15-minute window, excluding is_reversal rows and zero-amount verifications, with everything converted to one reporting currency before summing.
Attach the guardrail that makes the target honest: matured first-chargeback rate and net fraud loss in basis points of settled volume, read only on transaction months carrying at least 120 days of maturity.
Give the product manager something back: the replacement metric cuts cleanly by channel and issuer_country, which makes a roadmap of merchant-specific and authentication fixes legible in a way the blended rate never was.

Follow-up

How do you identify a retry chain when the merchant varies the amount slightly between attempts?
The product manager wants a weekly read on the guardrail. What is the earliest defensible signal, and how do you label it?