Interview guides · Deloitte

Deloitte Data Scientist Interview Questions & Guide 2026

Data scientist interview prep for Deloitte: original practice on experiment design, SQL and metric reasoning, with worked solutions.

The 60-second brief

As a Data Scientist at Deloitte, you sit at the intersection of advanced analytics, enterprise consulting, and digital transformation. You will drive the design and delivery of cutting-edge artificial intelligence, machine learning, and optimization solutions for a diverse portfolio of global clients. This position requires you to translate ambiguous business challenges into structured analytical frameworks, architect robust data products, and communicate high-impact recommendations directly to executive stakeholders.

Allocate prep to your weakest link rather than your favourite topic. Of the three things that usually gate the outcome (SQL that is correct under messy joins, sound reasoning about experiments, and structured framing of an open-ended problem), candidates tend to over-invest in modelling theory and under-invest in framing.

Focus areas

Cluster inference at the account, not the engagement
Separate bookings, recognised revenue and collected cash
Compute utilisation against a defended availability denominator

Overview

Your day-to-day impact involves more than just writing code; you will shape the technological and strategic roadmap for enterprise clients navigating complex operational shifts. Working within multidisciplinary teams across cloud environments, you will tackle high-stakes problems ranging from supply chain optimization and financial forecasting to natural language processing and computer vision applications. The role demands technical mastery paired with strong consultative presence, giving you a unique platform to influence major business outcomes across industries.

Expect an intellectually rigorous environment where continuous learning and collaboration are the norm. Whether you are building scalable MLOps pipelines or designing enterprise analytics platforms, you will be supported by a culture that values innovation and professional growth. Success here means balancing technical depth with business acumen, delivering solutions that make a measurable impact for clients and society alike.

The Deloitte interview process

Deloitte candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Interview

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Team Fit Interview

This conversation decides whether you can be handed a problem nobody has finished defining and left alone with it for a few weeks. The manager is listening for how you behave when the brief is thin: what you clarify before starting, and what you settle on your own rather than escalating. Most candidates over-index on technical depth here and under-describe the decisions they actually owned. Say who wanted the work, what you chose not to do, and where you would have stopped and asked. A clean account of your own judgement carries this round further than a longer project list.

What to demonstrate

Whether you can name a decision that was yours alone, as opposed to one the team arrived at
How you respond to a request that arrives with no success metric attached to it
Whether the effort you estimate for a piece of work matches the work you just described doing
What you escalate, and how long you sit on a problem before you do

How to prepare

For each project you plan to raise, write one sentence saying what would not have happened if you had not been on it, and check that the sentence is about an outcome rather than an artefact
List the decisions in your last project that were genuinely yours, and for each one write down the option you rejected and why
Prepare the project that went badly: the point at which you knew, who you told, and what it cost before it was caught

Where candidates lose points

Computing days-to-pay or proposal cycle time over completed records only

At any snapshot date, invoices that have already been paid are disproportionately the fast ones, and proposals that already have a decision are disproportionately the quick ones. Averaging over the completed set alone biases both numbers downward, and the bias grows exactly when the business is deteriorating, because the slow cases are the ones still open. Unpaid and undecided records are right-censored; use Kaplan-Meier or a restricted mean up to a fixed horizon, and never fill paid_at with a placeholder.

Pooling margin, realisation or overrun across pricing models

Fixed-fee margin falls with hours worked; uncapped time-and-materials margin rises with hours worked; retainer margin depends on neither. A quarter in which the firm sells more fixed-fee work will show a margin change caused entirely by mix, not by delivery performance, and the aggregate can move in the opposite direction to every individual pricing model. Always stratify by fct_engagement.pricing_model before comparing periods, and report the mix shift alongside the within-stratum change.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Crediting a treatment for regression to the mean

Selecting a group because it is extreme (lowest-engagement users, accounts having their worst month, the bottom decile of a score) moves that group's expected next-period value back toward the average even under no treatment, by exactly as much as the selecting measure is imperfectly correlated with its own later value. Compare against units that met the same selection rule and went untreated, or use two pre-periods so the bounce-back is visible before the intervention starts. A pre-post number on a group chosen for being extreme measures the selection rule, not the treatment.

Questions & practice

What is the difference between parametric and non-parametric methods, and when would you choose one over the other?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Say what the estimate is of, and over what population it generalises.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

What strategies do you use to prevent overfitting when working with high-dimensional tabular data?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.
Check what information would not exist at prediction time, and exclude it.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

Implement billable utilisation against a defended availability denominator

Medium · metric implementation · denominators · proration

Implement billable utilisation for one calendar month at consultant grain, then roll it up to practice_area. Inputs: time_entries (consultant_id, work_date, hours, is_billable, charge_code, status), consultants (consultant_id, practice_area, fte_fraction, home_region, is_billable_role, hire_date, termination_date) and holidays (region, holiday_date). Numerator is approved hours with is_billable true. Denominator is scheduled workdays in the month, clipped to [hire_date, termination_date] and net of that region's holidays, times 8 times fte_fraction, minus approved leave hours. Roll up as sum(numerator) / sum(denominator).

Approach

Build the workday set per region first: business days in the month minus that region's holiday_dates. State the Monday-to-Friday assumption out loud, because a region with a different working week makes np.busday_count wrong rather than approximate.
Prorate by clipping each consultant's interval to [max(month_start, hire_date), min(month_end, termination_date or month_end)] and counting workdays inside the clipped interval, so a mid-month start produces a smaller denominator instead of a fake dip in the ratio.
Subtract approved leave hours (charge_code in leave_paid, leave_unpaid) only for leave falling on days already counted as workdays; leave logged on a public holiday would otherwise be subtracted twice and can push a denominator negative.
Filter to is_billable_role, then drop consultant-months whose denominator is zero or negative (full-month leave, a hire on the last day) rather than emitting inf or NaN, and return the dropped count as part of the result.
Roll up by summing numerators and denominators separately. Averaging per-person ratios gives a 0.2 FTE consultant the same weight as a full-time one and produces a practice number that no individual reconciles to.

Worked solution (35 min)

Per region, workdays = np.busday_count(month_start, month_end + 1 day, holidays=region_holidays); per consultant, recompute on the clipped [start, end] interval so proration and holidays compose.
leave = time_entries[status=='approved' & charge_code.isin(['leave_paid','leave_unpaid']) & work_date.isin(region workdays)].groupby('consultant_id').hours.sum().
denominator = workdays_clipped * 8 * fte_fraction - leave, reindexed to the billable-role consultant list and filled with 0 leave where absent.
numerator = time_entries[status=='approved' & is_billable].groupby('consultant_id').hours.sum(), reindexed and zero-filled.
Drop rows with denominator <= 0 (counting them), compute utilisation_pct = 100 * numerator / denominator, then groupby practice_area and divide summed numerator by summed denominator.

Expected result: A consultant-month frame carrying numerator, denominator and utilisation_pct, a practice-area frame whose ratio equals sum(numerator)/sum(denominator), and an explicit count of consultant-months excluded for a non-positive denominator.

Verify it yourself

A full-time consultant with no leave in a 21-workday month has denominator exactly 168.0.
Every consultant's denominator is at most workdays * 8 * fte_fraction, with equality only when leave is zero.
The practice-area ratio equals the denominator-weighted mean of the per-consultant ratios, and sits between their min and max.
Summed numerators across practice areas equal total approved billable hours for in-scope consultants.

Follow-up

Utilisation rose two points this month while realisation fell. Reconcile those two movements with one mechanism.
A regional lead says their team looks four points below another region. What do you check before you answer, and in what order?

1–2 sentences introducing the category and what it tests. This category tests your proficiency in writing efficient queries and manipulating large datasets to extract actionable insights.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

How do you select the appropriate statistical test when comparing conversion rates across multiple non-normal distributions?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Build a client cohort revenue table from invoice lines

Easy · cohorts · date truncation · revenue recognition

dim_client holds client_id and first_engagement_date. fct_engagement maps engagement_id to client_id. fct_invoice_line holds engagement_id, line_type, period_start_date, amount_usd and status; recognised fees are lines with line_type IN ('fees','milestone','credit_note') and status <> 'draft', and credit notes carry negative amount_usd. Return one row per (cohort_month, months_since_first) giving distinct clients billed and net recognised fees. cohort_month is the month of first_engagement_date; months_since_first is the whole-month gap between period_start_date and first_engagement_date. Long format, no pivot. Decide explicitly what to do with clients whose first_engagement_date is NULL.

Approach

Filter fct_invoice_line to recognised lines first (line_type, status <> 'draft'), and key recognition on period_start_date rather than issued_at or paid_at. Billing period, issue date and payment date are three different clocks, and mixing them produces a cohort table nobody can reconcile against finance.
Join invoice lines to fct_engagement to reach client_id, since invoice lines carry engagement_id only, then to dim_client for first_engagement_date.
Truncate both dates to month, then compute months_since_first as 12 * (year(bill_month) - year(cohort_month)) + (month(bill_month) - month(cohort_month)). Doing it on truncated months is what keeps a client billed on day 2 of the third month in bucket 2 instead of bucket 1.
Aggregate to (cohort_month, months_since_first) with SUM(amount_usd) and COUNT(DISTINCT client_id), keeping credit_note lines inside the sum so the figure is net rather than gross.
Handle NULL first_engagement_date as an explicit exclusion and report how many clients and how much revenue that removes, rather than letting the join drop them silently. That exclusion is why the output's total is smaller than the filtered line set's total, so carry the excluded figure as a named number rather than discovering the gap later.

Worked solution (25 min)

Write the recognised-lines CTE with the line_type and status predicates, and check its row count against the unfiltered table so you know how much you excluded.
Join through fct_engagement to dim_client and confirm the join does not multiply rows: a line has exactly one engagement, an engagement exactly one client.
Add the two date_trunc expressions and the integer month arithmetic, then eyeball ten rows against their source dates.
Group and aggregate, then sort by cohort_month and months_since_first.
Run a separate count of clients with NULL first_engagement_date, and a separate sum of their recognised fees, and note both beside the output.

Expected result: A long-format table of (cohort_month, months_since_first, active_clients, net_fees_usd), plus two scalars reported beside it: the count of clients with a NULL first_engagement_date and the recognised fees attached to them. months_since_first is a non-negative integer on every row; a negative value means an invoice period precedes the recorded first_engagement_date, which is a data-quality finding to report, not something to clamp to zero.

Verify it yourself

SUM(net_fees_usd) over the whole result equals SUM(amount_usd) over the filtered lines whose engagement resolves to a client with a non-NULL first_engagement_date. It does not equal SUM(amount_usd) over the full filtered line set, which is larger by exactly the excluded NULL-cohort revenue: add that separately reported figure back and the two agree. Any remainder after that is a line whose engagement_id has no matching row in fct_engagement, which is a referential finding to name rather than absorb.
COUNT(DISTINCT client_id) at months_since_first = 0 within a cohort_month is less than or equal to the number of clients whose first_engagement_date falls in that month.
Zero rows have a negative months_since_first; if any appear, name the clients rather than filtering them out.
Re-running with period_end_date in place of period_start_date moves only lines whose billing period spans a month boundary.

Follow-up

How would you turn this table into a 12-month net revenue retention ratio, and what exactly becomes the denominator?
One account contributes 40 percent of a cohort's month-13 fees. How do you present the cohort so a reader sees that before they act on the ratio?
If subsidiaries roll up to parent_client_id, which first_engagement_date defines the cohort, and does the rollup change the cohort a subsidiary lands in?

Explain the architecture and attention mechanism behind transformers and self-attention models.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you design a recommendation engine for an enterprise retail client, and what metrics would you track to measure its success?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

If you were tasked with launching a new predictive maintenance feature for industrial machinery, what user and system metrics would you prioritize?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you approach designing a product metric framework for a digital transformation initiative?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Design an A/B test for a new enterprise software dashboard layout, detailing your sample size calculation and primary metrics.

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

Explain how you would run an experiment when traditional randomization is impossible due to cluster constraints.

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

Define success for a rate-card increase before any rollout

Hard · pricing · decision threshold · clustering · power

A 6% rate-card increase is proposed for one practice area next quarter. Before it ships, define what success means. Available: fct_proposal (expected_value_usd, stage, loss_reason, is_competitive, submitted_at, decided_at, practice_area), fct_engagement (pricing_model, contract_value_usd, practice_area, client_id) and fct_invoice_line (line_type, amount_usd, billed_hours). Name the primary metric, the guardrail that genuinely opposes it, the unit of analysis, and the numeric threshold that would reverse the decision. State how many account-quarters are needed to detect an effect large enough to change the decision, and what you do if that exceeds what the practice has.

Approach

Make the primary realised revenue per billable hour, not the list rate. The increase can be conceded straight back through discount and show up as no change at all. Numerator: SUM(amount_usd) on line_type IN ('fees','milestone'). Denominator: SUM(billed_hours) over the same lines.
Name the opposing guardrail: value-weighted competitive win rate in that practice area, with the share of losses carrying loss_reason = 'price' as the supporting diagnostic. Raising price lifts the primary and depresses the guardrail by construction, so the rollout pays off only if revenue per hour rises faster than volume falls.
Derive both break-evens before looking at outcome data. Revenue is preserved while the volume loss stays under 6 / 106, roughly 5.7%. For margin, write p for realised price per billable hour and c for avoidable cost per billable hour, and require c < p so the baseline margin is positive. Setting post-change margin (1 - L) x (1.06p - c) equal to pre-change margin (p - c) gives 1 - L = (p - c) / (1.06p - c), so the break-even volume loss is L = 0.06p / (1.06p - c). That is strictly above 5.7% whenever c > 0, collapses to 5.7% at c = 0, and stays below 100% as long as c < p; a practice already below cost at the old rate is not a pricing question. It holds only where c is genuinely avoidable on the volume you lose; on a salaried bench that nobody stands down or resells, no cost is saved and the margin break-even falls back to the revenue one. Report both ends and name the assumption the threshold uses.
Fix the unit and confront the identification problem. Assignment is at practice_area while the outcome sits at engagement or account, so cluster at client_id. With one treated practice area there is a single treated cluster and a conventional difference-in-differences is not identified; propose a synthetic control over the other practice areas with the donor-pool condition stated, or a pre-registered before-and-after whose parallel-trends check runs on the pre-period with a named failure action.
Run the power calculation before proposing the rollout and be willing to conclude the practice cannot detect the effect. Competitive decisions in one practice area number in the tens per quarter, so a 3-point win-rate move needs many quarters. If the requirement exceeds what exists, the honest outputs are a staged rollout with a pre-committed stopping rule, or shipping without measurement and saying so, not a manufactured p-value.

Worked solution (40 min)

Compute realised revenue per billable hour by practice-quarter for eight pre-period quarters from fct_invoice_line fees and milestone lines over billed_hours.
Compute value-weighted competitive win rate per practice-quarter, cohorted on decided_at, denominator stage IN ('won','lost'), is_competitive = TRUE.
Derive the two break-even volume losses: 6 / 106 for revenue, and 0.06p / (1.06p - c) for margin using the practice's realised price and avoidable cost per billable hour, then recompute with c treated as unavoidable to get the other end of the range.
Count competitive decisions per quarter in the practice and compute the minimum detectable win-rate change at 80% power from a two-proportion test, treating that as a floor because clustering raises the real requirement.
Write the rule: threshold, horizon, stopping condition, and the action taken when the power calculation says the effect is undetectable.

Expected result: Primary metric realised revenue per billable hour; opposing guardrail value-weighted competitive win rate. Revenue break-even volume loss 6 / 106, roughly 5.7%. Margin break-even 0.06p / (1.06p - c), strictly larger than 5.7% for any positive avoidable cost per hour and equal to 5.7% when that cost is zero or unavoidable; the reversal threshold sits in that range with the avoidability assumption stated. Unit of analysis engagement or account, clustered at client_id, with the single treated practice area named as the identification problem and a synthetic control or pre-registered before-and-after proposed in response.

Verify it yourself

With c < p, the margin break-even volume loss sits between 5.7% and 100%, equalling 5.7% exactly at c = 0. If you have written (p - c) / (1.06p - c) you are holding the retained-volume fraction, the complement of the answer: at c = 0 it returns 94.3% rather than 5.7%.
The power calculation runs on the count of competitive decisions, not on engagements or invoice lines, since the win-rate guardrail is what triggers the reversal.
The parallel-trends check is specified with a stated failure action, not merely mentioned as a caveat.

Follow-up

Realised revenue per hour rose 4% and win rate fell 2 points on 31 competitive decisions. What do you conclude, and what is your interval on the win-rate move?
How would you separate a price-driven loss from a capability-driven one when loss_reason is entered by the partner who lost the deal?

One region's delivery cost fell eighteen percent for two weeks

Medium · pipeline failure · data completeness · freshness checks

Cost per delivered engagement-week fell about 18% in one region for two consecutive weeks while fees were flat, so reported margin jumped. Source is fct_time_entry (consultant_id, engagement_id, work_date, entered_at, hours, cost_rate_usd, charge_code, status), loaded nightly from a regional timesheet system, joined to dim_consultant for home_region. Determine whether delivery genuinely got cheaper or rows are missing, name the evidence that distinguishes the two, and specify the automated check you would add.

Approach

Check row counts and coverage before any ratio: entries per workday and distinct consultant_id with at least one entry per workday, for the affected region against its own trailing four weeks and against other regions in the same weeks.
Use the shape of the loss to separate causes. A load failure removes whole days or whole consultants, so distinct-consultant coverage collapses while hours per reporting consultant stays normal. A real change lowers hours per consultant with coverage intact.
Split the cost identity: cost = hours x cost_rate_usd. Compute mean cost_rate_usd by level. If rates went NULL, zero or defaulted to a single value, the fault is in the dimension join, not the source extract.
Look for the loader's fingerprint: max(entered_at) per region per day, gaps aligned to a calendar boundary, and whether the missing dates fall on a particular day of week, which points at a scheduler that skipped a run rather than at delivery behaviour.
Confirm by waiting one cycle: a failed load backfills on re-run and the anomaly disappears from the restated weeks; a genuine change persists. Do not publish until that test has run.
Automate the check that would have caught it: per-region, per-workday assertions on row count and on distinct billable consultants covered, with thresholds derived from trailing medians rather than fixed constants, plus a freshness assertion on max(entered_at).

Follow-up

The same load also feeds utilisation. Why did utilisation not show an equally obvious break?
How do you choose the alert threshold so it survives holidays and genuine low-activity weeks?
If the rows never arrive, what do you do with the two weeks already reported to leadership?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

How do you handle multiple testing corrections when evaluating dozens of secondary metrics in an A/B test?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Defending a fixed-fee overrun finding against the selling partner

Easy · metric definition · pricing model mix · stakeholder pushback

Over the trailing four quarters, fixed_fee engagements in one practice area show a median scope overrun ratio of 1.34, computed as approved client_delivery hours in fct_time_entry divided by fct_engagement.contracted_hours, and inception-to-date gross margin nine points below the time_and_materials book. The partner who sold most of that work says the denominator is wrong because change orders were signed, and each change order is a separate row linked by prior_engagement_id. You have ten minutes with the practice lead. Present and defend the finding.

Approach

Say out loud what is being probed: whether a specific methodological objection makes you recompute or makes you repeat yourself. Restate the metric as numerator, denominator and window before defending anything.
Test the objection empirically instead of debating it. Rebuild the denominator by walking the prior_engagement_id chain with a recursive CTE, summing contracted_hours across the original SOW and every extension, and summing approved delivery hours over the same chain.
Report both numbers and say what each one answers: the original-SOW denominator answers whether the team held the scope that was signed, the chained denominator answers whether the firm estimated the total work correctly. Neither is a trick; they are different questions.
Keep the margin claim separate from the overrun claim, and keep it stratified. Fixed_fee margin falls with hours worked while uncapped time_and_materials margin does not, so the nine-point gap is only meaningful within pricing_model, and a shift in the fixed_fee share of fees is reported alongside it.
Close with the decision and a falsifier: if the chained overrun is near 1.0, the problem is scope control on the original SOW, and the check is whether change orders repriced at the standard rate card or at the original blended rate.

Follow-up

The chained denominator narrows the overrun to 1.11. Does your recommendation change, and which number goes in the practice review?
How would you distinguish an estimation problem at sale time from a delivery problem during execution, using only these tables?

Defending your own impact claim without randomisation or clean units

Hard · self-selection · clustered inference · impact measurement

At your review you plan to claim that the realisation dashboard you built recovered 1.4 million dollars. The evidence is that engagement-month realisation rose four points over two quarters among engagements whose leads used it. Adoption was voluntary. There are about sixty client accounts and the top five carry most fees. A new rate card shipped in the same quarter. Write the claim you can defend, the estimate you would actually produce, and what you say when asked for a causal number you cannot get.

Approach

Name the probe: whether you can separate the number you want from the number the data supports, under review pressure, without either inflating it or retreating to saying nothing can be known.
State both identification problems concretely. Voluntary adoption means adopting leads are plausibly the ones who already manage realisation, so the comparison is confounded at the person level. The rate card changes bill_rate_usd, which sits in the realisation denominator, so part of the four-point move is arithmetic rather than behavioural.
Neutralise what you can. Recompute realisation with bill rates snapshotted on work_date, or hold the denominator at the old rate card, so the rate-card change cannot move the metric by construction. Then rerun the comparison.
Get the inference right for the unit count. Cluster at client_id, not engagement, because engagements in one account share a partner, a rate card and a team. With sixty accounts and five carrying most fees, the effective cluster count is far below sixty, so report a wild cluster bootstrap interval rather than plain cluster-robust standard errors, which are biased downward in that regime.
Report both weightings and explain the divergence: an account-weighted estimate describes the typical account, a value-weighted one describes the revenue, and if they disagree a small number of accounts is carrying the result. Then give the decision-relevant sentence: the defensible range, whether its lower bound still clears the build cost, and what a proper staggered rollout would have bought.

Follow-up

The pre-period trends for adopters and non-adopters are not parallel. What do you report then?
You get to design the next rollout. What do you change so the same question is answerable, without randomising individual accounts?