Impact Recruitment Group Data Scientist Interview Questions & Guide 2026

The 60-second brief

At Impact Recruitment Group, the Data Scientist role serves as the bridge between raw, complex data and actionable business intelligence. You are not just building models; you are solving real-world business challenges, often working in domains like retail, marketing, or operational optimization. Your work directly influences product strategy and client outcomes, requiring you to translate technical findings into clear, persuasive narratives for non-technical stakeholders.

Nearly every loop contains a round whose deliverable is a recommendation to someone non-technical. Practise stating a conclusion, the confidence attached to it, and the cost of being wrong in each direction, because that triple is the artifact being graded.

Focus areas

• Compute utilisation against a defended availability denominator
• Correct late-entered timesheets before trending recent weeks
• Cluster inference at the account, not the engagement

Overview

The role demands a balance of high-level analytical thinking and deep technical rigor. You will frequently move from exploratory data analysis and model development to explaining the "why" behind your predictions. Because the organization values practical, business-driven solutions, you will find that the most successful Data Scientists here are those who can balance mathematical purity with the realities of business constraints and time-sensitive deliverables.

The organization prioritizes candidates who demonstrate a strong grasp of fundamental concepts over those who rely solely on complex, "black box" libraries without understanding the underlying mechanics.

The Impact Recruitment Group interview process

Impact Recruitment Group candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Technical Screening

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Business Case Discussion

A case has a fixed clock, and a good deal of what is being scored is how you spend it. Thirty to forty-five minutes buys one pass across the whole problem or a deep read of one part of it, and choosing between those is the work rather than a compromise forced on you. Announce the shape early: the structure you are using, the branch you think carries the decision, and what you are setting aside. An answer that is thorough for the first third and silent on the recommendation reads worse than one that is rougher throughout and lands.

What to demonstrate

• Whether a visible structure appears in the opening minutes and survives the rest of the case
• Whether the depth goes to the branch that carries the decision, rather than the branch you find most comfortable
• Whether you say what you are leaving out and why, instead of quietly omitting it and hoping nobody asks

How to prepare

• After each practice case, write down the branches you chose not to open and the reason for each, then check whether you said any of them out loud while the case was running. A branch you only cut privately reads to the interviewer as one you missed.
• Redo a case you have already worked in half the time, deciding in advance which single branch you keep, then compare which version a listener would find more useful.
• Write a two-sentence opening you can reuse, holding the restated question and your plan for the available time, and deliver it within the first ninety seconds of every practice run.

Leadership Interview

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

• Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
• The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
• Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

• Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
• Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
• Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Computing days-to-pay or proposal cycle time over completed records only

At any snapshot date, invoices that have already been paid are disproportionately the fast ones, and proposals that already have a decision are disproportionately the quick ones. Averaging over the completed set alone biases both numbers downward, and the bias grows exactly when the business is deteriorating, because the slow cases are the ones still open. Unpaid and undecided records are right-censored; use Kaplan-Meier or a restricted mean up to a fixed horizon, and never fill paid_at with a placeholder.

Trending utilisation or revenue on work_date without accounting for timesheet backfill

Time entries are created days to weeks after the work happens, and the backfill tail often runs two to six weeks. A dashboard keyed on work_date therefore shows the most recent weeks as a decline that reverses on every refresh. The fix is either to hold the reporting window back past the observed backfill tail (measure the tail with the timesheet submission lag metric rather than guessing) or to report an as-of-entered_at snapshot so the series is internally consistent, and to state which one you used.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Crediting a treatment for regression to the mean

Selecting a group because it is extreme (lowest-engagement users, accounts having their worst month, the bottom decile of a score) moves that group's expected next-period value back toward the average even under no treatment, by exactly as much as the selecting measure is imperfectly correlated with its own later value. Compare against units that met the same selection rule and went untreated, or use two pre-periods so the bounce-back is visible before the intervention starts. A pre-post number on a group chosen for being extreme measures the selection rule, not the treatment.

Questions & practice

What are the trade-offs between Clustering techniques for a retail brand?

Medium · machine learning and modelling

Approach

1. Pick an evaluation metric that matches the cost of each error type, not a default.
2. Say how the offline result would be validated online before it is trusted.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Explain the assumptions of Linear Regression and how you validate them.

Medium · machine learning and modelling

Approach

1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
2. Set a baseline first, so any model has something honest to beat.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Collapse time entries into contiguous staffing spells

Medium · sessionisation · gap and island · vectorisation

From approved delivery time entries (consultant_id, engagement_id, work_date, hours, charge_code, status, with engagement_id not null), build staffing spells. Per consultant and engagement, collapse weeks containing any logged hours into contiguous runs, where three or more consecutive zero-hour weeks end a spell. Output one row per spell with consultant_id, engagement_id, start_week, end_week, active_weeks, gap_weeks and total_hours. Consultants sit on several engagements at once, so spells from different engagements may overlap in time and must not be merged. No Python loop over rows.

Approach

1. Aggregate to (consultant_id, engagement_id, week_start) with summed hours and keep only weeks with positive hours. The absent weeks are the signal, so materialising zeros here would destroy the thing you are detecting.
2. Convert week_start into an integer week index, ((week_start - epoch_monday).dt.days // 7), so gap detection is integer subtraction rather than calendar arithmetic that breaks over month and year boundaries.
3. Sort by [consultant_id, engagement_id, week_index], diff the week index inside each pair, and mark a spell start where the diff is null (first row of the pair) or greater than 3. A diff of 1 is adjacent weeks and a diff of 3 is two empty weeks, which the tolerance permits.
4. Take spell_id = the cumulative sum of that boolean over the whole frame so ids are globally unique, then a single groupby on [consultant_id, engagement_id, spell_id] yields min and max week, active week count and summed hours; gap_weeks = (end - start + 1) - active_weeks.
5. Do not deduplicate overlapping spells across engagements. A consultant on two engagements in the same week is the normal case, and that overlap is the fact any capacity or context-switching question needs.

Worked solution (30 min)

1. Filter to status == 'approved', charge_code == 'client_delivery' and engagement_id.notna(); derive week_start by subtracting the weekday offset from work_date.
2. wk = df.groupby(['consultant_id','engagement_id','week_start'], as_index=False).hours.sum(); wk = wk[wk.hours > 0]; wk['wi'] = (wk.week_start - pd.Timestamp('1970-01-05')).dt.days // 7.
3. wk = wk.sort_values(['consultant_id','engagement_id','wi']); d = wk.groupby(['consultant_id','engagement_id']).wi.diff(); wk['new_spell'] = d.isna() | (d > 3); wk['spell_id'] = wk.new_spell.cumsum().
4. spells = wk.groupby(['consultant_id','engagement_id','spell_id']).agg(start_week=('week_start','min'), end_week=('week_start','max'), active_weeks=('wi','size'), total_hours=('hours','sum'), span=('wi', lambda s: s.max() - s.min() + 1)).reset_index(); gap_weeks = span - active_weeks.
5. Run the conservation assertion (spell hours sum to input hours) and the toy case below before returning.

Expected result: One row per spell, where a single pair with active week indices [1, 2, 5, 9] produces two spells: weeks 1 to 5 with active_weeks 3 and gap_weeks 2, and week 9 alone with active_weeks 1 and gap_weeks 0.

Verify it yourself

• Summed total_hours across spells equals the summed hours of the filtered input, exactly.
• Within a pair, spells are disjoint, ordered, and separated by at least three empty weeks.
• No spell contains an internal gap longer than two weeks, verified by re-expanding one spell against the week-level frame.
• Two consultants with identical week patterns on the same engagement produce two separate spells, not one.

Follow-up

• Re-run with a one-week and a four-week tolerance. What happens to the spell count, and which tolerance would you defend to a staffing lead?
• Using these spells, how would you measure how many engagements a consultant is split across in a given week, and why is that not just a count of rows?

Explain the difference between JOIN types and their impact on dataset size.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. State the window function and its partition and ordering out loud before writing it.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

How would you optimize a slow-running SQL query?

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. State the window function and its partition and ordering out loud before writing it.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Longest consecutive bench spell per consultant from weekly activity

Hard · gaps and islands · window functions · run length

You have a dense series (consultant_id, week_start, billable_hours, full_week_leave BOOLEAN) covering 26 weeks for every billable-role consultant, with zero-hour weeks present as 0. Return every maximal run of consecutive weeks where billable_hours = 0, as consultant_id, spell_start_week, spell_end_week and week_count, then each consultant's longest run. Weeks flagged full_week_leave must neither break a run nor count toward its length. Mark runs still open at the last week of the series rather than reporting them as finished.

Approach

1. Remove full_week_leave weeks from the sequence first, then re-number what remains with ROW_NUMBER() OVER (PARTITION BY consultant_id ORDER BY week_start). Deleting before numbering is precisely what makes a leave week transparent to a run instead of a break in it; filtering afterwards would leave a gap in the sequence and split the spell.
2. Flag each surviving week as bench (billable_hours = 0) or active, then compute two row numbers per consultant: one over all surviving weeks, one partitioned additionally by the bench flag. Their difference is constant inside any run of adjacent same-flag weeks, which is the island key.
3. Group the bench weeks by (consultant_id, island key) and take MIN(week_start), MAX(week_start) and COUNT(*). Do not island on week_start minus an interval times a row number: that variant assumes the spine has no holes, and the leave removal has just deliberately put holes in it.
4. Pick the longest spell per consultant with ROW_NUMBER() OVER (PARTITION BY consultant_id ORDER BY week_count DESC, spell_start_week) so ties resolve deterministically, rather than a MAX that cannot carry the accompanying dates.
5. Handle both edges as censoring. A spell touching the last week of the series is still open and its length is a lower bound, so flag it; a spell touching the first week may have started earlier, so either flag it too or exclude it from any length distribution and say which.

Worked solution (40 min)

1. Filter out full_week_leave rows and count the surviving weeks per consultant.
2. Add the bench flag and the two row numbers, then print one consultant's full sequence to see the island key stay constant across a run.
3. Group bench weeks by consultant and island key and aggregate to spell boundaries and lengths.
4. Add the open-at-the-edge flag by comparing spell_end_week to the consultant's last surviving week.
5. Rank spells per consultant and select the longest with the deterministic tiebreak.

Expected result: One row per maximal bench run with its first and last week, its length in weeks and an is_open flag, plus a one-row-per-consultant longest-spell table. A consultant with no bench weeks appears in neither, which is correct and worth stating.

Verify it yourself

• For each consultant, the sum of week_count across all spells equals their count of bench weeks in the leave-filtered series.
• No two spells for the same consultant are adjacent in the re-numbered sequence; adjacency would mean the island key was computed wrongly.
• The longest-spell table has at most one row per consultant and its week_count matches the maximum in the spells table.
• Hand-check one consultant by printing their 26 weeks as a strip of zeros, hours and leave markers, and reading the runs off it.

Follow-up

• A spell begins before your 26-week window. How do you detect that, and what do you report as its length?
• Rebuild this at daily grain. What changes about weekends, holidays and the island key?
• How would you separate true bench from unbilled delivery work on a live engagement, given only charge codes?

If you have a dataset with missing values, what is your strategy for imputation?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Fix the population and the time window before naming any metric.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Given a business scenario, what metrics would you track to measure the success of a new product launch?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How would you approach a guesstimate problem involving market sizing?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Explain a past project where you had to pivot your approach due to data limitations.

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Fix the population and the time window before naming any metric.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Diagnose a sample ratio mismatch before reading the result

Easy · sample ratio mismatch · randomisation integrity · chi-square test

A staffing-recommendation tool was randomised one to one across 840 billable-role consultants, with assignment drawn from a hash of consultant_id, so 420 sit in each arm and the assigned arm is recoverable for every consultant. The analysis table, built from exposure logs, holds 300 treatment and 420 control consultants: 720 rows against 840 assignments. Treatment shows a 1.9-point utilisation lift at p = 0.01, and the sponsor wants it in the deck tomorrow. Run the right test on the allocation, quantify how far the split sits from 1:1, decide whether the lift can be reported, and name the two most likely mechanisms given that the table came from exposure logs rather than the assignment table.

Approach

1. Test the split before touching the outcome. Chi-square goodness of fit on the 720 retained rows against an expected 360/360 gives chi-square = 2 * (60^2 / 360) = 20.0 on 1 degree of freedom, p around 7.8e-6. The equivalent binomial z is 60 / sqrt(720 * 0.25) = 4.47, and z^2 = 20.0 reproduces the chi-square.
2. Then run the sharper test that a recoverable assignment makes available: per-arm retention against the known 420. Control retains 420/420 = 100%, treatment retains 300/420 = 71.4%. That 28.6-point gap says more than the split test does, because it names which arm lost rows and how many, rather than only that the ratio is off.
3. Use a much stricter alpha for the allocation test than for the outcome, commonly 0.0005. The split test protects no multiplicity budget and should fire only on real breakage; 7.8e-6 clears even that bar by a factor of about sixty.
4. Treat the outcome as unreportable rather than as interesting-with-a-caveat. Under a sample ratio mismatch the analysis population is no longer the randomised population, so the comparison is observational and the lift carries no protection from confounding.
5. Trace the mechanism from the table's provenance, starting from the fact that the deficit is one-sided. A symmetric filter, such as an inner join to dim_consultant on is_current = TRUE, drops rows from both arms at similar rates: it shrinks n without moving the ratio, so it cannot be the cause here and should be ruled out rather than listed. Two one-sided candidates fit. First, the exposure log records a treatment consultant only once they open the tool, so non-openers vanish from treatment and the arm is silently conditioned on engagement, which is a post-assignment filter. Second, the arms are populated by different loggers: treatment exposure is written by the tool's own client-side beacon while control exposure is written server-side by the staffing screen, so the arms differ in logging coverage rather than in behaviour.
6. Rebuild on intention to treat from the assignment table, so all 840 assigned consultants sit in their assigned arm whether or not they opened the tool, then re-run the split test before re-running the outcome.

Worked solution (15 min)

1. Reconcile row counts first: the assignment table puts 420 in each arm, the analysis table holds 300 and 420, so 120 rows are missing and every one of them is on the treatment side.
2. Expected under 1:1 on the 720 retained rows: 360 per arm, observed deviation 60 in each direction.
3. chi-square = (300 - 360)^2 / 360 + (420 - 360)^2 / 360 = 10 + 10 = 20.0 on 1 degree of freedom.
4. Convert: z = sqrt(20.0) = 4.47, two-sided p = 2 * (1 - Phi(4.47)) = 7.8e-6.
5. Observed treatment share 300/720 = 0.4167; 95% interval 0.4167 +/- 1.96 * sqrt(0.4167 * 0.5833 / 720) = 0.4167 +/- 0.0360, so [0.381, 0.453], excluding 0.5.
6. Retention by arm against assignment: control 420/420 = 1.000, treatment 300/420 = 0.714. The loss is entirely one-sided, which is the signature of a treatment-side filter rather than of noise.

Expected result: chi-square = 20.0 on 1 degree of freedom, p around 7.8e-6, treatment share 0.4167 with a 95% interval of roughly [0.381, 0.453] that excludes 0.5, and a one-sided retention gap of 100% control against 71.4% treatment. The allocation is broken and the utilisation result is not reportable until the analysis table is rebuilt from assignment rather than exposure.

Verify it yourself

• The chi-square and the binomial z must agree: z^2 = 4.47^2 = 20.0 = chi-square.
• Arm counts must be reconciled against the 840 randomised consultants, not only against each other. They sum to 720, so 120 rows were dropped, and the drop sitting entirely on one arm is what makes it a filter.
• A candidate mechanism must be capable of producing a one-sided deficit. Any filter applying equally to both arms shrinks n while leaving the ratio near 1:1, so it fails this test and is not the explanation.
• Re-running the split test on the assignment table rather than the exposure table should come back non-significant if the hash itself is sound.

Follow-up

• Suppose the analysis table had held all 840 consultants, split 402 to 438. Run the same test and tell me whether you would report the result.
• If non-openers are themselves a treatment effect, because the tool is bad enough that people ignore it, what does intention to treat estimate, and what would you need to estimate the effect on compliers?
• How would you monitor for sample ratio mismatch continuously without creating a peeking problem on the outcome?

Realisation rate steps up overnight after a warehouse release

Medium · definition change · realisation · restatement

Realisation rate for one practice area steps from 0.93 to 1.02 on a single reporting date. Bill rates, hours worked and the client set are unchanged. The metric is SUM(amount_usd) from fct_invoice_line over SUM(hours * bill_rate_usd) from approved billable fct_time_entry rows for the same engagement and period. A warehouse release shipped that week. Determine what in the definition can produce a level shift with no operational change, and deliver a back-restated series plus the definition you would keep.

Approach

1. Treat the shape as evidence: a definition change is a discontinuity at one processing boundary, while a behaviour change ramps over weeks. Plot daily, not monthly, to see which one this is before forming any hypothesis.
2. Diff the metric's lineage across the release and enumerate the candidate predicates: which line_type values enter the numerator (dropping credit_note and write_off inflates it), whether the period join moved from period_start_date to issued_at, and whether the denominator counts status = 'approved' only or now includes 'submitted'.
3. Run both definitions over the same fixed pre-release month. If old and new differ by approximately the size of the step, the step is definitional; if they agree, look for a behaviour change coinciding with the release date.
4. Sanity-check the new level against the metric's physics: values above 1.0 are legitimate only on fixed_fee work delivered under its hour budget, so a practice dominated by time_and_materials sitting at 1.02 is itself a red flag.
5. Restate the full history under the chosen definition, publish old and new series together with the changeover marked, and add a reconciliation test that recomputes one known engagement-month from raw rows.

Follow-up

• Which definition is correct, and what does excluding write-offs from the numerator do to the guardrail relationship with utilisation?
• How do you prevent the next release from silently restating a headline metric?
• If the release also changed the denominator's status filter, which direction does that push realisation and by roughly how much?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Disagreeing with a proposed utilisation target using realisation evidence

Medium · realisation · guardrail metrics · observational evidence

A delivery lead proposes raising the billable utilisation target for analyst through senior_consultant from 72 to 85 percent. You have fct_time_entry, including is_billable, bill_rate_usd and written_off_hours, and fct_invoice_line. You believe the target will raise reported utilisation and lower fees. Prepare the disagreement: the evidence you pull, the mechanism you name, the metric pair you propose instead, and the condition under which you would concede that the target is correct.

Approach

1. Name the probe: whether you disagree with a mechanism and a measurement, or with an opinion about a metric being bad.
2. State the substitution precisely. Utilisation counts approved hours with is_billable = TRUE. An hour that is charged to the client and later written off stays in that numerator, so utilisation is unaffected while realisation, fees divided by hours times bill_rate_usd, falls and margin falls with it. That is the exact channel by which a higher target can raise the reported number and lower revenue.
3. Pull the evidence at consultant-month grain: plot realisation and the write-off share, written_off_hours over billable hours, against utilisation decile. If the current top decile already shows lower realisation, the proposed target moves a large share of the staff into that regime.
4. Stratify before concluding. Fixed_fee teams can show high utilisation and high realisation for reasons that have nothing to do with the proposal, so run the comparison within pricing_model and report the mix.
5. Propose the pair rather than the veto: utilisation published with realisation and write-off rate as standing guardrails, with the threshold at which the combination is net positive stated in advance. Then name your concession condition: if the top utilisation decile shows no realisation penalty and bench hours are the binding constraint, the target is right and you will say so.

Follow-up

• Utilisation and realisation are computed from overlapping hours. Does that make the relationship you found mechanical rather than behavioural?
• How many consultant-months would you need to detect a three-point realisation move, and does the firm have them?

Defending your own impact claim without randomisation or clean units

Hard · self-selection · clustered inference · impact measurement

At your review you plan to claim that the realisation dashboard you built recovered 1.4 million dollars. The evidence is that engagement-month realisation rose four points over two quarters among engagements whose leads used it. Adoption was voluntary. There are about sixty client accounts and the top five carry most fees. A new rate card shipped in the same quarter. Write the claim you can defend, the estimate you would actually produce, and what you say when asked for a causal number you cannot get.

Approach

1. Name the probe: whether you can separate the number you want from the number the data supports, under review pressure, without either inflating it or retreating to saying nothing can be known.
2. State both identification problems concretely. Voluntary adoption means adopting leads are plausibly the ones who already manage realisation, so the comparison is confounded at the person level. The rate card changes bill_rate_usd, which sits in the realisation denominator, so part of the four-point move is arithmetic rather than behavioural.
3. Neutralise what you can. Recompute realisation with bill rates snapshotted on work_date, or hold the denominator at the old rate card, so the rate-card change cannot move the metric by construction. Then rerun the comparison.
4. Get the inference right for the unit count. Cluster at client_id, not engagement, because engagements in one account share a partner, a rate card and a team. With sixty accounts and five carrying most fees, the effective cluster count is far below sixty, so report a wild cluster bootstrap interval rather than plain cluster-robust standard errors, which are biased downward in that regime.
5. Report both weightings and explain the divergence: an account-weighted estimate describes the typical account, a value-weighted one describes the revenue, and if they disagree a small number of accounts is carrying the result. Then give the decision-relevant sentence: the defensible range, whether its lower bound still clears the build cost, and what a proper staggered rollout would have bought.

Follow-up

• The pre-period trends for adopters and non-adopters are not parallel. What do you report then?
• You get to design the next rollout. What do you change so the same question is answerable, without randomising individual accounts?

Answering a request to redefine the utilisation denominator

Hard · denominator integrity · restatement · escalation

Before a quarterly review, a practice lead asks for two changes to reported billable utilisation: drop consultants allocated to engagements with status = 'paused' out of the denominator, and count approved travel hours booked to client_delivery as billable. Together the changes move reported utilisation from 71 to 79 percent. You must answer in writing today. Give your answer, the principle you apply to each change separately, and what you do if you are overruled.

Approach

1. Name the probe: whether you can hold a measurement line by adjudicating each proposal on its merits, rather than treating the request as bad faith or accepting it to avoid a fight.
2. Adjudicate the paused-engagement change against what the denominator means. Billable utilisation measures the share of available consultant hours that were sold. A consultant whose engagement is paused is still available and still costs salary, which is precisely the situation the metric exists to reveal. Removing them defines away the signal, so this one is refused on definitional grounds, not on suspicion.
3. Adjudicate the travel change as an empirical question, because it is one. If those hours are billed to the client, is_billable and bill_rate_usd should already reflect it; if they are charged and then written off or never invoiced, counting them raises utilisation while realisation falls. Check the invoice lines for the same engagement and period rather than arguing about principle.
4. Quantify each change separately, and the interaction. An eight-point move made of one point and seven points is a different conversation from two four-point moves, and the lead cannot make a good decision without that split.
5. If a change is accepted, restate history on the new definition and publish the old series alongside it with the date of the change. A definition applied forward only makes every period-over-period comparison meaningless. If overruled, comply with the restatement and the footnote, and escalate only the part that is a misstatement of fact rather than a defensible choice.

Follow-up

• Which of the two changes would you accept, and what evidence flips your answer on the other one?
• The lead wants the new definition applied only from this quarter forward. Why is that worse than applying it to history, and how do you explain that in one sentence?