4P Consulting Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at 4P Consulting serves as a vital bridge between raw information and strategic business direction. You are not just building models; you are solving complex, high-stakes problems that influence the trajectory of 4P Consulting's products and the experiences of its users. By leveraging advanced statistical techniques and machine learning, you transform massive datasets into actionable intelligence that guides leadership decisions.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

• Cluster inference at the account, not the engagement
• Reconstruct pipeline stage history from mutable rows
• Compute utilisation against a defended availability denominator

Overview

This role is inherently collaborative and product-focused. You will work alongside engineers, business analysts, and domain experts to integrate data-driven insights into 4P Consulting's service offerings. Whether you are designing experiments to test new product features, diagnosing sudden drops in key metrics, or deploying models into production environments, your work directly impacts the efficiency and growth of 4P Consulting. Candidates report that interviewers look for people who are as comfortable discussing algorithmic nuance as they are communicating the business impact of their findings to non-technical stakeholders.

The 4P Consulting interview process

4P Consulting candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Technical Screen

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Case Study

This round runs as a working session, so part of what it decides is whether you are useful to think with. The interviewer will interrupt: a hint that the data you assumed does not exist, a challenge to your metric, a nudge toward a branch you skipped. Treating those as interference is the common failure. Reason out loud while your thinking is still provisional so there is something to react to, and when a redirect arrives, take it instead of defending the path you had already started down.

What to demonstrate

• Whether your reasoning is audible while it is still unsettled, or only after you have privately decided
• What you do with a hint: absorb it and adjust, or argue for the original route
• Whether your clarifying questions have answers that would change your approach, as opposed to filling silence
• Whether you can be wrong about something in the middle of the case and keep moving without restarting

How to prepare

• Run practice cases with a partner instructed to interrupt twice: once to remove a data source you assumed existed, once to reject the metric you chose. Practise absorbing both without going back to the start.
• Before each practice case, write down the clarifying questions you plan to ask, then check afterwards whether any answer actually changed what you did. Drop the ones that did not.
• Explain an analysis you already know well to someone outside the field and have them stop you at every point where the reasoning jumped a step.

Panel Interaction

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

• Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
• Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
• Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
• Whether you ask what decision is actually on the table before explaining anything

How to prepare

• Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
• For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
• Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Computing days-to-pay or proposal cycle time over completed records only

At any snapshot date, invoices that have already been paid are disproportionately the fast ones, and proposals that already have a decision are disproportionately the quick ones. Averaging over the completed set alone biases both numbers downward, and the bias grows exactly when the business is deteriorating, because the slow cases are the ones still open. Unpaid and undecided records are right-censored; use Kaplan-Meier or a restricted mean up to a fixed horizon, and never fill paid_at with a placeholder.

Treating accounts as independent observations

Revenue is concentrated: a small number of client_ids typically carries a large share of fees, and engagements within one account share a partner, a rate card and a delivery team. Ordinary standard errors computed over engagements therefore understate uncertainty badly. Cluster at client_id, and with fewer than roughly 40 clusters use a wild cluster bootstrap or a CR2 correction, because cluster-robust standard errors are downward-biased in that regime and will manufacture significance that a replication will not reproduce.

Dropping rows with missing values without naming the mechanism

Say whether the values are missing at random, missing by a known process, or missing in a way that depends on the outcome, and handle them accordingly. Deleting incomplete rows silently redefines the population whenever missingness correlates with what you are measuring.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Questions & practice

How do you prevent overfitting when training a model on a small, noisy dataset?

Medium · machine learning and modelling

Approach

1. Pick an evaluation metric that matches the cost of each error type, not a default.
2. Say how the offline result would be validated online before it is trusted.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Build a dense consultant-week hours spine with zero weeks

Easy · calendar spine · groupby · missing rows

You are given time_entries (time_entry_id, consultant_id, engagement_id, charge_code, work_date as datetime64, hours, is_billable, status) and a list of consultant_ids in scope plus a start and end date. Produce a DataFrame with exactly one row per consultant per Monday-anchored week in that range, with columns consultant_id, week_start, billable_hours, total_hours. Weeks in which a consultant logged nothing must appear as zeros rather than be absent. Count only rows with status == 'approved'. Do not loop over consultants and do not use resample.

Approach

1. Anchor weeks arithmetically: week_start = work_date - pd.to_timedelta(work_date.dt.weekday, unit='D'). Avoid dt.isocalendar().week, whose numbers restart each January and do not sort across a year boundary.
2. Construct the spine from the inputs, not from the data: pd.date_range(start, end, freq='W-MON') crossed with the in-scope consultant_ids via pd.MultiIndex.from_product, so its size is fixed before any aggregation runs.
3. Aggregate approved entries once with a single groupby on [consultant_id, week_start], producing total hours and billable hours (hours.where(is_billable, 0).sum()) in the same pass.
4. Left-merge the aggregate onto the spine, fillna(0.0) on both hour columns, and keep float dtypes so later ratios do not integer-divide.
5. Assert the shape before returning: len(out) == n_consultants * n_weeks, and out.total_hours.sum() equals the sum of hours over the filtered input inside the window.

Follow-up

• Utilisation gets computed on this spine. What breaks for a consultant hired mid-window or terminated mid-window, and where would you fix it?
• How would you extend this to consultant x engagement x week without the spine exploding to a row count nobody can hold in memory?

Cluster bootstrap a margin change across few, unequal accounts

Hard · cluster bootstrap · few clusters · mix shift

engagements has engagement_id, client_id, pricing_model, fees_usd, cost_usd and period in {pre, post} around a rate-card change. There are about 35 client_ids across roughly 400 engagements, and the top five clients carry a large share of fees. Estimate the pre-to-post change in value-weighted gross margin within pricing_model, and give a 95% interval by resampling whole client_ids with replacement, B = 2000. Write the bootstrap yourself; no resampling library. Report the account-weighted estimate alongside and explain any divergence.

Approach

1. Write the statistic as a pure function of a dataframe first. Within each pricing_model, margin = (sum(fees) - sum(cost)) / sum(fees) per period; the headline is the fee-weighted average of the within-model differences using fixed pre-period weights, so a shift in the mix of work sold cannot masquerade as a change in delivery.
2. Resample clusters, not rows: draw 35 client_ids with replacement and concatenate all engagements of each drawn client. A client drawn twice contributes its rows twice under distinct pseudo-ids, which is what preserves within-account correlation instead of averaging it away.
3. Recompute the statistic on each replicate and take a percentile interval from the 2.5th and 97.5th quantiles. With concentrated fees the replicate distribution is skewed, so a symmetric point plus or minus 1.96 times a standard error is wrong in the tail that matters.
4. Compute the account-weighted version (mean across clients of each client's margin change) next to the value-weighted one, and attribute the divergence to concentration rather than to noise; if they disagree in sign, that fact is the finding.
5. Name the regime honestly: at roughly 35 clusters both cluster-robust standard errors and the pairs cluster bootstrap under-cover, so state the fix you would run next, which is a wild cluster bootstrap with Rademacher weights or CR2 with t(G-1) critical values.

Worked solution (40 min)

1. stat(df): pivot fees and cost by (pricing_model, period), compute per-model margins, difference post minus pre, and weight the differences by each model's pre-period fee share; return one scalar.
2. Pre-split the frame into a dict of client_id to its rows, so each replicate is a concat of 35 preselected blocks rather than a boolean filter over 400 rows.
3. rng = np.random.default_rng(11); for b in range(2000): draw client ids with replacement, concat their blocks, call stat, store; assert the replicate's nunique cluster draw count is 35 every time.
4. ci = np.quantile(reps, [0.025, 0.975]); compute the account-weighted estimate as the unweighted mean of per-client margin changes over clients present in both periods.
5. Re-run a row-level bootstrap on the same data for contrast and record both interval widths, then leave-one-client-out to see whether one account drives the point estimate.

Expected result: A point estimate stated in margin points, a percentile interval from 2000 cluster replicates that is wider than the row-level bootstrap interval on the same data, an account-weighted estimate reported beside it, and an explicit note that at 35 clusters the interval still under-covers.

Verify it yourself

• Every replicate draws exactly 35 clusters; replicate row counts vary while cluster counts do not.
• The row-level bootstrap interval is narrower than the cluster one; quantify the ratio rather than assert it.
• The point estimate lies inside the percentile interval, and the interval is asymmetric around it when fees are concentrated.
• The within-model differences plus the mix term reconcile to the naive pooled change, which shows the decomposition is complete.

Follow-up

• Implement the wild cluster bootstrap and show at what cluster count its p-value separates from the naive one.
• One client is 30% of fees. Show the interval with and without that account and say which one you would present, and to whom.
• What pre-period check would make you willing to call this a causal effect of the rate card rather than a correlation?

Explain the difference between RANK, DENSE_RANK, and ROW_NUMBER. When would you use each?

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. Say which table is the grain you start from, and join outward from it.
3. State the window function and its partition and ordering out loud before writing it.

Follow-up

• What breaks if events arrive late or out of order?
• How would you verify this result without re-running the same query?

Write a query using SQL window functions to calculate a rolling average of user engagement over the last 30 days.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Say which table is the grain you start from, and join outward from it.
3. State the window function and its partition and ordering out loud before writing it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Find signed engagements with no approved billable hour

Medium · anti-join · null semantics · not exists

fct_engagement holds engagement_id, start_date and status. fct_time_entry holds time_entry_id, consultant_id, engagement_id, charge_code, work_date, hours, is_billable and status, and engagement_id is NULL on bench, leave, training and internal rows. Return engagements with start_date in the trailing quarter and status IN ('active','closed_delivered','closed_early') that have no approved billable time entry with work_date within 45 days of start_date, giving engagement_id, start_date and days elapsed. Then explain why NOT IN over a subquery selecting fct_time_entry.engagement_id is the wrong instrument here.

Approach

1. Express the exclusion as NOT EXISTS with a correlated predicate covering all four conditions at once: te.engagement_id = e.engagement_id AND te.is_billable AND te.status = 'approved' AND te.work_date < e.start_date + INTERVAL '45 days'. NOT IN can only carry a single column, so a multi-condition anti-join forces a contorted subquery even before the NULL problem.
2. State the NULL semantics precisely: NOT IN (SELECT engagement_id FROM fct_time_entry) evaluates to NULL rather than TRUE for every candidate as soon as one NULL is present in the list, because x <> NULL is unknown. The WHERE clause keeps only TRUE, so the query returns zero rows and reads as a clean bill of health. This table guarantees NULLs, since bench and leave rows carry no engagement.
3. Note the equivalent LEFT JOIN ... WHERE te.engagement_id IS NULL form. It is correct, but it materialises a join only to discard most of it; NOT EXISTS states the intent and most planners execute both as the same anti-join.
4. Guard the window against truncation: an engagement that started 10 days ago cannot yet have failed a 45-day test, so require start_date <= current_date - 45 or report the too-early set separately rather than counting them as failures.
5. Decide on the late-timesheet risk explicitly. A missing entry may mean unstaffed work or an unsubmitted timesheet, so either hold the cutoff back past the measured submission lag or label the recent tail as provisional.

Worked solution (20 min)

1. Count engagements matching the status and start_date predicates; this is the denominator.
2. Write the NOT EXISTS version and record its row count.
3. Write the NOT IN version on the same data and observe zero rows, then confirm with SELECT COUNT(*) FROM fct_time_entry WHERE engagement_id IS NULL that NULLs are present.
4. Cross-check by counting engagements that DO have a qualifying entry and verifying the two counts sum to the denominator.
5. Open one engagement from the result and look at its raw time entries to see whether the cause is no staffing or unapproved timesheets.

Expected result: A short list of engagement_id, start_date and days_since_start for engagements that started at least 45 days ago and carry no approved billable hour in their first 45 days. The NOT IN variant returns zero rows on the same data.

Verify it yourself

• Count of flagged engagements plus count of engagements with at least one qualifying entry equals the count of engagements matching the status and date filters.
• Every returned engagement has start_date at or before current_date - 45.
• Spot-checking one flagged engagement in fct_time_entry shows either no rows or only unapproved or non-billable rows.
• Replacing NOT EXISTS with LEFT JOIN ... IS NULL returns an identical set.

Follow-up

• Turn this into a time-to-first-billable-hour distribution. What do you do with engagements that never recorded one?
• Should client_pursuit hours count as the engagement being staffed, and how does that choice change the list?
• How would you prove the empty result from the NOT IN version is a bug rather than good news?

How would you design a metric to measure the success of a new feature rollout?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

If a critical product metric suddenly drops by 10%, what is your step-by-step approach to diagnosing the root cause?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Fix the population and the time window before naming any metric.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

If we were to launch a new subscription tier, what metrics would you track to determine its long-term viability?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Fix the population and the time window before naming any metric.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize which product features to build based on user data?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How do you determine the required sample size for an A/B test?

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

If an experiment shows a positive result but the p-value is 0.07, how do you proceed?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. State the primary metric and the minimum effect worth shipping, then size the test.
3. Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

Handle a subgroup scan after a null primary result

Medium · multiple comparisons · false discovery rate · subgroup analysis · interaction test

The pre-registered primary outcome of an account-randomised pilot came back null. The sponsor asks for the breakdown by practice area and region: five practice areas by four regions, twenty cells, from 120 accounts. One cell shows p = 0.04 and the sponsor wants to roll out there. State the family-wise error rate of the unadjusted scan, apply both a family-wise and a false-discovery-rate correction to the reported p, name the test that actually asks whether the effect differs by segment, and give the per-cell detectable effect implied by the account count.

Approach

1. Quantify the scan. Twenty disjoint cells give near-independent tests, so the family-wise error rate at a nominal 0.05 each is 1 - 0.95^20 = 0.64. Under a true global null, one nominally significant cell is the expected outcome rather than a signal.
2. Apply Holm when any single cell will be acted on. The smallest p must clear 0.05/20 = 0.0025 for the first rejection, so p = 0.04 misses by a factor of sixteen. Holm dominates plain Bonferroni at the same family-wise level and costs nothing to use, so prefer it.
3. Apply Benjamini-Hochberg at q = 0.05 if the goal is to generate candidates for a follow-up rather than to act. With 0.04 as the smallest p, the rank-1 threshold is (1/20) * 0.05 = 0.0025, so nothing survives there either. The two procedures agreeing is itself worth stating.
4. Replace the scan with the test that answers the question asked. Fit one model with treatment, segment main effects and treatment-by-segment interactions, then run a single joint Wald or F test on the 19 interaction terms. That asks whether the effect differs, rather than asking twenty separate times whether it is non-zero somewhere.
5. Close with power arithmetic, which usually settles the argument faster than the correction does. 120 accounts spread over 20 cells is about six accounts per cell, so the per-cell MDE is roughly sqrt(20) = 4.5 times the overall MDE. A design that could not detect the pooled effect cannot detect a segment-specific one, so the winning cell is far more likely noise than discovery.

Worked solution (20 min)

1. Unadjusted family-wise error: 1 - 0.95^20 = 0.642.
2. Holm and Bonferroni threshold for the smallest p: 0.05 / 20 = 0.0025, against the observed 0.04.
3. Benjamini-Hochberg: sort all twenty p-values, find the largest k with p_(k) <= (k/20) * 0.05; with 0.04 smallest, the rank-1 threshold is 0.0025 and k = 0.
4. Fit treatment, segment main effects and treatment-by-segment interactions in one model, clustered at client_id, and run the joint test on the 19 interaction terms.
5. Compute the per-cell MDE as the overall MDE times sqrt(20) and set it beside the effect the winning cell is claiming.

Expected result: Unadjusted family-wise error of 0.64; the p = 0.04 cell clears neither Holm at 0.0025 nor Benjamini-Hochberg at q = 0.05; the defensible statement is that there is no evidence of effect heterogeneity, backed by a joint interaction test on 19 degrees of freedom and a per-cell MDE about 4.5 times the overall one.

Verify it yourself

• Benjamini-Hochberg must be at least as permissive as Holm on the same p-vector; if it rejects fewer, the implementation is wrong.
• Repeating the twenty-cell scan under a simulated global null should produce at least one p below 0.05 in roughly 64% of replications.
• The joint interaction test must carry 19 degrees of freedom, one fewer than the number of cells.

Follow-up

• How would you pre-register a subgroup analysis so that a finding inside it could actually be acted on?
• The twenty cells differ in size and in treated share. What breaks in the independence assumption behind 1 - 0.95^20?
• If the joint interaction test is significant but no individual cell survives correction, what do you report?

Fees per billable head fell nine percent; rank the causes

Hard · decomposition · seasonality · clustered inference

Recognised fees per billable head fell about 9% quarter over quarter and leadership wants a ranked cause by Friday. You have fct_time_entry, fct_invoice_line, fct_engagement and dim_consultant, plus a workday and regional holiday calendar. Decompose the move into available hours per head, billable utilisation and realised dollars per billable hour, separate calendar effects from behaviour, split within-segment change from mix, and deliver a ranked attribution with uncertainty. Name the one further cut you would run if the leading term is ambiguous.

Approach

1. Write the identity before computing anything: fees per head = (available hours per head) x (billable utilisation) x (realised dollars per billable hour), where the last term is recognised fees divided by approved billable hours. Verify the product reproduces the reported 9% exactly; if it does not, the scope of the three inputs disagrees and nothing downstream is trustworthy.
2. Decompose in logs: d ln(fees per head) = d ln(available per head) + d ln(utilisation) + d ln(realised rate). The log form is exactly additive for a product, so contributions sum to the total without an interaction residual and cannot attribute more than the whole move.
3. Strip the calendar first, because it is the term nobody can act on: available hours per head moves with scheduled workdays, regional public holidays, leave and any change in fte_fraction mix. Quantify how many of the nine points are calendar and quote the remainder as the behavioural move.
4. Split each surviving term by practice_area, level, region and pricing_model, and within each split separate within-segment change from mix using the weighted three-term decomposition, since a shift toward lower-rate levels reads as a rate decline that no individual rate produced.
5. Attach uncertainty at the unit that generates it: anything fee-side clusters at client_id because engagements in one account share a partner, a rate card and a team. With fewer than roughly forty clusters use a wild cluster bootstrap or a CR2 correction, since cluster-robust standard errors are downward-biased in that regime.
6. If realised rate leads and is ambiguous, the disambiguating cut is discount versus write-off versus fixed-fee overrun: compare invoiced fees to the standard-rate value of hours worked, and read written_off_hours and scope overrun by pricing_model.

Follow-up

• Two of the nine points are calendar. How do you say that to an executive without sounding like you are explaining the number away?
• Utilisation is up and realised rate is down. What single operational behaviour produces both, and how would you confirm it?
• What would you need to see to claim this is a demand problem rather than a delivery or pricing one?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Getting an analysis acted on

Work that nobody used is a common and unflattering pattern in data careers, and interviewers probe for it. Have a story about an analysis that changed a decision, and be specific about how you got it in front of the person who could act. Also have one about work that went nowhere, with your reading of why.

How do you handle missing or inconsistent data when joining multiple large datasets?

Medium · behavioural and stakeholder questions

Approach

1. Close with what you would do differently, concretely.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• What would you do differently if you ran that project again?
• How did you know the outcome was caused by your change?

Defending a fixed-fee overrun finding against the selling partner

Easy · metric definition · pricing model mix · stakeholder pushback

Over the trailing four quarters, fixed_fee engagements in one practice area show a median scope overrun ratio of 1.34, computed as approved client_delivery hours in fct_time_entry divided by fct_engagement.contracted_hours, and inception-to-date gross margin nine points below the time_and_materials book. The partner who sold most of that work says the denominator is wrong because change orders were signed, and each change order is a separate row linked by prior_engagement_id. You have ten minutes with the practice lead. Present and defend the finding.

Approach

1. Say out loud what is being probed: whether a specific methodological objection makes you recompute or makes you repeat yourself. Restate the metric as numerator, denominator and window before defending anything.
2. Test the objection empirically instead of debating it. Rebuild the denominator by walking the prior_engagement_id chain with a recursive CTE, summing contracted_hours across the original SOW and every extension, and summing approved delivery hours over the same chain.
3. Report both numbers and say what each one answers: the original-SOW denominator answers whether the team held the scope that was signed, the chained denominator answers whether the firm estimated the total work correctly. Neither is a trick; they are different questions.
4. Keep the margin claim separate from the overrun claim, and keep it stratified. Fixed_fee margin falls with hours worked while uncapped time_and_materials margin does not, so the nine-point gap is only meaningful within pricing_model, and a shift in the fixed_fee share of fees is reported alongside it.
5. Close with the decision and a falsifier: if the chained overrun is near 1.0, the problem is scope control on the original SOW, and the check is whether change orders repriced at the standard rate card or at the original blended rate.

Follow-up

• The chained denominator narrows the overrun to 1.11. Does your recommendation change, and which number goes in the practice review?
• How would you distinguish an estimation problem at sale time from a delivery problem during execution, using only these tables?

Answering a request to redefine the utilisation denominator

Hard · denominator integrity · restatement · escalation

Before a quarterly review, a practice lead asks for two changes to reported billable utilisation: drop consultants allocated to engagements with status = 'paused' out of the denominator, and count approved travel hours booked to client_delivery as billable. Together the changes move reported utilisation from 71 to 79 percent. You must answer in writing today. Give your answer, the principle you apply to each change separately, and what you do if you are overruled.

Approach

1. Name the probe: whether you can hold a measurement line by adjudicating each proposal on its merits, rather than treating the request as bad faith or accepting it to avoid a fight.
2. Adjudicate the paused-engagement change against what the denominator means. Billable utilisation measures the share of available consultant hours that were sold. A consultant whose engagement is paused is still available and still costs salary, which is precisely the situation the metric exists to reveal. Removing them defines away the signal, so this one is refused on definitional grounds, not on suspicion.
3. Adjudicate the travel change as an empirical question, because it is one. If those hours are billed to the client, is_billable and bill_rate_usd should already reflect it; if they are charged and then written off or never invoiced, counting them raises utilisation while realisation falls. Check the invoice lines for the same engagement and period rather than arguing about principle.
4. Quantify each change separately, and the interaction. An eight-point move made of one point and seven points is a different conversation from two four-point moves, and the lead cannot make a good decision without that split.
5. If a change is accepted, restate history on the new definition and publish the old series alongside it with the date of the change. A definition applied forward only makes every period-over-period comparison meaningless. If overruled, comply with the restatement and the footnote, and escalate only the part that is a misstatement of fact rather than a defensible choice.

Follow-up

• Which of the two changes would you accept, and what evidence flips your answer on the other one?
• The lead wants the new definition applied only from this quarter forward. Why is that worse than applying it to history, and how do you explain that in one sentence?