Interview guides · AArete

AArete Data Scientist Interview Questions & Guide 2026

Case practice for AArete data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

As a Data Scientist at AArete, you occupy a strategic position at the intersection of advanced analytics and high-impact business consulting. Your primary responsibility is to distill complex data into actionable insights that drive efficiency and profitability for clients. Unlike pure research roles, this position demands a pragmatic approach to problem-solving, where your models and analyses must directly translate into measurable business outcomes.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

Cluster inference at the account, not the engagement
Correct late-entered timesheets before trending recent weeks
Separate bookings, recognised revenue and collected cash

Overview

You will work closely with cross-functional teams to tackle diverse challenges, ranging from operational optimization to product metric evaluation. Because AArete operates in a fast-paced consulting environment, you must be comfortable with ambiguity and capable of communicating technical findings to stakeholders who may lack a data science background. Success in this role requires not only technical rigor in SQL, statistics, and experimentation, but also the ability to frame data problems within a broader product or business context.

The AArete interview process

AArete candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interviews

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Final Assessment Rounds

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Pooling margin, realisation or overrun across pricing models

Fixed-fee margin falls with hours worked; uncapped time-and-materials margin rises with hours worked; retainer margin depends on neither. A quarter in which the firm sells more fixed-fee work will show a margin change caused entirely by mix, not by delivery performance, and the aggregate can move in the opposite direction to every individual pricing model. Always stratify by fct_engagement.pricing_model before comparing periods, and report the mix shift alongside the within-stratum change.

Modelling win rate on proposals with a recorded outcome, using fields written after the decision

Two failures compound here. First, stage IN ('withdrawn','no_decision') is not missing at random: those are disproportionately deals that were going to be lost, so training on won-plus-lost only inflates apparent win rate and distorts the coefficients. Second, fields like engagement_id, final scope and revised pricing are populated after the outcome is known, so including them leaks the label and produces a model with excellent backtest accuracy and no forward value. Restrict features to values knowable at submitted_at.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

Simulate the chance a capped engagement crosses its cap

Medium · monte carlo · bootstrap resampling · pricing caps

A capped time-and-materials engagement has not_to_exceed_usd = 400000, has billed 250000 to date, and bills at a blended bill_rate_usd of 250. Fourteen delivery weeks remain before planned_end_date. You have that engagement's last twenty weekly totals of approved billable hours as a pandas Series. Estimate the probability that cumulative billable value crosses the cap before the planned end, and the expected unbillable hours if it does. Resample the observed weeks; do not assume normality. Report a Monte Carlo standard error and justify the number of draws.

Approach

Reduce the deterministic part first: the cap still affords (400000 - 250000) / 250 = 600 hours, so the whole question is the distribution of the sum of fourteen future weekly hour totals against a fixed threshold of 600.
Before resampling, check the twenty observed weeks for trend and lag-1 autocorrelation. Independent resampling is defensible only if the weeks are exchangeable; on a ramping engagement use a moving-block bootstrap or model the ramp, because i.i.d. draws understate the upper tail, which is exactly the tail the question is about.
Draw one (B, 14) array with np.random.default_rng().choice(observed, size=(B,14), replace=True), sum along axis 1, and compute p_hat = mean(total > 600) in one vectorised pass.
Report expected unbillable hours two ways: unconditional mean(maximum(total - 600, 0)) for expected loss, and the mean conditional on a breach for how bad a breach is when it happens. The second is the number that drives a change-order conversation.
Attach se(p_hat) = sqrt(p_hat(1 - p_hat)/B) and pick B from the precision you need: plus or minus one point at 95% confidence needs about 1.96^2 * 0.25 / 0.01^2, roughly 9600 draws at the worst case p = 0.5.
State the assumptions that would flip the answer: constant blended rate, no scope change, no holiday weeks inside the fourteen, and a cap that applies to fees rather than to fees plus expenses.

Worked solution (30 min)

affordable_hours = (400000 - 250000) / 250 = 600.0; print it, because a wrong threshold makes every later number wrong in a way no simulation reveals.
Plot or regress the twenty weekly totals against week index and compute lag-1 autocorrelation; record the result as the stated justification for i.i.d. versus block resampling.
rng = np.random.default_rng(7); draws = rng.choice(weekly.values, size=(10000, 14), replace=True); totals = draws.sum(axis=1).
p_hat = (totals > 600).mean(); se = np.sqrt(p_hat * (1 - p_hat) / 10000); overrun = np.maximum(totals - 600, 0); report overrun.mean() and overrun[totals > 600].mean().
Re-run with a second seed and confirm p_hat moves by less than two standard errors before reporting.

Expected result: A probability with its Monte Carlo standard error (about 0.005 at B = 10000 near p = 0.5), expected unbillable hours both unconditionally and conditional on a breach, and an explicit statement of the exchangeability assumption behind the resampling.

Verify it yourself

Two different seeds agree to within roughly two Monte Carlo standard errors.
If every observed week exceeds 600/14 = 42.86 hours, p_hat must be exactly 1.0; if every week is below it, p_hat must be 0.0.
Unconditional expected overrun is zero whenever p_hat is zero, and is always at most the conditional value times p_hat plus rounding.
Raising the cap by 25000 moves the threshold by exactly 100 hours, and p_hat must fall.

Follow-up

Hours past the cap still cost money. Restate the result as expected gross margin rather than a probability.
Two weeks of time are entered but not yet approved. How do you fold them in without double-counting?
A change order that raises the cap is judged 60% likely. How does that change the number you present, and to whom?

Collapse time entries into contiguous staffing spells

Medium · sessionisation · gap and island · vectorisation

From approved delivery time entries (consultant_id, engagement_id, work_date, hours, charge_code, status, with engagement_id not null), build staffing spells. Per consultant and engagement, collapse weeks containing any logged hours into contiguous runs, where three or more consecutive zero-hour weeks end a spell. Output one row per spell with consultant_id, engagement_id, start_week, end_week, active_weeks, gap_weeks and total_hours. Consultants sit on several engagements at once, so spells from different engagements may overlap in time and must not be merged. No Python loop over rows.

Approach

Aggregate to (consultant_id, engagement_id, week_start) with summed hours and keep only weeks with positive hours. The absent weeks are the signal, so materialising zeros here would destroy the thing you are detecting.
Convert week_start into an integer week index, ((week_start - epoch_monday).dt.days // 7), so gap detection is integer subtraction rather than calendar arithmetic that breaks over month and year boundaries.
Sort by [consultant_id, engagement_id, week_index], diff the week index inside each pair, and mark a spell start where the diff is null (first row of the pair) or greater than 3. A diff of 1 is adjacent weeks and a diff of 3 is two empty weeks, which the tolerance permits.
Take spell_id = the cumulative sum of that boolean over the whole frame so ids are globally unique, then a single groupby on [consultant_id, engagement_id, spell_id] yields min and max week, active week count and summed hours; gap_weeks = (end - start + 1) - active_weeks.
Do not deduplicate overlapping spells across engagements. A consultant on two engagements in the same week is the normal case, and that overlap is the fact any capacity or context-switching question needs.

Follow-up

Re-run with a one-week and a four-week tolerance. What happens to the spell count, and which tolerance would you defend to a staffing lead?
Using these spells, how would you measure how many engagements a consultant is split across in a given week, and why is that not just a count of rows?

Build a dense consultant-week hours spine with zero weeks

Easy · calendar spine · groupby · missing rows

You are given time_entries (time_entry_id, consultant_id, engagement_id, charge_code, work_date as datetime64, hours, is_billable, status) and a list of consultant_ids in scope plus a start and end date. Produce a DataFrame with exactly one row per consultant per Monday-anchored week in that range, with columns consultant_id, week_start, billable_hours, total_hours. Weeks in which a consultant logged nothing must appear as zeros rather than be absent. Count only rows with status == 'approved'. Do not loop over consultants and do not use resample.

Approach

Anchor weeks arithmetically: week_start = work_date - pd.to_timedelta(work_date.dt.weekday, unit='D'). Avoid dt.isocalendar().week, whose numbers restart each January and do not sort across a year boundary.
Construct the spine from the inputs, not from the data: pd.date_range(start, end, freq='W-MON') crossed with the in-scope consultant_ids via pd.MultiIndex.from_product, so its size is fixed before any aggregation runs.
Aggregate approved entries once with a single groupby on [consultant_id, week_start], producing total hours and billable hours (hours.where(is_billable, 0).sum()) in the same pass.
Left-merge the aggregate onto the spine, fillna(0.0) on both hour columns, and keep float dtypes so later ratios do not integer-divide.
Assert the shape before returning: len(out) == n_consultants * n_weeks, and out.total_hours.sum() equals the sum of hours over the filtered input inside the window.

Follow-up

Utilisation gets computed on this spine. What breaks for a consultant hired mid-window or terminated mid-window, and where would you fix it?
How would you extend this to consultant x engagement x week without the spine exploding to a row count nobody can hold in memory?

Write a query using SQL window functions to calculate a rolling average of sales.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

How would you identify duplicate records in a large dataset using SQL?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Deduplicate to the latest issued invoice line per engagement

Easy · window functions · dedup · determinism

fct_invoice_line holds invoice_line_id, invoice_id, engagement_id, line_type, period_end_date, amount_usd, issued_at and status; draft lines have issued_at NULL. Return exactly one row per engagement_id, its most recently issued line, with invoice_line_id, line_type, amount_usd and issued_at. Two lines on the same engagement can share an issued_at value to the second, so the result must be identical on every run against unchanged data. Write it with a window function, then say in one sentence when DISTINCT ON would be the better choice.

Approach

Filter to issued lines (issued_at IS NOT NULL, status <> 'draft') before ranking. Ranking first and filtering afterwards can leave an engagement whose top-ranked row was a draft with no row at all in the output.
Rank with ROW_NUMBER() OVER (PARTITION BY engagement_id ORDER BY issued_at DESC, invoice_line_id DESC). The surrogate key as the final ORDER BY term is what makes ties resolve the same way on every execution; without it the planner is free to return either tied row.
Wrap the ranked query in a CTE or subquery and filter rn = 1, since a window function cannot appear in the WHERE clause of the query that computes it.
Name why RANK() is wrong here: on tied issued_at it assigns 1 to both rows, so the result carries two rows for one engagement and any downstream SUM double counts.
Compare against DISTINCT ON (engagement_id) ... ORDER BY engagement_id, issued_at DESC, invoice_line_id DESC: same answer, one sort, usually cheaper, but the window form generalises the moment you also want rn <= 3 or a second ranking in the same pass.

Worked solution (15 min)

Count distinct engagement_id over the filtered (issued, non-draft) line set. This is your target row count.
Write the ROW_NUMBER CTE with both ORDER BY terms and select rn = 1.
Assert the contract with GROUP BY engagement_id HAVING COUNT(*) > 1 over the result; it must return nothing.
Deliberately construct or find an engagement with two lines sharing issued_at and confirm exactly one survives.
Rewrite with DISTINCT ON and diff the two result sets row by row.

Expected result: One row per engagement that has at least one non-draft issued line. Row count equals COUNT(DISTINCT engagement_id) over that filtered set. Engagements holding only draft lines are absent, which is correct here; listing them is an anti-join, not a dedup.

Verify it yourself

Result row count equals COUNT(DISTINCT engagement_id) on the filtered input.
GROUP BY engagement_id HAVING COUNT(*) > 1 over the result returns zero rows.
Two consecutive runs produce byte-identical output when ordered by engagement_id.
The DISTINCT ON rewrite returns the same invoice_line_id for every engagement.

Follow-up

Now return the latest line per engagement per line_type. What changes in the PARTITION BY, and what does that do to the row count?
Return the latest and second-latest line together. Which of the two constructs survives that requirement?
issued_at is TIMESTAMPTZ and finance reports on a single local calendar. Where can 'latest' flip between the two conventions?

If you notice a sudden 10% drop in daily active users, what steps would you take to diagnose the cause?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you design the metrics for a new subscription-based feature?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What product metrics would you prioritize for an e-commerce platform?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How do you balance long-term user engagement with short-term revenue goals?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Explain how you determine statistical significance in an experiment with a small sample size.

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

Walk me through the process of designing an A/B test for a new UI change.

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Measure a staggered process rollout across five practice areas

Hard · difference-in-differences · staggered adoption · wild cluster bootstrap · parallel trends

A delivery-review process was rolled out one practice area at a time over six quarters, and all five practice areas now have it, so there is no never-treated group. You have engagement-quarter gross margin reconstructible from fct_invoice_line and fct_time_entry, plus pricing_model on fct_engagement. Leadership wants the effect on margin. Explain why a two-way fixed effects regression of margin on a treated indicator is not trustworthy here, name the estimator you would use instead and what identifies it, and specify how you would obtain a defensible p-value from five clusters.

Approach

Explain the two-way fixed effects failure concretely. Under staggered timing the coefficient is a weighted average of every available two-by-two comparison, and that set includes comparisons that use already-treated units as controls for later-treated ones. Those comparisons difference out not only the common time shock but also the earlier cohort's ongoing treatment-effect growth, so when effects grow after adoption the aggregate is pulled toward zero and can flip sign. Decomposed onto the underlying ATT(g,t), the implied weights can be negative, which is why the coefficient is not an average of treatment effects at all. This is a bias in the estimand, not a small-sample artefact, so more data does not fix it.
Switch to a heterogeneity-robust estimator: Callaway and Sant'Anna group-time ATT(g,t) using not-yet-treated units as the comparison group, aggregated into an event-study path and a single summary. Sun and Abraham's interaction-weighted estimator or Borusyak-Jaravel-Spiess imputation answer the same objection. With every area eventually treated, identification rests entirely on not-yet-treated windows, so the last-adopting area carries disproportionate weight; say that out loud rather than burying it.
Test parallel trends honestly. Plot event-time leads with confidence intervals and refuse to read a non-rejection as confirmation, because at five clusters that test is badly underpowered. Pair it with a sensitivity analysis that bounds the post-treatment violation as a multiple of the largest pre-period violation and reports the multiple at which the conclusion breaks.
Fix the inference, and be precise about what the bootstrap can deliver at five clusters. Five sits far below the roughly forty at which cluster-robust standard errors become approximately valid, and below that they are sharply downward-biased. A wild cluster bootstrap with Rademacher weights draws from 2^5 = 32 sign vectors, but the null-imposed bootstrap t-statistic is odd in the sign vector: negating every weight negates t* and leaves |t*| unchanged, because the coefficient deviation is linear in the weights while the cluster-robust variance is quadratic in them. The 32 vectors therefore collapse to 2^4 = 16 distinct values of |t*|, so the smallest attainable two-sided p-value is 1/16 = 0.0625. That is above 0.05, so at five clusters a Rademacher bootstrap cannot reject at the conventional level however large the true effect is. Use Webb six-point weights, whose 6^5 = 7,776 vectors collapse to 3,888 distinct |t*| and a floor near 0.00026, or randomisation inference permuting the five observed adoption dates across the five practice areas, which gives 5! = 120 assignments and a floor of 1/120 = 0.0083.
Deal with the mix confound before reporting anything. Fixed-fee margin falls with hours worked while uncapped time-and-materials margin rises with them, so a rollout that coincided with a pricing-mix shift contaminates the pooled estimate. Estimate within pricing_model and report the within-model effects alongside the mix decomposition.

Worked solution (45 min)

Build the engagement-quarter panel with an adoption quarter per practice area, keyed on the quarter the process went live rather than when it was announced.
Run the Goodman-Bacon decomposition of the two-way fixed effects estimate and read off how much of the total weight sits on already-treated-as-control comparisons; that share is the diagnostic there, since those weights are non-negative by construction. For negative weights on the underlying ATT(g,t), run the de Chaisemartin and D'Haultfoeuille decomposition and report the negative share.
Estimate ATT(g,t) with not-yet-treated controls, aggregate to event time, and plot leads and lags with confidence bands.
Compute the p-value three ways, cluster-robust, Rademacher wild bootstrap and Webb wild bootstrap, and report the Webb figure while noting that the Rademacher floor at five clusters is 1/16 = 0.0625 and therefore cannot clear 0.05 at all.
Re-estimate separately for fixed_fee and time_and_materials engagements and report the mix decomposition against the pooled change.

Expected result: An event-study path whose pre-period leads are statistically indistinguishable from zero, a post-period aggregate ATT reported with a Webb wild-bootstrap p-value, an explicit note that a Rademacher bootstrap at five clusters cannot produce a p below 1/16 = 0.0625 and so cannot reject at 0.05 on any data, and separate estimates by pricing model with the mix shift quantified.

Verify it yourself

Goodman-Bacon weights are non-negative and sum to one by construction, so a negative one there is an implementation bug rather than a finding; the number to read is the share of weight resting on already-treated-as-control comparisons. Negative weights on the underlying ATT(g,t) come from the de Chaisemartin and D'Haultfoeuille decomposition, and a large negative share means the two-way fixed effects coefficient should not be reported as an average effect.
Restricted to one adoption cohort and two periods against not-yet-treated units, the group-time estimator and two-way fixed effects must return the identical two-by-two difference in differences; a gap there is an implementation bug, not heterogeneity.
At five clusters, enumerate all 32 Rademacher sign vectors rather than drawing B of them. The resulting |t*| must come back as 16 identical pairs, one per sign-flip orbit; 32 distinct values means the null is not being imposed or the statistic being bootstrapped is not the cluster-robust t.
Wild bootstrap p-values should be weakly larger than cluster-robust ones at five clusters; if they come out smaller, check that the null is being imposed in the bootstrap.
Within-pricing-model effects plus the mix-shift term must reconcile to the pooled period-over-period change.

Follow-up

If the rollout order was not random, say the worst-performing practice area went first, what does that do to parallel trends and what would you do about it?
How would you choose between a synthetic control built on a single practice area and the group-time estimator?
What would the event-study path have to look like for you to believe the effect is real rather than a continuation of a pre-existing trend?

Realisation rate steps up overnight after a warehouse release

Medium · definition change · realisation · restatement

Realisation rate for one practice area steps from 0.93 to 1.02 on a single reporting date. Bill rates, hours worked and the client set are unchanged. The metric is SUM(amount_usd) from fct_invoice_line over SUM(hours * bill_rate_usd) from approved billable fct_time_entry rows for the same engagement and period. A warehouse release shipped that week. Determine what in the definition can produce a level shift with no operational change, and deliver a back-restated series plus the definition you would keep.

Approach

Treat the shape as evidence: a definition change is a discontinuity at one processing boundary, while a behaviour change ramps over weeks. Plot daily, not monthly, to see which one this is before forming any hypothesis.
Diff the metric's lineage across the release and enumerate the candidate predicates: which line_type values enter the numerator (dropping credit_note and write_off inflates it), whether the period join moved from period_start_date to issued_at, and whether the denominator counts status = 'approved' only or now includes 'submitted'.
Run both definitions over the same fixed pre-release month. If old and new differ by approximately the size of the step, the step is definitional; if they agree, look for a behaviour change coinciding with the release date.
Sanity-check the new level against the metric's physics: values above 1.0 are legitimate only on fixed_fee work delivered under its hour budget, so a practice dominated by time_and_materials sitting at 1.02 is itself a red flag.
Restate the full history under the chosen definition, publish old and new series together with the changeover marked, and add a reconciliation test that recomputes one known engagement-month from raw rows.

Follow-up

Which definition is correct, and what does excluding write-offs from the numerator do to the guardrail relationship with utilisation?
How do you prevent the next release from silently restating a headline metric?
If the release also changed the denominator's status filter, which direction does that push realisation and by roughly how much?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Describe a time you had to explain a complex technical finding to a non-technical stakeholder.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Three requests, one week, and the one you defer

Medium · prioritisation · stakeholder management · scope reduction

Three requests arrive in one week. Finance needs days sales outstanding recomputed for a board meeting in four days. A partner wants a proposal win-rate model for a pursuit review in three weeks. Delivery wants a staffing forecast, with no date attached. You have one week of your own capacity and no analyst. Write the prioritisation you send back, the request you defer, and the message you send to the person whose request you defer.

Approach

Name the probe: whether you prioritise on decision dates and reversibility, or on who asked most forcefully.
Score each request on four things: what decision it changes, the date that decision is made, the cost of being late, and the cost of being wrong. A board figure has a hard date and a high cost of being wrong; a forecast with no date has neither.
Reduce scope rather than dropping work. The win-rate model is the largest piece and the easiest to get wrong, because features written after the decision, such as engagement_id and revised pricing, leak the label, and because withdrawn and no_decision proposals are not missing at random. A two-day descriptive win-rate cut by is_competitive and loss_reason answers most of what a pursuit review needs, with the model scoped separately.
Defer explicitly, with a date and a smaller substitute, rather than leaving a request to decay quietly. Silence is read as agreement and then as failure.
Put the trade-off in writing so it can be overturned by someone with more context than you have, and say what you would drop if the deferred request becomes urgent.

Follow-up

The partner escalates to the practice lead. What do you change, and what do you refuse to change?
What evidence would make you drop the finance work instead?

Disagreeing with a proposed utilisation target using realisation evidence

Medium · realisation · guardrail metrics · observational evidence

A delivery lead proposes raising the billable utilisation target for analyst through senior_consultant from 72 to 85 percent. You have fct_time_entry, including is_billable, bill_rate_usd and written_off_hours, and fct_invoice_line. You believe the target will raise reported utilisation and lower fees. Prepare the disagreement: the evidence you pull, the mechanism you name, the metric pair you propose instead, and the condition under which you would concede that the target is correct.

Approach

Name the probe: whether you disagree with a mechanism and a measurement, or with an opinion about a metric being bad.
State the substitution precisely. Utilisation counts approved hours with is_billable = TRUE. An hour that is charged to the client and later written off stays in that numerator, so utilisation is unaffected while realisation, fees divided by hours times bill_rate_usd, falls and margin falls with it. That is the exact channel by which a higher target can raise the reported number and lower revenue.
Pull the evidence at consultant-month grain: plot realisation and the write-off share, written_off_hours over billable hours, against utilisation decile. If the current top decile already shows lower realisation, the proposed target moves a large share of the staff into that regime.
Stratify before concluding. Fixed_fee teams can show high utilisation and high realisation for reasons that have nothing to do with the proposal, so run the comparison within pricing_model and report the mix.
Propose the pair rather than the veto: utilisation published with realisation and write-off rate as standing guardrails, with the threshold at which the combination is net positive stated in advance. Then name your concession condition: if the top utilisation decile shows no realisation penalty and bench hours are the binding constraint, the target is right and you will say so.

Follow-up

Utilisation and realisation are computed from overlapping hours. Does that make the relationship you found mechanical rather than behavioural?
How many consultant-months would you need to detect a three-point realisation move, and does the firm have them?