Interview guides · Cobblestone Energy

Cobblestone Energy Data Scientist Interview Questions & Guide 2026

Data scientist interview prep for Cobblestone Energy: original practice on experiment design, SQL and metric reasoning, with worked solutions.

The 60-second brief

A Data Scientist at Cobblestone Energy operates at the critical intersection of quantitative modeling, software engineering, and commercial energy trading. Unlike traditional data science roles that focus on offline analysis or long-term product features, data science here is directly integrated into the core trading engine. You will be responsible for building, refining, and scaling predictive models that forecast volatile electricity prices, weather patterns, grid constraints, and generation mixes across major European and UK power markets.

Product-sense cases reward reasoning from a mechanism to a testable prediction. Reciting every metric you can name reads as pattern matching; naming the single quantity that would move if your explanation were true reads as thinking.

Focus areas

Join daily snapshots to shipment events safely
Trace an on-time miss to one node
Evaluate forecasts at the lag ordering actually uses

Overview

The impact of this role is immediate and highly visible. Your models, analyses, and trade ideas will directly influence real-time trading decisions, directly impacting the firm's performance and profitability. To succeed, you must possess not only exceptional mathematical and programming skills but also a sharp commercial mindset that allows you to translate complex statistical patterns into actionable market strategies.

This position is highly challenging and intellectually demanding, requiring you to handle massive, messy datasets under tight time constraints. For the right candidate, it offers an unparalleled opportunity to see your work directly validated by the financial markets every single day, working alongside some of the sharpest quantitative minds in the proprietary trading industry.

The Cobblestone Energy interview process

Cobblestone Energy candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Application Review

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Mathematical Intuition Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Research Project

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Market Study Round

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Cultural Fit Evaluation

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Where candidates lose points

Attributing demand variability to the node where it is observed

Order variance amplifies as it moves upstream: batching to a truckload, minimum order quantities, forecast-driven ordering and promotional pull all convert smooth end demand into lumpy replenishment orders, so a plant can see a coefficient of variation several times that of the underlying consumption. Diagnosing the plant's schedule instability as a plant problem then produces interventions that cannot work, because the generating process sits one or two echelons downstream. Measure the bullwhip ratio explicitly (variance of orders placed by a node over variance of demand it received) at each echelon, and fix the ordering rule at the node where the ratio jumps rather than the node where the pain is felt.

Scoring intermittent demand with MAPE

MAPE is undefined whenever the actual is zero, which is most days for a C-class item at a forward-stocking location, and it is asymmetric even where it is defined: under-forecasting is bounded at 100 percent error while over-forecasting is unbounded. Optimising it therefore drives systematic under-forecasting on exactly the items whose stockouts are most expensive, and the bias is invisible in the headline because the zero-actual rows were dropped before averaging. Use WMAPE (sum of absolute errors over sum of actuals), a scaled error such as RMSSE, or a pinball loss at the service quantile the policy targets, and always state the aggregation level and forecast lag, because the same series scores very differently at daily SKU-store level and weekly SKU-region level.

Crediting a treatment for regression to the mean

Selecting a group because it is extreme (lowest-engagement users, accounts having their worst month, the bottom decile of a score) moves that group's expected next-period value back toward the average even under no treatment, by exactly as much as the selecting measure is imperfectly correlated with its own later value. Compare against units that met the same selection rule and went untreated, or use two pre-periods so the bounce-back is visible before the intervention starts. A pre-post number on a group chosen for being extreme measures the selection rule, not the treatment.

SQL that silently fans out on a one-to-many join

State the grain of each table and the grain you want in the result before writing the join. Pre-aggregate the many side to the join key, or use EXISTS or a window function, and verify with a row count against COUNT(DISTINCT id) rather than trusting that the numbers look plausible.

Questions & practice

Calculate 18 multiplied by 15 mentally.

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.
Write down the assumption the method needs before you use the method.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

Solve the following sequence pattern: 2, 6, 12, 20, 30,... What is the next number, and what is the underlying mathematical rule?

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Say what the estimate is of, and over what population it generalises.
Write down the assumption the method needs before you use the method.

Follow-up

How would you explain this result to someone who does not know statistics?
Which assumption here is most likely to be violated in practice?

A box contains 3 red marbles and 7 blue marbles. If you draw two marbles sequentially without replacement, what is the probability that both are blue?

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Sanity-check the answer against a simple bound or a simulated case.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

Given a dataset of historical power grid load and temperature anomalies, how would you structure a machine learning model to predict peak demand periods?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.
Say how the offline result would be validated online before it is trusted.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Cluster bootstrap for landed cost per delivered unit

Medium · bootstrap · clustering · ratio estimator · mix shift

You have delivered outbound legs from fct_shipment_leg: leg_id, shipment_id, leg_seq, carrier_id, origin_location_id, destination_location_id, shipped_units, cube_m3, linehaul_cost_cents, fuel_surcharge_cents, accessorial_cost_cents, expedite_premium_cents. In this extract the linehaul for a multi-stop load is booked entirely on leg_seq 1, so allocate it across the load's legs by cube first. Then estimate the difference in landed cost per delivered unit between two carriers on lanes both serve, with a 95 percent interval. Write the resampling yourself; no bootstrap library.

Approach

Allocate the linehaul before anything else, in proportion to each leg's cube over the load's total cube, and keep the other three cost columns where they already sit. State the choice: allocating by weight instead reorders lanes whenever freight is bulky rather than dense.
Restrict to the set of lanes both carriers actually serve in the window. Comparing over all lanes measures which carrier was assigned the cheap lanes, not which carrier is cheaper.
Resample shipments, not legs. Legs of one load share a single allocated linehaul and a single dispatch decision, so treating them as independent draws understates the variance of the estimate.
Recompute the statistic as a ratio of sums on every resample, total allocated cost over total shipped units. Taking the mean of leg-level cost-per-unit values instead gives a different estimand in which a one-unit leg counts as much as a full truckload.
Report both the raw difference and a lane-standardised difference where lane weights are fixed at the pooled volume share, and say which one you would put in front of a decision maker.

Worked solution (35 min)

Compute per-shipment total cube, allocate the leg_seq 1 linehaul across legs by cube share, and form leg_total_cost from the four cost columns.
Build the lane key from origin_location_id and destination_location_id, and keep only lanes with delivered volume from both carriers.
Compute the point estimate per carrier as sum(leg_total_cost) / sum(shipped_units), and take the difference.
Draw 2,000 bootstrap replicates by sampling shipment_ids with replacement within each carrier, rebuilding both sums from the sampled legs and recomputing the ratio difference.
Take the 2.5th and 97.5th percentiles of the replicate differences, then repeat the whole procedure with resampling stratified inside lane to produce the mix-standardised interval.

Expected result: A point difference in cents per delivered unit with a 95 percent percentile interval from 2,000 replicates, reported twice: raw, and standardised to a fixed lane mix. The two differ whenever the carriers' lane mixes differ, and can differ in sign.

Verify it yourself

Run the same bootstrap resampling legs independently instead of shipments: that interval should be narrower, and the gap is the intra-load correlation you would have thrown away.
Confirm the mean of the bootstrap replicates is close to the point estimate; a large gap indicates a bias worth investigating before quoting the interval.
Confirm total allocated linehaul per shipment equals the original leg_seq 1 linehaul to within rounding, and that no leg received a negative allocation.
Confirm the interval narrows roughly with the square root of the number of shipments by rerunning on a random half of the loads.

Follow-up

The interval crosses zero. What would you need in volume or in window length to resolve a difference of 2 cents per unit?
One carrier's accessorials are rising while its linehaul is flat. What is that signature telling you about execution versus rates?
How does your interval change if one shipment accounts for 15 percent of the units?

Bullwhip ratio by echelon from daily pipeline deltas

Hard · window functions · bullwhip · variance ratio

fct_inventory_daily gives on_order_qty (ordered, not yet shipped) and in_transit_qty (shipped, not yet received) per sku x location x date; dim_location gives echelon and parent_location_id. There is no replenishment-order table, so derive placements: a new order raises on_order, a shipment moves quantity from on_order to in_transit, and a receipt lowers in_transit, so placed_qty on a date is the positive part of the day-over-day change in (on_order_qty + in_transit_qty). Compute weekly placements per node and the bullwhip ratio against the demand that node received, reported per echelon. State what the derivation misses.

Approach

Per sku-location, take LAG(on_order_qty + in_transit_qty) OVER (PARTITION BY sku_id, location_id ORDER BY inventory_date) and set placed_qty = GREATEST(pipeline - lag_pipeline, 0). The first date in each partition has no lag and must be dropped, not read as a placement equal to the whole standing pipeline.
Name the blind spots before anyone else does: an order placed and received inside one day nets to zero, a cancellation is indistinguishable from a receipt, and minimum order quantities plus lot-size multiples make placements lumpy by construction rather than by anyone's behaviour.
Define received demand by echelon. At the lowest echelon it is demand_qty in the same table. At a parent node it is the sum of its children's placed_qty joined through parent_location_id, not the parent's own demand_qty column, which is close to empty for a pure replenishment node.
Aggregate both series to weekly buckets before computing dispersion, because day-of-week ordering patterns otherwise dominate the variance, and require a minimum number of weeks before quoting any ratio.
Compute CV = stddev_samp / NULLIF(avg, 0) on each weekly series and report the ratio of squared CVs, stating that convention explicitly: a raw variance ratio is not level-free and is not comparable across nodes of different size.
Read the output by echelon and locate where the ratio first crosses one going upstream. That is the node whose ordering rule amplifies, and it is usually not the node where the schedule instability is being felt.

Worked solution (45 min)

CTE pipeline: select sku_id, location_id, inventory_date, on_order_qty + in_transit_qty AS pipe, and LAG(pipe) over the partition; drop rows where the lag is NULL and count them.
CTE placements: placed_qty = GREATEST(pipe - lag_pipe, 0), aggregated to sku-location-week.
CTE received: weekly demand_qty per node for the lowest echelon, and for higher echelons the weekly sum of children's placed_qty joined on parent_location_id.
Join the two weekly series per node, require a minimum week count, and compute stddev_samp and avg on each side.
Emit echelon, node, weeks_observed, cv_orders, cv_demand, bullwhip = (cv_orders / cv_demand) ^ 2, and the dropped first-date row count.

Expected result: One row per node with its echelon, weeks_observed, cv_orders, cv_demand and bullwhip as the squared CV ratio, plus an echelon-level rollup and the count of partition-start rows excluded for having no lag value.

Verify it yourself

Total placed_qty over a long window is close to the total of receipts implied by falls in in_transit_qty. A large shortfall means same-day place-and-receive is common and the derivation is understating placements.
Nodes at echelon 1 or 2 with no children under parent_location_id are excluded explicitly rather than silently producing a NULL or zero demand denominator.
Recomputing the CVs on daily instead of weekly buckets changes the ratio. An identical ratio means the weekly aggregation never took effect.
placed_qty is never negative, and its maximum is bounded by the largest single-day pipeline increase in the raw data.

Follow-up

A node shows a ratio near 3. Name three ordering rules that produce that and say how you would tell them apart using only these tables.
Why might a plant's schedule instability be untouchable from inside the plant, and what would you change instead?
What would you want logged so that placements no longer have to be inferred from pipeline deltas, and what is the smallest such change?

Perfect order rate against the original promise in local time

Hard · timezone boundaries · order-grain rollup · semi-join

Compute weekly perfect order rate at order grain. An order qualifies only if every fct_order_line has line_status = 'shipped_full', every line's actual_delivery_at_utc falls on or before original_promised_delivery_date at the ship-from node's local end of day (dim_location.timezone holds IANA names), and no fct_shipment_leg reached through shipment_id carries exception_code in ('damage','address_issue') or leg_status in ('refused','lost'). The denominator is orders whose original_promised_delivery_date lands in the week. Undelivered lines fail. Publish with a fourteen-day lag and declare invoice accuracy as out of scope.

Approach

Convert the event into local time rather than comparing against a UTC-anchored deadline: (actual_delivery_at_utc AT TIME ZONE 'UTC' AT TIME ZONE loc.timezone)::date <= original_promised_delivery_date. Casting the promise date to a timestamp puts the cut-off at UTC midnight starting the promise day, which is up to a full day early and wrong in both directions depending on the node's offset.
Evaluate the three components as booleans at line grain, then roll to the order with BOOL_AND, coercing a NULL actual_delivery_at_utc to FALSE explicitly so an undelivered line fails rather than propagating NULL and dropping the order out of the comparison.
Test the shipment exceptions with EXISTS against fct_shipment_leg, never with a join. One shipment has many legs and one order has many lines, so a join multiplies the order into the denominator and changes the metric.
Aggregate to order grain first, then count qualifying orders over orders promised in the week. The rate is orders over orders; a line-level average is a different and easier metric.
Apply the publication lag by restricting to weeks whose promise dates are at least fourteen days old, and emit the three component pass rates beside the headline so any movement is attributable to completeness, timeliness or exceptions.

Follow-up

All three component rates are flat but perfect order fell two points. How is that arithmetically possible, and what does it tell you about where the failures now land?
A node sits at UTC+13 and many orders are promised on the last day of the month. What does that do to your weekly and monthly buckets?
How do you handle an order whose lines ship from two nodes in different timezones, and which node's clock owns the promise?

If wind generation in the UK drops unexpectedly by 40% over a 12-hour period, which power generation assets would you expect to ramp up, and what would be the expected impact on spark spreads?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Describe a recent major weather event (e.g., a severe winter storm or …

Medium · product sense and metrics

Describe a recent major weather event (e.g., a severe winter storm or a summer heatwave) and explain step-by-step how it would impact local electricity demand, supply generation, and short-term energy prices.

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

A planner scorecard that aggregation cannot flatter

Hard · wmape · gameable metrics · forecast bias

Demand planners will be scored on forecast accuracy. fct_inventory_daily carries demand_qty, forecast_qty_lag7, forecast_qty_lag28 and stockout_flag at sku by location by date, with NULL forecasts before a series existed. Specify the scorecard: error metric, grain, lag, exclusion policy and guardrail. Then show the three ways a planner raises the headline without forecasting better, and state which single number you would prefer to accuracy if you could publish only one.

Approach

Fix the grain and lag to the decision. The ordering policy consumes a forecast at one horizon for one sku-location-week cell, so the score is computed there; a number quoted at any other grain is not the number the policy used and cannot be compared to it.
Choose WMAPE, sum of absolute errors over sum of actuals, or a pinball loss at the service quantile the policy targets. Rule out MAPE explicitly: it is undefined at zero actuals, which is most days for a C item at a forward-stocking location, and it is asymmetric where it is defined, so optimising it drives systematic under-forecasting.
Show why aggregation flatters, and show it as arithmetic rather than opinion. The absolute value of a sum is at most the sum of absolute values, so errors cancelling across SKUs, locations or weeks make aggregated WMAPE weakly lower than the disaggregated figure on the same denominator, with no forecasting improvement involved.
Name the other two routes: scoring at a shorter lag than ordering actually uses, and exclusion drift, where NULL-forecast rows, zero-demand cells or stockout days are dropped without publishing the excluded count, quietly removing the hardest cells from the sample.
Set the guardrail on signed bias, weighted mean percentage error over the same window and cells, reported per ABC/XYZ cell. A series can be inaccurate but unbiased, or accurate on average and persistently short, and only the second wrecks inventory in one direction.
Answer the last question directly: prefer the outcome that accuracy is a means to, achieved fill rate at a given inventory at cost per cell, because that is what the business buys, and accuracy that does not move it is not worth paying for.

Worked solution (40 min)

Build the scored panel at sku by location by week, summing demand_qty and taking the forecast at the lag the ordering policy uses, excluding NULL-forecast cells and counting every exclusion.
Compute WMAPE and signed WMPE per planner per ABC/XYZ cell by re-summing numerator and denominator, never by averaging cell ratios.
Recompute the same WMAPE at region and at month to demonstrate the mechanical improvement, and record the difference as the aggregation premium.
Recompute with and without stockout days and with and without zero-demand cells, recording each shift as the exclusion premium.
Publish WMAPE, WMPE, the excluded-row count and the paired inventory-and-service outcome, with grain and lag printed on the page rather than held in a footnote.

Expected result: A scorecard pinned to sku by location by week at the ordering lag, using WMAPE with signed bias beside it and per-ABC/XYZ reporting so the tail cannot be abandoned; three gaming routes each sized on the real data as an aggregation premium, a lag premium and an exclusion premium; and a stated preference for achieved fill rate at a given inventory at cost as the single number, with forecast accuracy kept as a diagnostic rather than as the target.

Verify it yourself

Aggregated WMAPE is less than or equal to disaggregated WMAPE on the same rows in every roll-up tested; a case where it is higher means the two were computed over different denominators.
On one panel, the absolute value of WMPE is at most WMAPE; a violation means the two used different exclusions.
Excluded rows plus scored rows equal the full panel; if they do not, rows are being dropped somewhere other than the stated policy.

Follow-up

Two planners have identical WMAPE and opposite bias. Which one is doing more damage, and to what?
The scorecard is per planner but forecasts are hierarchical and reconciled. Whose number is the reconciled one, and who gets charged for the reconciliation loss?

On-time rate improved the week the promise field changed

Medium · definition change · metric hygiene · release artefact

Customer on-time delivery jumped from 88 percent to 95 percent in one week and has stayed there. That week a release began writing re-confirmed dates into fct_order_line.promised_delivery_date, while original_promised_delivery_date still carries the first promise and both columns are populated for all history. Using fct_order_line (actual_delivery_at_utc, promised_delivery_date, original_promised_delivery_date, line_status) and dim_location.timezone, determine how much of the seven points is real. Deliverable: two numbers that sum to seven and one sentence of recommendation.

Approach

Recompute the whole series twice on identical rows, once against promised_delivery_date and once against original_promised_delivery_date. Both columns exist for all history, so the counterfactual is free and the split is arithmetic rather than an estimate.
Read the gap between the two series in the weeks before the release. If it was near zero and opens abruptly in the release week, the jump is the measurement changing. If the original-promise series also rises, that component is real and should be credited.
Measure the mechanism directly rather than inferring it: by week, the share of lines where promised_delivery_date > original_promised_delivery_date and the mean slip in days. Any metric that can be satisfied by moving its own target is not measuring service.
Handle the date-versus-timestamp comparison explicitly. actual_delivery_at_utc is a UTC instant while the promise is a local date, so convert using dim_location.timezone and compare against local end of day. Doing the comparison in UTC moves an entire timezone's deliveries across the boundary and invents a second, smaller step.
Recommend the original-promise definition for the published metric and keep the re-promise series as a separate operational measure, stating that the two answer different questions rather than that one is simply wrong.

Follow-up

Re-promises are legitimate when a customer agrees to the new date. How would you let those count without reopening the loophole?
What automated check would catch the next definition change before it reaches a chart?
Supplier inbound on-time has the same structure. Where is its equivalent loophole, and what does the metric tree already say about it?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Describe a time when you had to learn a highly complex, unfamiliar topic in a very short timeframe. How did you approach it, and what was the outcome?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

What drives your intellectual curiosity, and how do you maintain high performance during long, intense analytical projects?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Defend a finding that the expedite program bought nothing

Medium · defending findings · counterfactual reasoning · landed cost

Over two quarters, expedite_premium_cents on outbound legs in fct_shipment_leg rose from 0.4 to 1.3 percent of landed cost, while perfect order rate moved from 91.2 to 91.5 percent, inside the week-to-week spread. The transport lead who sponsored the expedite program disputes the finding in a review in front of his director, arguing that your window contains a port disruption that would have made service worse without the spend. Present the finding, say what you concede on the spot, say what you hold, and name the evidence that would change your conclusion.

Approach

Restate his objection in its strongest form before answering it, because a counterfactual worsening is a legitimate argument and treating it as an excuse ends the conversation.
Separate what you measured from what you claimed: the data show no detectable service gain, not that expedite has no effect, and the difference is the entire argument.
Test his hypothesis with the data you already have rather than defending in the abstract: split legs by exception_code = 'customs_hold' and by lane, and compare expedited against non-expedited legs on the same lanes in the same weeks, since if expedite were holding the line the expedited lanes should show a service gap over comparable non-expedited ones.
Concede what is true: without a holdout you cannot rule out a protective effect, and the pre-period is contaminated by the disruption, so the honest statement is an upper bound on the gain rather than a zero.
Hold the part that survives: the spend is real, it is concentrated in a small set of lanes, and nobody set a decision rule for when a leg gets expedited, which is a controllable problem independent of the counterfactual.
Name the next measurement: a lane-level staggered switch-off with a stated burn-in, and say what it would cost and how long it would take.

Follow-up

He offers to run the switch-off only on his two best lanes. Why is that a problem, and what do you counter with?
His director asks you for a yes or no on cutting the budget today. What do you say?