Delivery ETA Data Scientist Interview Case: Calibration, Late Deliveries, and Dispatch Bias

An ETA model lowers mean absolute error, yet more customers receive their orders after the promised time. The dispatch team says the model looks accurate on completed deliveries. The customer team says the promises are getting worse. A delivery ETA data scientist interview case asks you to reconcile those observations before choosing a new model.

Start with three separate contracts: the duration predicted at a particular timestamp, the deadline shown to the customer, and the dispatch policy that determines which outcomes become observable. A good completed-order forecast does not automatically produce good promises or better assignments.

This article uses original synthetic cases. Official engineering documentation and primary research are labeled where introduced. No candidate reports establish an employer-specific assessment format.

What does “delivery ETA” actually predict?

Define the target as elapsed minutes between a prediction timestamp and the delivery event. At checkout, the remaining duration includes unassigned waiting, merchant preparation, pickup, travel, and handoff. At pickup, most of those components are already historical. A model evaluated at pickup cannot justify promises made at checkout.

Persist a prediction record with order ID, prediction time, horizon, feature snapshot, model version, dispatch version, predicted quantiles, and the deadline displayed. Keep timestamps in one documented convention and define what counts as delivered. Courier arrival at a building and customer handoff are different events.

Official engineering example: Uber's 2022 DeepETA article describes a routing estimate followed by a learned residual correction, using spatial, temporal and request-related information. That illustrates one engineering decomposition; it is not a current specification for every delivery business or a required interview architecture. DeepETA.

For the case, assume the customer sees a checkout deadline and the dispatch model scores alternatives before assignment. State both assumptions. They determine available features, comparable labels, and how the resulting prediction is used.

Why can identical MAE produce different late-delivery rates?

Let signed error be actual duration - quoted duration. Positive error means the order arrived later than quoted. This sign convention must be stated because reversing it reverses the interpretation.

Consider four completed orders. Model A quotes two minutes longer than each actual duration; model B quotes two minutes shorter. Assume the quote itself is the deadline, with no grace period.

Model	Signed errors, minutes	MAE	Arrived after quote
A	-2,-2,-2,-2	2 minutes	0/4
B	+2,+2,+2,+2	2 minutes	4/4
C	+1,+1,+1,+1	1 minute	4/4

C improves MAE over A while worsening the late fraction. The example disproves the idea that smaller symmetric error necessarily improves an asymmetric customer promise. These four records do not establish a population effect, statistical significance, or a useful production policy.

Also report late minutes, not just the late flag. One late minute and thirty late minutes can have different customer consequences. If the product promises a window, define lateness against its upper boundary. Record whether refunds or operational escalation use another threshold.

Do not solve the problem by padding every ETA indefinitely. Long promises may reduce conversion, and excessive slack can hide operational deterioration. A decision needs both the cost of missed promises and the cost of unnecessarily conservative promises, under explicit assumptions.

How should a p90 ETA be evaluated?

A conditional 90th-percentile forecast aims to place roughly 90% of comparable future durations at or below the quoted value. It is a one-sided upper quantile, not the same object as a central 90% interval. A central interval often uses lower and upper quantiles such as p05 and p95.

Official metric definition: scikit-learn's pinball-loss documentation identifies the quantile parameter and its asymmetric loss. For residual e = actual - prediction, use q×e when e >= 0, otherwise (q-1)×e. At q = 0.9, underprediction is penalized more heavily than equal-sized overprediction. Pinball-loss reference.

For an actual duration 30 minutes, predicting 28 has p90 loss 1.8; predicting 32 has loss 0.2. That loss encodes an asymmetry. Choosing q = 0.9 should still reflect the product's decision costs rather than a belief that p90 is universally appropriate.

Measure empirical upper-quantile coverage: the fraction with actual <= predicted_p90. Examine the size of exceedances and how conservative quotes become. A system quoting ten hours for every delivery can have excellent coverage and terrible usefulness.

The official quantile-regression interval example evaluates predictions beyond their nominal quantile labels. The relevant lesson is to check realized coverage; training with a quantile objective is not a calibration certificate.

Can pooled coverage hide a peak-hour problem?

Suppose ten orders all receive a p90 quote of 40 minutes. Eight off-peak orders arrive in 30 minutes. Two peak orders arrive in 30 and 50 minutes. Nine of ten arrive by the quote, so pooled empirical coverage is 90%.

Slice	On time by p90 quote	Empirical coverage
Off-peak	8/8	100%
Peak	1/2	50%
Pooled	9/10	90%

The pooled number exactly matches the nominal target while the peak slice misses it badly. These samples are intentionally tiny: neither 100% nor 50% is a precise estimate of future coverage. The asset demonstrates aggregation, not a statistically established calibration gap.

With realistic data, inspect coverage across forecast horizon, market, merchant volatility, weather, and quoted-duration buckets. Predefine important slices, show counts and uncertainty, and distinguish exploration from launch criteria. A global calibration adjustment may help the average while leaving a high-risk slice undercovered.

Calibration over the observed population is weaker than calibration conditional on every feature combination. Do not claim the latter from a handful of slice plots. Use the plots to discover where additional data, features, or conservative fallbacks deserve testing.

What would you calculate in a small Python check?

def pinball(actual, forecast, q):
    e = actual - forecast
    return q * e if e >= 0 else (q - 1) * e

def upper_coverage(actual, upper):
    if len(actual) != len(upper) or not actual:
        raise ValueError("aligned nonempty observations required")
    return sum(y <= p for y, p in zip(actual, upper)) / len(actual)

assert abs(pinball(30, 28, .9) - 1.8) < 1e-12
assert abs(pinball(30, 32, .9) - .2) < 1e-12
actual = [30] * 8 + [30, 50]
assert upper_coverage(actual, [40] * 10) == .9
assert upper_coverage(actual[-2:], [40, 40]) == .5

These checks were executed for the fixed arrays. They verify arithmetic and the equality convention, not a trained model, statistical interval, or marketplace simulation. In production, also reject mismatched horizons and duplicate prediction records; align each outcome with the quote the customer actually saw.

How do feature timing and incomplete labels distort evaluation?

A feature's event time is not necessarily its availability time. A scan recorded before checkout but ingested afterward was unavailable to the checkout model. Point-in-time feature joins need both clocks where delayed ingestion matters.

Actual pickup time, final route, eventual support contact, and a corrected merchant preparation duration can leak the outcome into an earlier forecast. Reconstruct snapshots as the service would have seen them. Separate predictions made at different lifecycle stages rather than pooling easier late-stage forecasts with early promises.

Use rolling temporal splits and prevent an order's repeated forecasts from landing in both training and evaluation partitions. Keep a final untouched period, and inspect new merchants or markets separately when deployment includes cold starts. A random split of highly correlated historical records can overstate the evidence for future performance.

Recently opened orders may not yet have delivery labels. Wait for a stated maturity window and report outstanding counts. Cancelled orders are not delivered in zero minutes. Cancellation can be informative: excluding frustrated customers leaves a more successful population. Analyze cancellations as an operational outcome, and explicitly define whether the duration estimand conditions on completion.

Why does dispatch make historical labels selective?

For one order, suppose the historical dispatcher considered couriers A and B, assigned A, and observed a 24-minute delivery. B's delivery duration for that same order is unobserved. A model cannot treat an unchosen estimate for B as a measured outcome.

Assignment also changes waiting time, batching, courier location, and the options available for other orders. Completed deliveries therefore reflect the behavior of the historical policy. A model that recommends different actions is asking a counterfactual question beyond ordinary held-out prediction on that policy's outcomes.

Keep candidate sets, chosen action, decision timestamp, relevant predecision state, and any actual randomized selection probabilities when permitted. Logging these fields supports later analysis. It does not recreate missing labels for all alternatives or erase marketplace interference.

Primary research boundary: contextual-bandit off-policy evaluation studies estimators for assessing a target policy using observations collected under another policy. That mathematical setting requires assumptions about support, action selection, and outcomes. Wang, Agarwal and Dudík,2016.

Case inference: propensity weighting may help when genuine probabilities are known or defensibly estimated and the target actions have support. It cannot identify outcomes for actions assigned with zero probability. A deterministic dispatcher with no exploration does not supply trustworthy nonzero propensities merely because a classifier can predict its choices. Sequential dispatch and shared supply may also exceed a simple one-step bandit model.

How would you decide whether to launch?

Separate forecast validation from policy validation. First replay prediction-time snapshots against mature observed labels. Compare signed error, MAE, pinball loss, quote coverage, interval width if applicable, and relevant slices with a stable baseline.

Then assess the actual proposed intervention. Shadow scoring can verify latency, feature availability, and feasible recommendations without changing dispatch; it cannot prove the customer effect of assignments never taken. A controlled marketplace experiment may require geographic or time clusters because orders compete for shared couriers.

For a switchback, justify treatment block length, carryover handling, and uncertainty at the assignment structure. There is no universal safe block duration. Customer promises, courier earnings, cancellations, merchant waits, and service latency belong in the decision. Do not launch solely on completed-order MAE while missing-order rates rise.

An interview-ready recommendation is: “I will evaluate checkout quotes against checkout-time features, distinguish symmetric error from promise violations, check p90 coverage by operational slice, and separately test dispatch effects. Completed-order accuracy alone does not identify the result of a new assignment policy.”

Practice ETA and marketplace follow-ups

Question	Specific follow-up
Build model to predict package delivery time	Fix the forecast timestamp and prevent late-signal leakage.
Build ETA prediction and simulate impact	Map quantile errors into customer and dispatch costs.
Design a Ride-Hailing ETA System	Address cancellations and selected completed-trip labels.
Design and analyze a switchback experiment	Explain marketplace interference and carryover.
Improve biker delivery with metrics and levers	Balance delivery outcomes against courier guardrails.

Open the ETA impact exercise and defend which observations can evaluate the forecast, which can evaluate the policy, and which remain missing.

Sources and Further Reading

• Uber Engineering: DeepETA,2022
• scikit-learn: Mean pinball loss
• scikit-learn: Prediction intervals for gradient boosting
• Optimal and Adaptive Off-policy Evaluation in Contextual Bandits,2016