Interview guides · Amazon Advertising

Amazon Advertising Data Scientist Interview Questions & Guide 2026

Amazon Advertising data scientist prep on data pitfalls: selection bias, target leakage, and checking an outlier's provenance before you drop it.

The 60-second brief

As a Data Scientist at Amazon Advertising, you sit at the powerful intersection of massive scale, complex user behavior, and high-stakes business decisions. You will build and optimize the machine learning models, measurement systems, and experimentation frameworks that power ad targeting, bidding strategies, and campaign attribution for millions of global advertisers. The insights and models you produce directly influence multi-billion-dollar revenue streams, making this role both strategically vital and technically demanding.

Allocate prep to your weakest link rather than your favourite topic. Of the three things that usually gate the outcome (SQL that is correct under messy joins, sound reasoning about experiments, and structured framing of an open-ended problem), candidates tend to over-invest in modelling theory and under-invest in framing.

Focus areas

Reason about bid shading under first-price auctions
Design geo holdouts that survive budget interference
Separate invalid traffic from genuine performance shifts

Overview

Your daily work will involve diving deep into petabyte-scale data to solve ambiguous problems that have no pre-written playbook. You might design sophisticated A/B testing architectures to evaluate new advertising features, build attribution models that measure ad effectiveness across multiple touchpoints, or diagnose sudden metric drops in campaign performance. You will collaborate closely with product managers, software engineers, and applied scientists to turn complex statistical insights into production-grade advertising products.

This role requires a rare blend of rigorous statistical thinking, expert coding abilities, and deep product intuition. Expect to be challenged on both your mathematical rigor and your ability to tie analytical solutions back to core business objectives. While the scale of data can be daunting, the opportunity to impact millions of vendors, sellers, and buyers makes it one of the most rewarding data science careers in the industry.

The Amazon Advertising interview process

Amazon Advertising candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interviews

Rounds of this kind usually include one question about work that did not go well, and it is the part that carries the most information. Anyone can narrate a shipped win. What the interviewer learns from a project that stalled is how you behave without a result to hide behind: whether you noticed the problem yourself, how long it took, and who you told. Answers that route the failure onto a data pipeline or a reorganisation close the topic without answering it, and the follow-up comes back to your own part.

What to demonstrate

Whether you found the error yourself or someone else found it, and how long it sat before anyone knew
What you changed afterwards, stated as a check you now run rather than a lesson you now believe
Whether the mistake you choose has real cost attached, such as a quarter of misdirected roadmap or a metric that was reported upward, instead of one that flatters you

How to prepare

Choose a failure you caught yourself and be ready to say what tipped you off. A story where someone else caught it is still usable, but you will be asked why you missed it.
Write down the check you added afterwards and where it lives now, so the correction is a concrete artefact rather than a resolution.
Rehearse saying the cost out loud. Candidates shrink the number by instinct once the interviewer is in the room.

Engagement with Interviewers

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Where candidates lose points

Treating a bid as if it does not affect the price paid, under a first-price auction.

In a second-price auction the winner pays the runner-up's bid, so bidding true value is close to optimal and bid and price are usefully separable. Under first-price clearing, which is now the common case in open exchanges, the winner pays exactly what it bid, so raising the bid raises both the win probability and the cost of every win that would have been won anyway. A bid-increase experiment can therefore show more impressions, more conversions and worse profit simultaneously, and reading only the volume metrics will call it a success. The related second-order problem is that your own bid increase raises the clearing prices your control cells face, which contaminates the comparison unless the cells are separated by market.

Reading a recent week's cost per action or return on ad spend before the conversion window has closed.

Spend for a period is final within hours, but conversions attributed to that period keep arriving for as long as the click window allows plus ingestion lag, and offline uploads can land weeks later. The denominator of cost per action is therefore systematically incomplete for recent dates, so recent CPA is biased high and recent ROAS biased low, and a dashboard comparing a fresh week to a matured week will show a regression that does not exist. The correct handling is to freeze a period only after click_window_days plus the observed ingestion lag, and to publish a maturation curve so readers can see how much of a given day's conversions have landed so far.

Building features from data that postdates the prediction time

Check every feature against the timestamp at which the model would actually score, and drop anything computed from a window that includes or follows the label event. For a forecasting use case, split train and test by time rather than at random, and split by entity when the same entity recurs.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Questions & practice

Explain the trade-offs between parametric and non-parametric tests when analyzing skewed ad-revenue distributions.

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Write down the assumption the method needs before you use the method.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

How does multicollinearity affect your coefficient estimates in a regression model, and how do you mitigate it?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Permutation-test a market-level holdout for incremental conversions

Hard · permutation-test · incrementality · geo-experiment · statistics

geo_daily has one row per (geo_dma, date): arm ('treatment' or 'control'), conversions, conversion_value_usd, spend_usd. Forty DMAs assigned in twenty matched pairs on pre-period conversions, with fifty-six pre-period dates and twenty-eight flight dates. Control DMAs received no delivery for this advertiser and each arm carried its own budget. Without using a library hypothesis test, estimate incremental conversions per $1,000 of treatment spend with a p-value and a 95% interval. State the permutation scheme, why it matches the assignment mechanism, and the minimum effect this design can detect.

Approach

Fix the estimator before looking at the outcome. Per DMA take y as flight-period conversions and x as conversions over a matched-length pre-period window, form the adjusted outcome y - theta * x, and estimate theta from control DMAs only so it cannot absorb any part of the treatment effect. The statistic is then the mean of the paired differences of the adjusted outcome.
Permute within matched pairs, not across all forty DMAs. Assignment was pairwise, so the exact null distribution is the 2^20 sign flips of the pair differences; sample 20,000 of them. Permuting freely across DMAs tests a weaker null, reintroduces between-pair variance the matching already removed, and makes a real effect harder to detect.
Compute the p-value in the plus-one form, (1 + count of |T_perm| >= |T_obs|) / (1 + B). With B replicates a raw count of zero would report p = 0, which is not an attainable statement about a sampled reference distribution.
Build the interval by inverting the test: find the set of shifts delta for which the same permutation test applied to y minus delta times the treatment indicator is not rejected at 0.05. That interval is consistent with the p-value you just reported, whereas a bootstrap over twenty pairs is thin and can disagree with its own test.
Convert to the per-$1,000 scale by dividing the incremental-conversion estimate by treatment spend over 1,000. State the precondition that makes this a causal quantity: each arm carried its own budget, so treatment wins did not starve control. Had they shared one budget, the arms would compete in the same auctions and the difference would include cannibalisation rather than the ad effect alone.
Report power before the p-value. Compute the minimum detectable effect from the observed standard deviation of the pair differences at twenty pairs; if the MDE is larger than any effect the advertiser would act on, say the design cannot answer the question rather than presenting a non-significant result as evidence of no effect.

Worked solution (40 min)

Collapse geo_daily to one row per DMA with pre-period conversions x, flight conversions y, flight spend, and the pair id and arm; confirm every pair has exactly one treatment and one control DMA.
Estimate theta by regressing y on x within control DMAs only, form adj = y - theta * x, and compute the twenty pair differences d = adj_treatment - adj_control and the observed statistic T_obs = mean(d).
Draw 20,000 sign-flip vectors in {-1, +1}^20, compute mean(sign * d) for each, and evaluate the plus-one two-sided p-value against |T_obs|.
Invert the test over a grid of delta to get the 95% interval, then divide the point estimate and both endpoints by treatment spend over 1,000.
Compute the MDE at 80% power from sd(d) and twenty pairs, and run a pre-period placebo by splitting the fifty-six pre-dates into two 28-day halves and testing the second against the first.

Expected result: A point estimate of incremental conversions per $1,000 of treatment spend, a 95% interval from test inversion, a plus-one permutation p-value with B stated, and an explicit MDE. With twenty pairs the p-value grid is coarse: the smallest attainable two-sided p is 1/20,001 at B = 20,000, against an exact-enumeration floor of 2/2^20.

Verify it yourself

The permutation null under sign flipping is symmetric about zero, and its standard deviation is close to sd(d)/sqrt(20), the analytic standard error of the paired mean.
Re-running with theta = 0 leaves the point estimate essentially unchanged and only widens the interval. A shifted point estimate means theta was fitted using treatment data and is absorbing part of the effect.
The pre-period placebo returns no significant lift and, across repeated seeds, p-values that look roughly uniform. A significant placebo means the matching or the pre-period window is wrong, not that the ads worked.
The reported interval contains the point estimate and excludes zero exactly when the p-value is below 0.05; if the two disagree, the inversion grid is too coarse or the statistic differs between the two computations.

Follow-up

One treatment DMA holds 30% of all conversions. What does that do to the permutation distribution, and would you weight, trim, or re-pair?
The flight ends on day 28 but the click window is 7 days. What is your analysis window, and what does cutting the measurement on day 28 cost you?
How would you detect spillover between adjacent DMAs, and what would you do to the estimate if you found it?

Write a query to identify top-performing ad campaigns by click-through rate, handling ties and partitioning by product category.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Optimize a slow-running SQL query that joins multi-terabyte ad impression and conversion tables.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Time-weighted daily budget from slowly changing line item versions

Hard · scd type 2 · window functions · interval weighting · pacing

line_item is SCD type 2: one row per version with line_item_id, daily_budget_usd, pacing_mode, status, valid_from_ts, valid_to_ts (null on the current row) and is_current. ad_impression has line_item_id, served_ts, billed_price_micros_usd and is_billable. For one UTC date, compute budget delivery rate per line item: spend divided by a time-weighted daily budget, where the day's budget is each overlapping version's daily_budget_usd weighted by the fraction of the 24 hours that version was in force. Return spend, weighted budget, the weight coverage, and attainment.

Approach

Select versions that overlap the day with valid_from_ts < day_end AND COALESCE(valid_to_ts, 'infinity'::timestamptz) > day_start. Use strict inequality on both sides so a version that ends exactly at midnight does not also claim the following day, which would make the weights sum above 1.
Clip each version to the day before weighting: start at GREATEST(valid_from_ts, day_start) and end at LEAST(COALESCE(valid_to_ts, 'infinity'::timestamptz), day_end). Weighting on the unclipped interval gives a month-long version a weight of 30 rather than 1.
Derive the weight as EXTRACT(EPOCH FROM (clipped_end - clipped_start)) / 86400.0 and sum daily_budget_usd * weight per line item. Keep SUM(weight) in the output as coverage: it should be 1.0 for a line item live all day and below 1 for one created or ended mid-day, and anything above 1 means the interval predicate is wrong.
If valid_to_ts is unreliable on superseded rows, derive the interval end with LEAD(valid_from_ts) OVER (PARTITION BY line_item_id ORDER BY valid_from_ts) instead, and say which source you trust. Mixing the two produces overlapping intervals that double-count budget.
Aggregate spend in its own CTE at impression grain, filtered to is_billable and bucketed on served_ts in UTC, then join the two pre-aggregated results on line_item_id. Joining the SCD table to impressions before aggregating multiplies every impression by the number of versions.
Compute attainment as spend / NULLIF(weighted_budget, 0), and decide explicitly how to treat versions whose status was paused or ended for part of the day, since including a paused stretch in the denominator reads as underdelivery that never had a chance to happen.

Worked solution (40 min)

Declare day_start and day_end as explicit TIMESTAMPTZ bounds, then select line_item rows where valid_from_ts < day_end AND COALESCE(valid_to_ts, 'infinity'::timestamptz) > day_start.
Add w := EXTRACT(EPOCH FROM (LEAST(COALESCE(valid_to_ts, 'infinity'::timestamptz), day_end) - GREATEST(valid_from_ts, day_start))) / 86400.0 to each surviving row.
Build budget AS (SELECT line_item_id, SUM(daily_budget_usd * w) AS weighted_budget, SUM(w) AS coverage FROM clipped GROUP BY 1).
Build spend AS (SELECT line_item_id, SUM(billed_price_micros_usd) / 1e6 AS spend FROM ad_impression WHERE is_billable AND served_ts >= day_start AND served_ts < day_end GROUP BY 1), entirely independent of the dimension.
FULL OUTER JOIN budget and spend on line_item_id, compute spend / NULLIF(weighted_budget, 0), and inspect the rows where coverage differs from 1.0 as a separate list.

Expected result: Per line item: spend, weighted_budget, coverage and attainment. A budget of $1,000 in force from 00:00 to 18:00 and $2,500 from 18:00 to 24:00 gives weighted_budget = 10000.75 + 25000.25 = $1,375, and spend of $1,210 gives 88.0% attainment at coverage 1.0.

Verify it yourself

For any line item live the whole day, the clipped intervals tile the 24 hours with no overlap and no gap, so SUM(w) is exactly 1.0.
Exactly one row per line_item_id has is_current = TRUE. If more do, the SCD writer is broken and every weight downstream is wrong.
Spend from this query equals a plain aggregate over ad_impression with no line_item join at all. A higher figure is proof the version join fanned out.
A line item with a single version covering the whole day returns attainment identical to the naive spend / daily_budget_usd calculation, which is the degenerate case the weighting must reduce to.

Follow-up

A line item shows 180% attainment. Give three explanations that are not overspend.
How does the same calculation change for lifetime_budget_usd across a multi-week flight rather than a daily budget?
pacing_mode is 'even' but hourly spend is front-loaded and stops at 14:00. Which query distinguishes budget exhaustion from a pacing throttle, and which column settles it?

What metrics do you prioritize when tuning a model for ad-click conversion versus ad-impression delivery?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a product metric to measure the long-term value of a new sponsored-products ad format?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

What core metrics would you track to evaluate the health and success of a new real-time bidding algorithm?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

An advertiser notices a 15% drop in conversion rate week-over-week. How would you investigate and diagnose this metric drop?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Explain how you would run an experiment when traditional randomization at the user level is impossible due to marketplace interference.

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

How do you determine statistical significance when your key metrics exhibit high variance and seasonal spikes?

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

How would you decide whether to launch a feature that increases short-term ad revenue but slightly degrades user experience?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Decide whether a 50.4 to 49.6 arm split is chance

Easy · srm · randomisation · diagnostics

A 7-day two-arm test assigns devices to arm A or arm B by hash of device_id, intended 50/50. Counting distinct device_id in ad_impression, arm A has 1,204,338 devices and arm B has 1,186,902. The hash function is unit-tested and uniform. State whether this split is consistent with fair 50/50 assignment, show the test you use, and give the two mechanisms in a bidding system most likely to produce exactly this kind of imbalance.

Approach

Run a chi-square goodness-of-fit against the intended ratio on the randomisation unit: expected 1,195,620 per arm, so chi-square = 2 * 8718^2 / 1,195,620 = 127 on 1 degree of freedom. At this sample size a 0.36 percentage-point deviation is not noise.
Notice the count was taken on ad_impression, which is a post-treatment population: a device only appears there once it has been served. The assignment population is the set of devices that entered an auction the line item was eligible for, and that lives in bid_request_log.
Re-run the same test on distinct device_id in bid_request_log. A 50/50 split there with a skew in ad_impression localises the problem to differential delivery rather than broken randomisation.
Enumerate the mechanisms that make one arm's devices less likely to be logged: one creative losing more auctions on price or format (win_flag), asymmetric no_bid_reason rates such as brand_safety or frequency_capped, and bid requests with NULL device_id falling into a non-uniform fallback bucket.
Do not read the treatment effect until the mechanism is identified. An imbalance this size invalidates the comparison regardless of how well covariates balance, because the filter that produced it is plausibly correlated with the outcome.

Worked solution (15 min)

Expected per arm = (1,204,338 + 1,186,902) / 2 = 1,195,620, so each arm deviates by 8,718.
chi-square = 2 * 8718^2 / 1,195,620 = 2 * 63.57 = 127.1 on 1 degree of freedom.
Compare with the 3.84 critical value: p is on the order of 2e-29, so a fair split is rejected decisively.
Repeat the count on bid_request_log distinct device_id per arm, and again restricted to response_status = 'bid', to find the funnel step where the split first diverges.

Expected result: chi-square = 127.1 on 1 df, p about 2e-29: a genuine sample ratio mismatch, not chance. The observed 50.36/49.64 share sits roughly 11.3 standard errors from 50/50, since the standard error of the share at n = 2,391,240 is 0.032 percentage points.

Verify it yourself

The z-statistic squared must equal the chi-square: (0.3647 pp / 0.0323 pp)^2 = 127, matching the goodness-of-fit value.
A 50/50 split on bid_request_log alongside a skew on ad_impression pins the cause to delivery rather than assignment.
Running the same chi-square on an A/A slice of the identical hash and salt should give p above 0.05; if it does not, the hash or the salt is the fault.
The 17,436-device gap should reconcile against a concrete mechanism count, for example a comparable difference in rows carrying no_bid_reason = 'brand_safety' between the arms.

Follow-up

The split on bid_request_log is also 50.4/49.6. What do you check next?
What share of bid requests carry a NULL device_id on each supply source, and how would that alone manufacture a sample ratio mismatch?
If the mechanism turns out to be one creative timing out more often, is any part of the test salvageable?

Attributed CPA improved everywhere while holdout lift stayed flat

Hard · attribution · model version · restatement

Attributed CPA improved between 40% and 55% on every line item on the same day, across all objectives and bid_strategy values. The always-on geo holdout shows incremental conversions per $1,000 unchanged over the same period. attribution_credit is keyed on (conversion_id, touchpoint_id, attribution_model, model_version) with credit_fraction, click_window_days and view_window_days; the dashboard filters attribution_model='last_click' but nothing else. Explain the improvement, quantify the reported error, and write the restatement you would send.

Approach

Take the uniformity as the strongest evidence available. Optimisation does not improve every line item by a similar factor on the same calendar day regardless of objective, bid strategy or vertical; a measurement or definition change does exactly that.
Test the model's own invariant: SUM(credit_fraction) grouped by conversion_id within one attribution_model should be 1.0. Where it is near 2.0, the query is summing two model_versions because model_version is in the key and the dashboard does not filter it.
Separate the double-count from the definition change underneath it. Compare click_window_days and view_window_days between the two model_versions. A view window widened from 1 to 7 days credits more conversions to impressions and improves reported CPA on its own, with nothing real having changed.
Reconcile against the holdout, which is the only estimate here that cannot be moved by editing the attribution rules, and state plainly that the flat holdout is the correct read of reality.
Produce the restatement: which metric, which date range, that the magnitude splits into a double-count component recoverable by pinning model_version and a definition component that makes pre- and post-change CPA non-comparable at all, and whether any budget was already moved on the inflated numbers.

Follow-up

How do you publish CPA across a model_version cutover so a reader can compare periods honestly?
Who benefits from a wider view window, and what would you require before agreeing to one?
The advertiser saw the improved CPA and asked for more budget. What do you say, given the holdout?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Your part, not the team's

Most data work is done by groups, so an interviewer has to work out which piece was yours. An answer that runs on 'we' for several minutes gets interrupted with a question about what you personally did, and by then the answer sounds defensive even when it is true. Mark your own contribution as you go, and name the parts that belonged to someone else instead of leaving them ambiguous. Keep a few specifics back as well, like the name of the metric or who actually objected, so a probe can be answered with something you had not already said.

Describe a situation where you disagreed with a product manager or engineer on a modeling approach. How did you resolve it?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Defending a null incrementality result against a headline ROAS

Hard · influence · incrementality · attribution · conflict

Your geo holdout shows a retargeting line item's incremental conversions per $1,000 of spend are indistinguishable from zero. The same line item reports last-click ROAS of 8.3 and is the headline number in a renewal deck presented in six days. The account team believes your test is wrong; the advertiser has never questioned the 8.3. Deliverable: the internal case you make, the specific evidence you bring, the limits you concede, and the smallest reversible experiment you propose if your reading is rejected.

Approach

The interviewer is probing whether you can hold an unpopular finding without turning it into a credibility fight, so open by granting that both numbers are computed correctly: last-click measures which touchpoint was nearest in time, the holdout measures what changed because of the spend. Framing attribution as 'wrong' converts a methodological point into a turf argument you will lose on relationship grounds.
State the null as a bound rather than as zero. Report the minimum detectable effect the 20-market design could resolve and say 'we can rule out incremental ROAS above this value'. A null without an MDE is not a finding and is trivially dismissed as an underpowered test.
Bring converging evidence from tables you already have, and say what each piece can and cannot show. Credit sensitivity: recompute the same conversion cohort under a second attribution_model, first-touch or even credit across in-window touchpoints, holding cohort and windows fixed, and report what share of the line item's credited volume survives; that bounds how much of the 8.3 is a property of the crediting rule rather than of the spend, and it is a sensitivity result, not a lift estimate. Harvesting signature: the share of retargeting-credited conversions whose user already had an earlier impression, click or site session inside click_window_days, together with the distribution of conversion_ts minus click_ts, where a heavy mass inside a few minutes is consistent with an ad served into a session that was already converting.
Quantify both error directions before recommending anything: the spend at risk if you are right and ignored, and the renewal revenue at risk if you are wrong and act. This is what separates a defensible position from an ideological one.
Propose the reversible version rather than the shutdown: a 20% budget reduction in half the matched markets for six weeks, powered against the MDE you just computed, with the analysis window extended past the flight by the click window plus ingestion lag.

Follow-up

The account lead says telling the advertiser their favourite channel does nothing will cost us the renewal. How do you respond without either caving or escalating?
What specific evidence would change your mind about this line item?
How do you handle it if the budget-down test also comes back underpowered?

Scoping a one-line request that attribution is wrong

Easy · scoping · stakeholder-management · attribution

A sales lead forwards a one-line request: 'attribution is wrong for this advertiser, fix it.' You have read access to conversion_event, attribution_credit and ad_click for the account, and thirty minutes with the account manager. You may not contact the advertiser this week. Deliverable: a one-page scope naming the single question you will answer, the questions you are explicitly not answering, the data you need, and the decision the answer changes. Bring the three clarifying questions you would ask the account manager first.

Approach

The interviewer is probing whether you convert a complaint into a comparison before you start work, so begin by writing the complaint as 'number A versus number B' and leave B blank until the account manager names it: attributed conversions against the advertiser's own order table, attributed CPA against last month, or our report against a second vendor's. Each has a different investigation.
Run the two cheap arithmetic checks that resolve a large share of these before any modelling: confirm SUM(credit_fraction) per conversion_id equals 1.0 within one attribution_model and model_version, and measure the duplication rate by counting dedup_key values that appear with more than one distinct source in conversion_event.
Measure identity coverage for the account: share of conversion_event rows carrying a non-null click_tracking_id and share carrying device_id. A low match rate means the channel is under-credited, which is the opposite complaint and needs the opposite fix.
Decide the scope from what the answer will change. If the decision is budget reallocation, the honest scope is an incrementality read, not an attribution repair. If the decision is an invoice dispute, the scope is a reconciliation against the advertiser's order count.
Write the page with an explicit out-of-scope list: model choice, window changes and anything requiring advertiser data you cannot get this week, each with the condition that would bring it back in scope.

Follow-up

The account manager says the advertiser just wants the numbers to match. What do you tell them is actually achievable, and why is exact agreement not one of the options?
How does your scope change if the discrepancy is 4% rather than 40%?