Adform Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at Adform sits at the intersection of advertising technology, large-scale data processing, and predictive modeling. You will work within a high-velocity environment where milliseconds matter, directly impacting how digital advertising is bought, sold, and optimized across the globe. Your work is fundamental to the Adform platform, as you will be responsible for turning massive datasets into actionable insights that drive product features and business strategy.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

• Diagnose pacing: underdelivery, overdelivery, budget exhaustion
• Separate invalid traffic from genuine performance shifts
• Reconstruct the funnel from bid request to conversion

Overview

This role requires a unique blend of technical rigor and product intuition. You will not just be building models in isolation; you will be collaborating with engineering and product teams to design experiments, diagnose performance fluctuations, and refine the metrics that define success for Adform clients. Whether you are optimizing bidding algorithms or analyzing complex user behavior patterns, your contributions will have a tangible effect on the company’s competitive edge in the programmatic advertising space.

The Adform interview process

Adform candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

• Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
• Whether you name what you have not done instead of stretching to cover every line of the posting
• Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

• Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
• Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
• Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Behavioral Discussion

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

• Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
• The counterfactual: what would have been done without your analysis, and why that default was worse
• How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

• Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
• Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
• Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Technical Assessment

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

• Whether the query you write matches the plan you just described
• What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
• Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

• Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
• Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
• Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Where candidates lose points

Treating a bid as if it does not affect the price paid, under a first-price auction.

In a second-price auction the winner pays the runner-up's bid, so bidding true value is close to optimal and bid and price are usefully separable. Under first-price clearing, which is now the common case in open exchanges, the winner pays exactly what it bid, so raising the bid raises both the win probability and the cost of every win that would have been won anyway. A bid-increase experiment can therefore show more impressions, more conversions and worse profit simultaneously, and reading only the volume metrics will call it a success. The related second-order problem is that your own bid increase raises the clearing prices your control cells face, which contaminates the comparison unless the cells are separated by market.

Reallocating budget using last-click attribution.

Last-click assigns full credit to whichever touchpoint sits closest to the conversion in time, which structurally favours channels that harvest existing demand — retargeting an already-interested user, or catching a branded search — over channels that create demand in the first place. Optimising against it therefore moves money toward tactics that would have converted many of those users anyway, and the reported cost per action improves at the exact moment true incremental performance gets worse. The signature of this failure is a portfolio where every channel's attributed conversions sum to well above the advertiser's total conversion count. The counter is to treat attributed numbers as a budget-splitting convention and to source the actual reallocation decision from holdout-based incrementality.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Accepting a metric definition without asking about the denominator

Pin down the denominator, the eligibility filter and the time window before computing anything: conversion rate per session, per user, per eligible user and per new user are four different numbers with different behaviour. Restate the definition in one sentence and get agreement before you analyse.

Questions & practice

Simulate bid shading profit under first-price auction clearing

Hard · simulation · auctions · bid-shading · monte-carlo

auctions holds 200,000 backtest rows for one line_item_id: predicted_cvr (per impression), floor_price_cpm_usd (nullable when the floor is undisclosed), and best_competing_cpm_usd, the highest competing bid, observable only in this sample. Clearing is first price. The advertiser values a conversion at $40. For shading factors s from 0.50 to 1.00 in steps of 0.05, the submitted bid is s * 1000 * 40 * predicted_cvr and the line item wins when that bid clears both the floor and the best competing bid. Return win rate, spend, conversions and profit per thousand auctions, the profit-maximising s, and an uncertainty estimate for it.

Approach

1. Separate what is deterministic from what is random before writing any sampling code. Given the bid, winning is deterministic and the price paid under first-price clearing is exactly the submitted bid, so the only stochastic element is whether a won impression converts. Use the closed form sum of predicted_cvr over wins for the expected conversion count and reserve Monte Carlo for the profit distribution, not the mean.
2. Vectorise across s: build the bid vector once, then evaluate the 11 shading factors as numpy operations over the full array. A per-row Python loop over 200,000 rows by 11 factors is about two million iterations and turns a 3-second job into minutes.
3. Make the win condition explicit and handle nulls deliberately: treat a null floor as no floor, require bid > best_competing_cpm_usd, and require bid >= floor_price_cpm_usd. Ties at exactly the floor behave differently from ties against a competitor, and the convention should be stated rather than inherited from a comparison operator chosen by accident.
4. Compute profit per thousand auctions as (40 * conversions - sum of bid/1000 over wins) / n * 1000. Spend must use your own submitted bid, not the competing bid: under first-price clearing every inframarginal win, one you would have taken at a lower bid, now costs more, and that term is what creates an interior optimum.
5. Get uncertainty by bootstrapping auction rows, recomputing the whole curve per replicate, and reporting the distribution of the argmax s rather than only a confidence band around the profit level. The decision uses the argmax, and it is far less stable than the profit curve it comes from.
6. State the backtest's limit honestly: best_competing_cpm_usd was observed while you bid what you actually bid. If your change alters competitors' behaviour or triggers exchange-side floor adjustments, the counterfactual does not hold and the curve overstates the achievable gain.

Follow-up

• Raising this line item's bid also raises the clearing price faced by another line item in the same account bidding on overlapping supply. How does that change the optimum, and how would you detect it in the logs?
• predicted_cvr over-predicts by 15% at the head of the distribution. Which way does the optimal s move, and roughly how far?
• How would you run this as a live test instead of a backtest, and what is the randomisation unit given that both arms draw on one budget?

Sessionise a device event stream with a 30-minute gap

Medium · sessionisation · event-streams · algorithms · pandas

events holds about 5 million unsorted rows: device_id (nullable, roughly 18% null), event_ts, event_type in {impression, click, landing_arrival, conversion}, line_item_id, value_usd (nullable). Define a session as a maximal run of events for one device_id in which consecutive events are at most 30 minutes apart. Assign a session_id to every sessionisable row and return a per-session summary with device_id, start_ts, end_ts, n_impressions, n_clicks, converted and total value_usd. Do not use a library sessionisation helper, and do not scan the frame more than once after sorting.

Approach

1. Separate null device_id rows first and report their share. Null identity is a supply-source property, not random missingness, so grouping them under one null key would fabricate a single session spanning millions of unrelated events; excluding them means the summary describes identified traffic only, and that has to be said out loud.
2. Sort once by (device_id, event_ts) with a deterministic tiebreak such as an event_type rank then a row id. Without a tiebreak, an impression and its click sharing a timestamp order differently between runs and the summary is not reproducible.
3. Compute the boundary mask as (device_id != device_id.shift()) | (event_ts - event_ts.shift() > 30min) and take its cumsum as session_id. This is a grouped run-length in two vectorised operations: O(n log n) for the sort and O(n) afterwards.
4. Create indicator columns (is_impression, is_click, is_conversion) before the groupby, then summarise with one named aggregation pass. Building each count from its own filtered groupby re-scans the frame per metric and violates the single-pass constraint.
5. Assert the structural invariant rather than eyeballing it: within a device, sessions ordered by start_ts must be disjoint and increasing, every within-session consecutive gap at most 30 minutes, every between-session gap strictly greater.

Follow-up

• A conversion arrives three hours after the click and lands in its own session. Is that the right answer, and how does a session boundary differ from an attribution window?
• One device shows negative gaps from a clock four hours off. What does your boundary mask do with it, and what would you prefer it did?
• Make the 30 minutes a parameter. How would you choose it from the data rather than from convention?

Check predicted conversion-rate calibration with a reliability table

Medium · calibration · model-evaluation · statistics · numpy

scored has 4 million rows at bid_request_log grain for one campaign: predicted_cvr, win_flag, and converted, which is observed only where win_flag is true and a conversion landed within 7 days. The win rate is about 2% and the conversion rate among won impressions about 0.4%. Without using a library calibration helper, build a 20-bin equal-frequency reliability table with bin count, mean predicted_cvr, observed rate and an interval around it, then compute count-weighted expected calibration error. Say whether the model is miscalibrated, in which direction, and over which population your answer holds.

Approach

1. Restrict to win_flag == True and state the precondition plainly: converted is only defined where the ad served, so the table measures calibration on the won slice. That is the population the advertiser pays for, but it is selected by the bid and the auction, so the result cannot be extended to no-bid or lost opportunities where the prediction is never tested.
2. Bin with equal frequency (pd.qcut with duplicates='drop'), not equal width. Prediction distributions in this setting are heavily right-skewed with point masses, so equal-width bins put most of the mass in one bin and leave the tail bins with too few conversions to say anything. Report how many bins survived if edges collapsed.
3. Per bin compute n, mean predicted_cvr, observed rate, and a Wilson score interval. At a 0.4% base rate the normal approximation produces negative lower bounds and understates uncertainty in the low bins, so the interval choice is not cosmetic here.
4. Compute ECE as the count-weighted mean of |observed - predicted| across bins, and report it next to the base rate: 0.002 of absolute error is small against a 5% base rate and enormous against 0.4%.
5. Read the direction from the sign pattern rather than the scalar. Observed below predicted across the top bins is head over-prediction, typically from training on a differently-selected population or from scoring against a conversion window that has not matured; a flat observed rate across rising predictions means the score has little ranking power in this range, which is a different problem from miscalibration.

Worked solution (25 min)

1. Subset to won rows, record n_won and the observed base rate, and confirm converted has no nulls within that subset.
2. bins = pd.qcut(predicted_cvr, 20, labels=False, duplicates='drop'); report the realised bin count.
3. groupby(bins).agg(n, pred_mean, obs_rate), then compute Wilson bounds elementwise from n and obs_rate at z = 1.96.
4. ece = ((n / n.sum()) * (obs_rate - pred_mean).abs()).sum(); print it alongside the base rate and the ratio of the two.

Expected result: A table of at most 20 rows with bin, n, pred_mean, obs_rate, wilson_lo, wilson_hi, each bin holding close to n_won/20 rows by construction, plus a scalar ECE reported against the base rate.

Verify it yourself

• sum(n) equals the number of won rows, and the count-weighted mean of pred_mean equals the overall mean predicted_cvr to floating point.
• The count-weighted mean of obs_rate equals the overall observed conversion rate among won rows; a mismatch means rows were lost in binning or the weights are wrong.
• Every Wilson bound lies inside [0, 1], including the lowest bin where a normal approximation would return a negative lower bound.
• Shuffling the row order leaves the table unchanged, confirming the binning depends only on predicted_cvr and not on input order through tie handling.

Follow-up

• The pooled curve looks calibrated but ctv supply is badly off. How do you test for segment-level miscalibration without fishing across dozens of cuts?
• The model retrains daily while conversions take up to 7 days to land. What does that do to the head of the reliability curve, and in which direction?
• How would you turn this diagnostic into a bid correction, and what would you verify before shipping it?

Can you demonstrate a time you used complex joins or subqueries to solve a data quality issue?

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. Say which table is the grain you start from, and join outward from it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

Write a SQL query using window functions to calculate a rolling average of ad performance over a seven-day period.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. State the window function and its partition and ordering out loud before writing it.

Follow-up

• How does the query change if the join becomes one-to-many?
• What breaks if events arrive late or out of order?

Advertiser spend retention at a three-month horizon by cohort

Medium · cohort retention · self-join · scd join

ad_impression carries line_item_id, served_ts, billed_price_micros_usd and is_billable but no advertiser_id. line_item carries advertiser_id and is SCD type 2: one row per version with valid_from_ts, valid_to_ts and is_current, and exactly one current row per line_item_id. Build monthly cohorts. For each cohort month M, return the distinct advertisers with at least one billable impression in M, the count still billing in month M+3, the count-based retention rate, and a spend-weighted rate using M+3 spend from retained advertisers over cohort-month spend. State which line_item row you join to and why.

Approach

1. Resolve advertiser_id through a collapsed mapping, not through the full dimension. SELECT DISTINCT line_item_id, advertiser_id FROM line_item returns one row per line item because advertiser_id does not change across versions. Joining ad_impression to the raw line_item table multiplies every impression by the number of versions that line item has.
2. Note the precondition that makes that shortcut legal: it works only because the attribute being read is version-invariant. For daily_budget_usd or bid_value_usd, which do change, you must join on the validity interval instead.
3. Build an advertiser-by-month activity table in one pass: GROUP BY advertiser_id and date_trunc('month', served_ts AT TIME ZONE 'UTC'), filtered to is_billable, carrying SUM of spend so both retention variants come from the same aggregate.
4. Self-join that table to itself with a LEFT JOIN on advertiser_id AND retained.month = cohort.month + interval '3 months'. The LEFT JOIN keeps cohorts with zero retention in the output instead of making them disappear, which is what an INNER JOIN would do and is indistinguishable from the cohort not existing.
5. Say which spend-weighted definition you are reporting, because at least two circulate: M+3 spend from retained advertisers over cohort-month spend from all cohort advertisers (a spend-expansion measure) versus cohort-month spend from retained advertisers over cohort-month spend from all (a dollar-coverage measure).
6. Truncate the series at the last cohort month whose M+3 is fully elapsed, otherwise the three most recent cohorts read as zero retention and someone will escalate it.

Worked solution (30 min)

1. Build li AS (SELECT DISTINCT line_item_id, advertiser_id FROM line_item) and assert it returns exactly one advertiser per line item before relying on it.
2. Build monthly AS (SELECT l.advertiser_id, date_trunc('month', i.served_ts AT TIME ZONE 'UTC') AS m, SUM(i.billed_price_micros_usd) / 1e6 AS spend FROM ad_impression i JOIN li l USING (line_item_id) WHERE i.is_billable GROUP BY 1, 2).
3. Self-join: FROM monthly c LEFT JOIN monthly r ON r.advertiser_id = c.advertiser_id AND r.m = c.m + interval '3 months'.
4. GROUP BY c.m and emit COUNT(DISTINCT c.advertiser_id), COUNT(DISTINCT r.advertiser_id), SUM(c.spend) and SUM(r.spend).
5. Filter the output to cohort months where m + interval '3 months' has fully elapsed.
6. Plot both retention series together; if they overlay exactly, the spend weighting is not actually being applied.

Expected result: One row per cohort month. For example cohort 2026-01 with 1,480 advertisers, 902 still billing in 2026-04, 60.9% count retention and 78.4% spend-weighted retention, with cohorts after 2026-06 excluded as immature.

Verify it yourself

• retained_advertisers never exceeds cohort_advertisers in any row.
• Total spend summed across all cohort months equals an ungrouped total over billable impressions for the same period; a higher figure means the line_item join fanned out.
• Spend-weighted retention sits above count retention whenever large advertisers retain better. The two being equal to three decimals almost always means the weighting collapsed.
• Dropping the is_billable filter changes cohort membership, confirming that make-good-only advertisers are not being counted as spenders.

Follow-up

• Count retention is flat but spend-weighted retention dropped 8 points. What is happening to the book of business?
• An advertiser is dormant in M+3 and spends again in M+4. Is that churn? What does each answer do to the series?
• How would you restate this on rolling 90-day windows instead of calendar months, and which artefact does that remove?

If you notice a sudden drop in our primary engagement metric, how would you go about diagnosing the root cause?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

How do you balance long-term user experience with short-term revenue goals in an ad-tech environment?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a metric to measure the success of a new bidding feature?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

What are the most important trade-offs when defining a "successful" impression?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Fix the population and the time window before naming any metric.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

What are the most common experimentation pitfalls you have encountered in your career?

Medium · a/b testing and experimentation

Approach

1. Say whether units interfere with each other, and switch design if they do.
2. Decide the analysis before seeing data, including how long it runs and when you look.
3. State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

Choose a randomisation unit for a frequency cap change

Medium · interference · randomisation-unit · switchback · geo-holdout

An advertiser wants to know whether raising frequency_cap_per_day on a line item from 3 to 8 increases conversions. The proposed design splits device_id 50/50 inside the existing line item, with both arms drawing on the same daily_budget_usd and the same even pacing_mode. Say what this design actually estimates, in which direction it is wrong, and propose a design that estimates the quantity the advertiser asked about. Evaluate a switchback explicitly before discarding it.

Approach

1. Name the violation as interference, not imbalance. With a fixed daily budget, every extra impression a treatment device absorbs is one the control arm does not get, so the arms are not independent. Covariate balance will look clean and tells you nothing about this.
2. Sign the bias: control delivery is pushed below its status-quo level, so the treatment-minus-control gap overstates the true effect of the cap change. Size it by comparing control-arm impressions per device against the same line item's pre-period rate.
3. Evaluate the switchback on its merits. It removes budget contention because only one policy is live at a time, but the outcome is a conversion with a 7 or 28 day click window, so blocks would need to run several window-lengths to avoid carryover, leaving a handful of effective observations. A switchback is right here only for a fast-settling outcome such as impressions per device or auction win rate.
4. Move the randomisation unit up to something that contains a budget: cluster-randomise geo_dma with a per-market daily budget set from pre-period market spend, or split the line item into two separately budgeted line items with disjoint targeting and half the budget each.
5. Price the replacement honestly. A market-level design has tens of effective units, so compute and state the MDE before running it, and use pre-period market series for stratified assignment rather than simple randomisation.
6. Define the analysis window as the flight plus the full click window plus observed ingestion lag, and bucket spend on touchpoint_ts so spend and outcome describe the same cohort.

Worked solution (30 min)

1. Pull control-arm impressions per device per day and compare against the same line item's pre-period rate; a drop is direct evidence of budget starvation.
2. Compute hourly spend against the time-weighted daily budget for both arms; under even pacing, a binding budget shows as earlier exhaustion during the test than in the pre-period.
3. Design the replacement: rank geo_dma by pre-period conversions, form matched pairs, randomise within pair, and set each market's daily budget to its pre-period share of spend.
4. Power the market design on pre-period market-level conversion variance and report the MDE at 80% power before committing to it.
5. Pre-register the estimator as a paired difference on market-level conversions per thousand billable impressions, with the window set to flight plus click window plus two days of ingestion lag.

Expected result: The proposed design estimates the effect of the cap change under a contended budget, which is biased upward because the control arm is starved rather than left at status quo. The defensible replacement is matched-pair geo_dma cluster randomisation with per-market budgets; the switchback is ruled out on carryover grounds because conversion outcomes do not settle inside a feasible block.

Verify it yourself

• Under a valid design, control-arm impressions per device should match the pre-period rate; a fall means the budget is still contended.
• Under the market design, each market's hourly spend should track its own budget rather than a pooled one.
• Replaying the matched-pair assignment as an A/A on historical market data should produce an estimate centred on zero with the predicted interval.

Follow-up

• The advertiser has budget in only one market. What is the best design left, and what does it cost in precision?
• How would you detect from the logs alone that the control arm was being starved during the test?
• Which outcome would you run a switchback on here, and how long would the blocks have to be?

Post-click conversion rate dropped on mobile app supply only

Hard · identity · funnel decomposition · instrumentation

Post-click conversion rate for one advertiser fell from 3.1% to 0.9% on device_type='mobile_app', with desktop and mobile_web flat. The metric is deduplicated conversion_event rows joined to ad_click on click_tracking_id within click_window_days, over valid billable ad_click rows, cohorted on click_ts. ad_click also has impression_id, device_id, landing_arrival_ts, ivt_status; conversion_event has source, device_id, click_tracking_id, dedup_key. The advertiser shipped a mobile SDK update that week. Decide whether conversions stopped happening or stopped being joinable, and recommend an action.

Approach

1. Cohort correctly before anything else: restrict to click_ts dates older than click_window_days plus ingestion lag, so an immature window is not mistaken for a segment-specific drop.
2. Split the single ratio into two consecutive steps — clicks to non-null landing_arrival_ts, and arrivals to joined conversions. A drop at the first step means users are not arriving, which is a real problem on the ad or landing side; a drop only at the second means they arrive and the join fails, which is instrumentation.
3. If the break is at the join step, measure click_tracking_id null rate in conversion_event split by source and advertiser, by day. A step change on the mobile_sdk source against a flat browser_pixel source localises it to the advertiser's SDK release, and the break will be a single timestamp rather than a ramp.
4. Rule out the denominator explanation explicitly, because it moves the metric the other way: rising sivt reclassification on mobile_app clicks removes rows from the denominator and would raise the rate, so an unchanged ivt_status distribution is a necessary check, not a formality.
5. Establish that the conversions still exist by counting total conversion_event rows for the advertiser across the break, then recover the joinable population on device_id within the window and report both the recovered rate and the residual recovery gap from null device_id.

Follow-up

• What fraction of mobile_app clicks carry a null device_id, and what does that cap your recovery at?
• Would you restate the advertiser's reported CPA for the affected week, or footnote it, and what changes that decision?
• How would you monitor the joinable share as a first-class metric so this is caught the day it starts?

Suggested practice checklist

Instead of guessing where the week should go, day one measures it under a fixed rubric and allocates the remaining hours in proportion to the gaps. The method is deliberately rigid: the allocation is written down before any studying starts and is not renegotiated when a topic turns out to be unpleasant.

Day 1 — Diagnostic, scored before you study anything

• Sit a 100-minute timed diagnostic in four blocks: 30 minutes of SQL across three prompts, 25 minutes of short-answer statistics, 25 minutes on one modelling or case prompt, and 20 minutes delivering one behavioural story aloud.
• Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct, and 0 is stuck, grading the output rather than how the attempt felt.
• Allocate the hours for days two to five roughly in proportion to 3 minus the score in each block, write the allocation down, and commit to not revising it midweek.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

• Break the weakest area into five named sub-skills (for query work: grain control, window frames, date arithmetic, set logic with NULLs, and reading a query plan) and rate each one, so the rest of the week targets a sub-skill instead of a subject.
• Solve three problems chosen to sit just above where the rating drops off, and for each write the first move you failed to make.
• Re-solve one of them from memory four hours later, on paper, with nothing open.

Deliverable: A five-item sub-skill map with the two blocking sub-skills circled.

Day 3 — Largest gap: drill the blocking sub-skill

• Do eight short repetitions of the same shape rather than eight different problems, so what you practise is the pattern and not the puzzle.
• Write the rule you now hold in one sentence, then test it against a case built to break it: a ranking function over a column with ties, or a two-sample test on observations that are obviously dependent.
• Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on your strongest area

• Run the same sub-skill map and boundary protocol on the second-largest gap, compressed into half the day.
• Spend 25 timed minutes on your strongest area to stop it decaying, choosing the hardest problem you can still finish rather than an easy warm-up.
• Compare how the two areas fail: whether you lose time on recall, on setup, or on arithmetic, because the fix differs for each.

Deliverable: A second sub-skill map plus a one-line diagnosis of how each area fails you.

Day 5 — The gap that is not a skill

• Record yourself answering one technical and one behavioural prompt, then count two things in the playback: how many seconds before your first clarifying question, and how many sentences you started without knowing where they ended.
• Rewrite your three most-used stock phrases into shorter versions, and practise saying "I do not know, here is how I would find out" without softening it into a guess.
• Deliver one answer again with a hard 90-second limit to force structure before detail.

Deliverable: Two recordings with a counted improvement in time-to-first-question.

Day 6 — Retest under day-one conditions

• Sit the same 100-minute diagnostic structure with new prompts of comparable difficulty and score it on the identical rubric.
• Compare block by block, and for any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
• Write which single block you would still lose the offer on.

Deliverable: A second scored rubric placed next to the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

• Run a 60-minute mock covering the two blocks that moved least, with an interviewer instructed to interrupt and change direction.
• Write your recovery script for the moment you go blank: restate the question, state your assumption, name the first thing you would check.
• Reduce the week to the rule statements you wrote, each with its preconditions attached, then say every one of them out loud without reading it and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive being interrupted.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

If your test results are conflicting, what steps do you take to verify the validity of the data?

Medium · behavioural and stakeholder questions

Approach

1. Name the disagreement or constraint, and how you resolved it with evidence.
2. Close with what you would do differently, concretely.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• What did you decide not to do, and why?
• How did you know the outcome was caused by your change?

Ranking three teams' requests against one sprint of capacity

Medium · prioritisation · stakeholder-management · delivery

You have three requests and capacity for roughly one and a half. The supply team wants eligible-RPM impact sizing for a floor-price change shipping in two weeks. An advertiser team wants a pacing diagnosis for a $400k line item delivering at 60% of budget. The invalid-traffic team wants help validating a new sophisticated-invalid-traffic classifier before it restates last quarter. Deliverable: your ranking with the reason for each, exactly what you say to the two teams you deprioritise, and the trigger that would reorder the list.

Approach

1. The interviewer is probing whether you rank on decision structure or on social pressure, so attach two attributes to each request before ranking: the date the decision is actually made, and how expensive it is to reverse afterwards. Seniority of the requester is not one of the inputs.
2. Look for the request that can be collapsed rather than sequenced. The pacing diagnosis is usually a single grouped query over bid_request_log by no_bid_reason and response_status per hour for that line_item_id; time-boxed to two hours it stops competing for the sprint at all.
3. Size the other two in revenue at risk per week so the ranking is arguable rather than asserted: the floor change's expected eligible-RPM delta against the spend it touches, and the invoiced spend inside the period the classifier would restate.
4. Rank on irreversibility where deadlines are close together. A classifier that restates already-invoiced periods is the hardest of the three to unwind once shipped, which can outrank a nearer deadline.
5. Deprioritise with a date and a fallback rather than an apology: name what you will not do, when you could, the smallest thing you can give them now, and who else could take it.

Follow-up

• The underdelivering advertiser escalates to your director. What actually changes in your plan, and what do you refuse to change?
• How do you stop the pacing question arriving fresh every sprint?

Defending a null incrementality result against a headline ROAS

Hard · influence · incrementality · attribution · conflict

Your geo holdout shows a retargeting line item's incremental conversions per $1,000 of spend are indistinguishable from zero. The same line item reports last-click ROAS of 8.3 and is the headline number in a renewal deck presented in six days. The account team believes your test is wrong; the advertiser has never questioned the 8.3. Deliverable: the internal case you make, the specific evidence you bring, the limits you concede, and the smallest reversible experiment you propose if your reading is rejected.

Approach

1. The interviewer is probing whether you can hold an unpopular finding without turning it into a credibility fight, so open by granting that both numbers are computed correctly: last-click measures which touchpoint was nearest in time, the holdout measures what changed because of the spend. Framing attribution as 'wrong' converts a methodological point into a turf argument you will lose on relationship grounds.
2. State the null as a bound rather than as zero. Report the minimum detectable effect the 20-market design could resolve and say 'we can rule out incremental ROAS above this value'. A null without an MDE is not a finding and is trivially dismissed as an underpowered test.
3. Bring converging evidence from tables you already have, and say what each piece can and cannot show. Credit sensitivity: recompute the same conversion cohort under a second attribution_model, first-touch or even credit across in-window touchpoints, holding cohort and windows fixed, and report what share of the line item's credited volume survives; that bounds how much of the 8.3 is a property of the crediting rule rather than of the spend, and it is a sensitivity result, not a lift estimate. Harvesting signature: the share of retargeting-credited conversions whose user already had an earlier impression, click or site session inside click_window_days, together with the distribution of conversion_ts minus click_ts, where a heavy mass inside a few minutes is consistent with an ad served into a session that was already converting.
4. Quantify both error directions before recommending anything: the spend at risk if you are right and ignored, and the renewal revenue at risk if you are wrong and act. This is what separates a defensible position from an ideological one.
5. Propose the reversible version rather than the shutdown: a 20% budget reduction in half the matched markets for six weeks, powered against the MDE you just computed, with the analysis window extended past the flight by the click window plus ingestion lag.

Follow-up

• The account lead says telling the advertiser their favourite channel does nothing will cost us the renewal. How do you respond without either caving or escalating?
• What specific evidence would change your mind about this line item?
• How do you handle it if the budget-down test also comes back underpowered?