Interview guides · Impact

Impact Data Scientist Interview Questions & Guide 2026

SQL practice for Impact data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

As a Data Scientist at Impact, you operate at the core of the world's leading commerce partnership marketing platform. Your work transforms how global brands discover, manage, and scale partnerships across the entire customer journey, spanning affiliates, influencers, content publishers, and brand advocates. By building sophisticated algorithms, predictive models, and experimentation frameworks, you directly influence how millions of partnerships deliver measurable business results for thousands of enterprise brands.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Defend attribution windows and model choice
Diagnose pacing: underdelivery, overdelivery, budget exhaustion
Reconstruct the funnel from bid request to conversion

Overview

This role combines high-craft technical execution with deep marketplace economics. Whether you are embedded within the Programmatic Experience Group optimizing yield, pricing, and inventory allocation, or designing next-generation recommendation systems that connect advertisers with media publishers and creators, your contributions drive the company's core engine. You will own problems end-to-end, architecting data pipelines, engineering features, and deploying real-time inference systems that operate at massive scale while balancing advertiser performance with publisher monetization.

Expect a fast-paced, high-ownership environment where autonomy is expected and rigorous scientific thinking is valued. You will collaborate closely with product management, platform engineering, and business stakeholders, positioning you as a strategic partner who shapes both technical architecture and product roadmap. Success in this role requires translating complex multi-sided marketplace dynamics into clean mathematical formulations and robust, scalable software systems.

The Impact interview process

Impact candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Conversation

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Technical Interview

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Where candidates lose points

Treating a bid as if it does not affect the price paid, under a first-price auction.

In a second-price auction the winner pays the runner-up's bid, so bidding true value is close to optimal and bid and price are usefully separable. Under first-price clearing, which is now the common case in open exchanges, the winner pays exactly what it bid, so raising the bid raises both the win probability and the cost of every win that would have been won anyway. A bid-increase experiment can therefore show more impressions, more conversions and worse profit simultaneously, and reading only the volume metrics will call it a success. The related second-order problem is that your own bid increase raises the clearing prices your control cells face, which contaminates the comparison unless the cells are separated by market.

Comparing conversion rates between users who were exposed to an ad and users who were not.

Exposure is the output of a targeting and bidding system that ranks users by predicted conversion propensity, so the exposed group has a higher baseline conversion rate before the ad renders. The measured difference therefore estimates the quality of the targeting model, not the effect of the advertising, and it is biased upward by an amount that grows as the model gets better. The valid control is the set of users the system would have shown the ad to but deliberately did not, captured through a ghost-bid or public-service-announcement holdout that records auction wins without serving the advertiser's creative. Anyone who proposes matching or covariate adjustment to fix this should be asked what unobserved feature the bidder used that the matching model does not have.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Questions & practice

What is the role of p-values and confidence intervals in decision-making, and how do you explain false positive rates to non-technical stakeholders?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

Which assumption here is most likely to be violated in practice?
What sample size would you need to detect an effect half this size?

What feature engineering techniques work best when building predictive models for dynamic pricing and inventory allocation in programmatic advertising?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Point-in-time join of impressions to versioned line items

Medium · scd2 · point-in-time-join · numpy · pandas

impressions has 20 million rows: impression_id, served_ts, line_item_id, billed_price_micros_usd, is_billable. line_item is SCD type 2 with 400,000 versions: line_item_id, bid_strategy, bid_value_usd, daily_budget_usd, pacing_mode, status, valid_from_ts, valid_to_ts (null on the current row), is_current. A version covers the half-open interval from valid_from_ts to coalesce(valid_to_ts, infinity). Attach to every impression the attributes of the version in force at its served_ts. You may not use merge_asof and you may not materialise the impression-by-version cross product. Return the joined frame and a count of impressions no version covers.

Approach

Validate the dimension before joining: per line_item_id, sort by valid_from_ts and assert the intervals are non-overlapping, and assert at most one is_current row per key. Overlapping versions make the as-of answer ambiguous and are the usual root cause behind duplicated-revenue reports, so finding them is part of the job.
Encode the composite as-of key into one sortable integer so a single np.searchsorted does the work: factorise line_item_id to codes shared across both frames, dense-rank all timestamps from both frames into a small integer, then form code * (n_ranks + 1) + rank. That keeps the composite inside int64 and makes lexicographic order on (key, time) equal to numeric order.
Sort the dimension by that composite, run np.searchsorted(dim_keys, imp_keys, side='right') - 1, and you have a candidate version index for every impression in one vectorised call rather than 400,000 per-group searches.
Reject candidates that do not actually cover the impression: the candidate must belong to the same line_item_id, and served_ts must be less than coalesce(valid_to_ts, far future). Report the uncovered count instead of dropping those rows; impressions before a key's first version normally indicate a backfill gap in the dimension, not bad events.
Materialise attributes with a positional .take on the matched index rather than a merge. One row in, one row out by construction, so fan-out is impossible regardless of what the dimension contains.

Worked solution (30 min)

Run the dimension validation and collect two lists: keys with overlapping intervals and keys with more than one is_current row.
codes, uniques = pd.factorize(pd.concat([...])) applied consistently to both frames; dense-rank the union of served_ts and valid_from_ts; build the int64 composites.
Sort the dimension by composite, np.searchsorted(side='right') - 1 for the impressions, and mask positions of -1 as uncovered.
Validate coverage vectorised: same code, and served_ts < coalesce(valid_to_ts, Timestamp.max); then .take the attribute columns at the matched positions and assemble the output.

Expected result: A frame with exactly len(impressions) rows carrying one attribute set each, an integer count of uncovered impressions, and the two dimension-integrity lists (overlapping keys, duplicate-current keys).

Verify it yourself

Output row count equals input row count exactly; any increase is fan-out, any decrease is a silently dropped uncovered row.
Asserted vectorised over the whole output: valid_from_ts <= served_ts < coalesce(valid_to_ts, Timestamp.max) on every covered row.
On a hand-built case with three versions and an impression exactly at a boundary timestamp, the impression at served_ts == valid_from_ts attaches to the newer version, matching the half-open interval.
On a 10,000-row sample, results agree row for row with a deliberately slow reference implementation that loops each impression over its key's versions. That brute-force check is what proves the composite-key encoding is sound.

Follow-up

Two versions overlap by one second because the writer used an inclusive end timestamp. What does your join return, and how would you make that fail loudly instead of quietly picking one?
The pacing report needs the budget in force at every hour boundary, not just at impression time. What changes in the construction?
Estimate the memory and time cost of the naive merge-on-key-then-filter at these row counts, and say where it breaks first.

Write a query to find the top three highest-earning advertisers within each category using rank or dense-rank window functions.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

How would you optimize a slow-running SQL query that joins multi-million-row attribution tables across disparate database shards?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Attributed cost per action without double counting spend or credit

Hard · fan-out · attribution · bucketing

attribution_credit holds one row per (conversion_id, touchpoint_id, attribution_model, model_version), with touchpoint_ts, line_item_id, credit_fraction, credited_value_usd and click_window_days, the model's click lookback in whole days, which is constant within one (attribution_model, model_version) pair; credit_fraction sums to 1.0 across a conversion's touchpoints within one model and one version. ad_impression has line_item_id, served_ts, billed_price_micros_usd and is_billable. For a UTC date range, return per line_item_id and day: spend, attributed conversions, attributed CPA, attributed ROAS, and a maturity flag derived from the model's own click window, for one named attribution_model and model_version. Both sides must land in the same daily buckets. State which timestamp each side is bucketed on and why.

Approach

Pin attribution_model AND model_version before aggregating anything. The table stores several models, and several versions of each, for the same conversion. An unfiltered SUM(credit_fraction) therefore reports conversions multiplied by the number of (model, version) pairs present, which is usually a clean integer multiple and so survives a sanity glance.
Bucket credit on touchpoint_ts, never on conversion_ts. Spend is incurred when the impression served, so aligning the outcome to the touchpoint puts numerator and denominator on the same cohort. Bucketing on conversion_ts shifts every conversion forward by a variable lag and produces day-over-day CPA swings that are pure bucketing artefacts and cannot be corrected downstream.
Aggregate the two sides in separate CTEs at their own grain, then FULL OUTER JOIN on (line_item_id, day). Joining ad_impression to attribution_credit directly matches each impression against every credit row that references it and multiplies spend by the path count.
Use a full outer join deliberately: a day with spend and no credit is a genuine CPA of infinity worth surfacing, and a day with credit and no spend means the touchpoint predates the spend window, which is a signal about window choice rather than a row to discard.
Attributed conversions is SUM(credit_fraction), a fractional quantity, not COUNT(*). Counting rows credits one conversion once per touchpoint and inflates the total by the mean path length, which makes CPA look far better than it is.
Flag recent days as immature. Spend is final within hours while credit keeps arriving for the model's click_window_days plus ingestion lag, so the most recent click_window_days plus roughly two days have CPA biased high and ROAS biased low. Read that window off the filtered rows rather than hard-coding a number: click_window_days is constant within a pinned (attribution_model, model_version), so MAX(click_window_days) over the pinned filter is a read of that constant. Verify the precondition first, because if MIN and MAX differ the column is not constant, the flag is being computed against a window that does not exist, and it has to come from the model configuration instead.

Worked solution (40 min)

Run SELECT attribution_model, model_version, MIN(click_window_days), MAX(click_window_days), COUNT(*) FROM attribution_credit GROUP BY 1, 2 first, so you know what is actually in the table before filtering to one pair, and so you have checked that the window really is constant inside the pair you are about to pin.
Build credit AS (SELECT line_item_id, (touchpoint_ts AT TIME ZONE 'UTC')::date AS d, SUM(credit_fraction) AS conv, SUM(credited_value_usd) AS value, MAX(click_window_days) AS win FROM attribution_credit WHERE attribution_model = :model AND model_version = :version AND touchpoint_ts >= :range_start AND touchpoint_ts < :range_end GROUP BY 1, 2).
Build spend AS (SELECT line_item_id, (served_ts AT TIME ZONE 'UTC')::date AS d, SUM(billed_price_micros_usd) / 1e6 AS spend FROM ad_impression WHERE is_billable AND served_ts >= :range_start AND served_ts < :range_end GROUP BY 1, 2).
FULL OUTER JOIN on (line_item_id, d), coalescing both key columns in the select list so rows present on only one side keep their identity.
Compute cpa = spend / NULLIF(conv, 0), roas = value / NULLIF(spend, 0), and is_mature = d < current_date - (win + 2), where win is the click_window_days read off the pinned model and version. On a spend-only day win is null, so is_mature is null rather than TRUE; decide whether to fall back to the model's configured window or to leave the flag unknown, and say which.
Re-run the same range a week later and diff the CPA column to measure the actual maturation curve rather than assuming the lag.

Expected result: Per (line_item_id, day): spend_usd, attributed_conversions as a fractional SUM(credit_fraction), cpa, roas and is_mature. A day with $4,400 spend and 137.5 attributed conversions returns a CPA of $32.00; $19,800 of credited value gives a ROAS of 4.5.

Verify it yourself

SUM(credit_fraction) across all line items for any single conversion_id equals 1.0 within the chosen model and version. A value of 2.0 or 3.0 means the model filter is missing or partial.
MIN(click_window_days) equals MAX(click_window_days) inside the pinned (attribution_model, model_version). If they differ, the constancy precondition is false, MAX is silently picking the longest window, and every is_mature value in the output is unsupported.
Total spend across the output equals an ungrouped SUM over billable impressions for the same dates; a larger figure proves the credit join fanned the impressions out.
Attributed conversions over the whole range does not exceed the count of non-duplicate conversion_event rows for that advertiser, since credit cannot exceed one per conversion.
Recomputing the last seven days a week later moves CPA downward. If nothing moves, the credit table is not being backfilled and the pipeline is stale.

Follow-up

Last Tuesday's CPA improved 15% overnight with no change in delivery. What happened, and what would you have shipped to prevent the question?
The same conversion has credit rows under last_click and under data_driven. Can you sum them? What do you give a client who wants one number?
model_version changed on the 14th. How do you present a series that spans that boundary without implying a performance change?

Explain the trade-offs between batch inference and real-time inference when deploying graph-based recommendation models at scale.

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you design a core metric to measure the health and long-term value of a newly launched affiliate marketing partnership channel?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Suppose daily active usage of our creator discovery tool drops by fifteen percent week-over-week. Walk us through your systematic approach to diagnosing this metric drop.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

If we wanted to introduce a completely new type of commission structure for publishers, what metrics would you track to ensure it benefits both sides of the marketplace?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Explain how you calculate statistical significance when dealing with highly skewed conversion rate distributions and low conversion volume metrics.

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

How do you design quasi-experiments or causal inference frameworks when standard randomized controlled trials are technically infeasible?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Measure invalid traffic when only labels are observable

Hard · measurement-quality · proxy-bias · restatement

ad_impression.ivt_status is one of valid, givt, sivt, written by a detection system and frequently rewritten days after delivery (ivt_classified_at_ts). Leadership wants a metric that answers whether invalid traffic is getting worse. True invalid traffic is unobservable; you have labels only. Deliverable: the metric, the direction and cause of its bias in one sentence, the auxiliary measurement that bounds that bias, and the evidence that would make you report that the series is not comparable across a period boundary.

Approach

Separate the estimand from the observable and keep them separate for the rest of the answer. The estimand is the share of delivered impressions that were not genuine human ad opportunities. The observable is the share a classifier labelled as invalid. They differ by the classifier's recall, which is unknown and changes with every detection release.
State the bias direction plainly and put it on the dashboard, not in an appendix: the labelled rate is at most the true rate, so the metric is biased downward, and the bias shrinks whenever detection improves. That means a detection upgrade and a genuine increase in invalid traffic produce the same upward movement in the published number, which is the entire difficulty of the question.
Split the two label families because they behave differently. General invalid traffic is list-based and settles within hours. Sophisticated invalid traffic is behavioural and is reclassified days later, so it restates numbers you already published. Measure the reclassification tail directly — the distribution of ivt_classified_at_ts minus served_ts — and freeze a period only after its 95th percentile has elapsed.
Bound the unknown recall with auxiliary measurement rather than pretending it is known. A seeded canary set of known-invalid traffic injected at a controlled rate yields a recall estimate. A reference slice of supply whose composition is held fixed separates classifier drift from supply-mix drift. An independent second vendor label on a sample yields a disagreement rate. Publish labelled rate, recall estimate and reference-slice rate together; none of the three is interpretable alone.
Define the abstention condition in advance, because the honest answer here is sometimes that you cannot tell. If a detection release landed inside the comparison window and the reference slice moved by more than the change in seeded-canary recall can account for, report the series as not comparable across that boundary instead of publishing a delta.
Prepare the restatement note as part of the metric, not as an incident response: what changed, which periods are affected, and the direction and size of the correction — a published number that moves later damages trust far less than one that moves silently.

Worked solution (45 min)

Compute the labelled rate split by givt and sivt, with the denominator as all delivered impressions including invalid ones, bucketed on served_ts.
Measure the reclassification tail from ivt_classified_at_ts minus served_ts and set the freeze lag at its 95th percentile.
Compute the labelled rate on a reference supply slice held fixed on publisher_id, ad_format and geo_country, so composition cannot move the series.
Estimate recall from the seeded canary set and express the labelled rate as a lower bound on the true rate given that recall.
Write the comparability rule and draft the restatement template.

Expected result: A metric spec: givt and sivt rates reported separately over all delivered impressions, frozen only after the measured 95th-percentile reclassification lag, published alongside a canary-derived recall estimate and a fixed-composition reference-slice rate, with a written statement that the labelled rate is a downward-biased proxy whose bias shrinks as detection improves, and a pre-defined condition under which no period-over-period delta is published at all.

Verify it yourself

Re-running a past month today yields a higher sivt rate than the number published at the time, confirming the reclassification tail is real and the freeze lag is necessary.
The fixed-composition reference slice does not move when the overall supply mix moves, which is the test that it isolates classifier behaviour.
The seeded canary recall estimate changes across a detection release, and the size of that change is compared against the movement in the published rate before any conclusion is drawn.
The denominator includes invalid impressions; excluding them makes the rate a ratio of invalid to valid rather than an invalid share.

Follow-up

The detection vendor ships a detector upgrade mid-quarter. How do you restate the numbers already published?
An advertiser demands a credit citing their own vendor's higher invalid rate. What is your response, and what would change it?
How does this bias interact with the viewability measurement rate on the same impressions?

Post-click conversion rate dropped on mobile app supply only

Hard · identity · funnel decomposition · instrumentation

Post-click conversion rate for one advertiser fell from 3.1% to 0.9% on device_type='mobile_app', with desktop and mobile_web flat. The metric is deduplicated conversion_event rows joined to ad_click on click_tracking_id within click_window_days, over valid billable ad_click rows, cohorted on click_ts. ad_click also has impression_id, device_id, landing_arrival_ts, ivt_status; conversion_event has source, device_id, click_tracking_id, dedup_key. The advertiser shipped a mobile SDK update that week. Decide whether conversions stopped happening or stopped being joinable, and recommend an action.

Approach

Cohort correctly before anything else: restrict to click_ts dates older than click_window_days plus ingestion lag, so an immature window is not mistaken for a segment-specific drop.
Split the single ratio into two consecutive steps — clicks to non-null landing_arrival_ts, and arrivals to joined conversions. A drop at the first step means users are not arriving, which is a real problem on the ad or landing side; a drop only at the second means they arrive and the join fails, which is instrumentation.
If the break is at the join step, measure click_tracking_id null rate in conversion_event split by source and advertiser, by day. A step change on the mobile_sdk source against a flat browser_pixel source localises it to the advertiser's SDK release, and the break will be a single timestamp rather than a ramp.
Rule out the denominator explanation explicitly, because it moves the metric the other way: rising sivt reclassification on mobile_app clicks removes rows from the denominator and would raise the rate, so an unchanged ivt_status distribution is a necessary check, not a formality.
Establish that the conversions still exist by counting total conversion_event rows for the advertiser across the break, then recover the joinable population on device_id within the window and report both the recovered rate and the residual recovery gap from null device_id.

Follow-up

What fraction of mobile_app clicks carry a null device_id, and what does that cap your recovery at?
Would you restate the advertiser's reported CPA for the affected week, or footnote it, and what changes that decision?
How would you monitor the joinable share as a first-class metric so this is caught the day it starts?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Tell me about a time you had to push back on a product manager or stakeholder who wanted to launch a feature without sufficient experimental evidence.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Claiming impact for three projects with honest counterfactuals

Easy · self-assessment · impact · measurement

You are writing a performance self-review. Over two quarters you shipped a bid-shading recalibration, a rewrite of the click-through-rate query that fixed an invalid-traffic filter applied to only one side of the ratio, and an incrementality test that led to a line item being paused. Deliverable: for each, the number you would claim, the counterfactual you claim it against, and the portion of credit that belongs to someone else. One of the three has no defensible dollar figure. Say which, and what you claim instead.

Approach

The interviewer is probing whether you apply the same causal standard to your own work that you apply to an advertiser's, so write the counterfactual for each claim before the number: what would the metric have been over the same period without this work?
For the shading recalibration, compare clearing price and profit on won impressions at matched supply mix across exchange_id, ad_format, device_type and geo_country, and report the profit change rather than the win-rate change; under first-price clearing a higher win rate can accompany worse profit.
For the query fix, claim the magnitude of the bias removed rather than a revenue number: the gap between the old rate, which filtered invalid traffic on the numerator only, and the corrected symmetric rate, plus the decision that was being made on the wrong figure.
For the incrementality test, claim the decision, not the savings. The paused budget's fate was never measured, so any dollar figure would assume that money vanished rather than moved. This is the item with no defensible number.
Name specific collaborators and what they contributed, and pre-empt the obvious challenge by stating the confound you know about, such as seasonality or supply mix drift, and the bound you put on it.

Follow-up

Your manager points out the paused line item's spend simply moved to another line item. Does your claimed impact survive?
How do you claim credit for work whose value is a bad decision that did not happen?

Disclosing an invalid-traffic reclassification that changes invoiced numbers

Hard · restatement · data-integrity · ethics · communication

A measurement vendor reclassifies sophisticated invalid traffic for a six-week period. On re-read, 4.1% of one advertiser's delivered impressions flip from valid to sivt, and their reported CPA rises about 6% across two already-invoiced months. You did not cause it, the contract's make-good language is ambiguous, and a colleague suggests waiting to see whether the advertiser notices. Deliverable: the note you send the advertiser, what you say to finance and the account team beforehand, and the position you take internally on waiting.

Approach

The interviewer is probing whether you will bound an uncomfortable number before negotiating about it, so recompute the six weeks with the restated ivt_status first, splitting givt from sivt, and produce the impact by campaign and by invoiced month: impressions flipped, billed amount on those exact rows, and the CPA delta.
Sequence disclosure so one number exists before anyone speaks externally: finance first, because they own the credit and the invoices; the account team second, because they own the relationship; the advertiser third, with the figure both internal parties have already approved.
Write the note in the restatement shape rather than as an apology: what was reclassified and by whom, which dates, the direction and size, the billing position, and what the advertiser needs to do, which is nothing.
Take the position on waiting explicitly and in commercial terms, not only ethical ones. The reclassification is visible in the vendor's own dashboard, so waiting converts a correction the advertiser receives from you into a discrepancy they discover themselves, which is the version that costs the renewal.
Close it structurally: treat invalid-traffic rates as restatable on read with a published reclassification lag, and subscribe to the vendor's reclassification feed so the next event is detected rather than discovered.

Follow-up

The account team says this disclosure will cost us the renewal. Does that change what you send?
The contract is silent on reclassification make-goods. How would you price one, and what precedent are you setting?
What do you do if finance instructs you not to send the note?