Interview guides · MiQ

MiQ Data Scientist Interview Questions & Guide 2026

Data scientist prep for MiQ candidates: drill A/B design, power and minimum detectable effect on original exercises.

The 60-second brief

As a Data Scientist at MiQ, you sit at the intersection of programmatic advertising, high-scale data engineering, and advanced statistical modeling. MiQ operates in a fast-paced environment where the ability to derive actionable insights from massive, fragmented datasets is the primary competitive advantage. Your work directly influences how global brands optimize their marketing spend, requiring you to bridge the gap between complex algorithmic outputs and tangible product metrics.

Coding rounds for this role are usually data-manipulation shaped rather than data-structure shaped: group-bys, joins, time windows, ranking within a partition. Confirm the format before spending a week on graph traversal.

Focus areas

Reason about bid shading under first-price auctions
Defend attribution windows and model choice
Diagnose pacing: underdelivery, overdelivery, budget exhaustion

Overview

You will contribute to building and refining the machine learning pipelines that power MiQ’s proprietary advertising solutions. This role is not just about building models; it is about product-sense and experimentation. You will be expected to design robust metrics, diagnose performance drops in real-time, and ensure that every experiment you run is statistically sound. Because the advertising landscape is highly dynamic, your ability to simplify complex technical concepts for stakeholders is just as critical as your ability to write efficient SQL or debug a model.

The MiQ interview process

MiQ candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Technical Rounds

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Managerial Rounds

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Final Conversation

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Counting conversions without deduplicating across reporting sources and identity spaces.

The same purchase routinely arrives twice, once from a browser pixel and once from a server-side API, and the two reports carry different identifiers, so a naive count inflates conversions and deflates cost per action by the duplication rate. Deduplication depends on the advertiser populating a shared dedup_key correctly, which many do not, and the failure is silent because both rows look individually valid. The mirror-image error is under-counting: when a user clicks on a mobile app and converts in a desktop browser, the identity join fails and a real conversion is attributed to nothing, which makes the channel look worse than it is. Any conversion count reported without stating the dedup rule and the identity-match rate behind it should be treated as unverified.

Reading a recent week's cost per action or return on ad spend before the conversion window has closed.

Spend for a period is final within hours, but conversions attributed to that period keep arriving for as long as the click window allows plus ingestion lag, and offline uploads can land weeks later. The denominator of cost per action is therefore systematically incomplete for recent dates, so recent CPA is biased high and recent ROAS biased low, and a dashboard comparing a fresh week to a matured week will show a regression that does not exist. The correct handling is to freeze a period only after click_window_days plus the observed ingestion lag, and to publish a maturation curve so readers can see how much of a given day's conversions have landed so far.

Never asking what decision the analysis will inform

Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.

SQL that silently fans out on a one-to-many join

State the grain of each table and the grain you want in the result before writing the join. Pre-aggregate the many side to the join key, or use EXISTS or a window function, and verify with a row count against COUNT(DISTINCT id) rather than trusting that the numbers look plausible.

Questions & practice

What is the Bias-Variance Tradeoff, and how do you manage it in practice?

Medium · statistics and probability

Approach

Translate the result into the decision it informs, in one plain sentence.
Write down the assumption the method needs before you use the method.
Sanity-check the answer against a simple bound or a simulated case.

Follow-up

How would you explain this result to someone who does not know statistics?
What sample size would you need to detect an effect half this size?

How would you design an A/B test to measure the effectiveness of a new bidding algorithm?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Simulate bid shading profit under first-price auction clearing

Hard · simulation · auctions · bid-shading · monte-carlo

auctions holds 200,000 backtest rows for one line_item_id: predicted_cvr (per impression), floor_price_cpm_usd (nullable when the floor is undisclosed), and best_competing_cpm_usd, the highest competing bid, observable only in this sample. Clearing is first price. The advertiser values a conversion at $40. For shading factors s from 0.50 to 1.00 in steps of 0.05, the submitted bid is s * 1000 * 40 * predicted_cvr and the line item wins when that bid clears both the floor and the best competing bid. Return win rate, spend, conversions and profit per thousand auctions, the profit-maximising s, and an uncertainty estimate for it.

Approach

Separate what is deterministic from what is random before writing any sampling code. Given the bid, winning is deterministic and the price paid under first-price clearing is exactly the submitted bid, so the only stochastic element is whether a won impression converts. Use the closed form sum of predicted_cvr over wins for the expected conversion count and reserve Monte Carlo for the profit distribution, not the mean.
Vectorise across s: build the bid vector once, then evaluate the 11 shading factors as numpy operations over the full array. A per-row Python loop over 200,000 rows by 11 factors is about two million iterations and turns a 3-second job into minutes.
Make the win condition explicit and handle nulls deliberately: treat a null floor as no floor, require bid > best_competing_cpm_usd, and require bid >= floor_price_cpm_usd. Ties at exactly the floor behave differently from ties against a competitor, and the convention should be stated rather than inherited from a comparison operator chosen by accident.
Compute profit per thousand auctions as (40 * conversions - sum of bid/1000 over wins) / n * 1000. Spend must use your own submitted bid, not the competing bid: under first-price clearing every inframarginal win, one you would have taken at a lower bid, now costs more, and that term is what creates an interior optimum.
Get uncertainty by bootstrapping auction rows, recomputing the whole curve per replicate, and reporting the distribution of the argmax s rather than only a confidence band around the profit level. The decision uses the argmax, and it is far less stable than the profit curve it comes from.
State the backtest's limit honestly: best_competing_cpm_usd was observed while you bid what you actually bid. If your change alters competitors' behaviour or triggers exchange-side floor adjustments, the counterfactual does not hold and the curve overstates the achievable gain.

Worked solution (35 min)

Precompute value_cpm = 1000 * 40 * predicted_cvr and floor_eff = floor_price_cpm_usd filled with -inf.
For each s: bid = s * value_cpm; win = (bid > best_competing_cpm_usd) & (bid >= floor_eff); spend = bid[win].sum()/1000; expected conversions = predicted_cvr[win].sum().
Profit per thousand auctions = (40 * conversions - spend) / n * 1000; assemble the 11-row curve with win rate = win.mean().
Bootstrap: resample row indices with replacement 500 times, recompute the curve on each replicate, and collect the argmax s plus a percentile band on profit at the point-estimate optimum.
Optionally draw Bernoulli(predicted_cvr) on won rows for one replicate set to show the profit distribution around the expectation, and confirm the sampled mean matches the closed form.

Expected result: An 11-row curve of (s, win_rate, spend_per_1k, conversions_per_1k, profit_per_1k) with win rate non-decreasing in s, profit single-peaked at an s strictly below 1.00 under a typical competing-bid distribution, and expected profit approaching zero at s = 1.00. The bootstrap returns a distribution over the argmax that usually spans two or three adjacent grid points rather than one.

Verify it yourself

Win rate is non-decreasing across the s grid; a dip means the floor or tie comparison is inconsistent between iterations.
At s = 1.00 expected profit is approximately zero, because the bid equals the expected revenue on every impression won. A materially positive value there means spend is being charged at the competing bid rather than at your own submitted bid. This identity holds only if predicted_cvr is calibrated on the won population, so verify that before reading it as a bug.
Total spend at any s equals the sum of submitted bids over won rows divided by 1000, reconciled against a direct recomputation from the bid vector.
Monte Carlo conversions per thousand match the closed-form sum of predicted_cvr over wins within sampling error; a systematic gap means the Bernoulli draw is using the wrong probability or is being applied to lost auctions.

Follow-up

Raising this line item's bid also raises the clearing price faced by another line item in the same account bidding on overlapping supply. How does that change the optimum, and how would you detect it in the logs?
predicted_cvr over-predicts by 15% at the head of the distribution. Which way does the optimal s move, and roughly how far?
How would you run this as a live test instead of a backtest, and what is the randomisation unit given that both arms draw on one budget?

How would you handle missing data or outliers during the data pre-processing stage?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Write a query using SQL window functions to calculate a moving average of ad spend.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Describe how you would join and aggregate data from multiple disparate sources to build a training dataset.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Time-weighted daily budget from slowly changing line item versions

Hard · scd type 2 · window functions · interval weighting · pacing

line_item is SCD type 2: one row per version with line_item_id, daily_budget_usd, pacing_mode, status, valid_from_ts, valid_to_ts (null on the current row) and is_current. ad_impression has line_item_id, served_ts, billed_price_micros_usd and is_billable. For one UTC date, compute budget delivery rate per line item: spend divided by a time-weighted daily budget, where the day's budget is each overlapping version's daily_budget_usd weighted by the fraction of the 24 hours that version was in force. Return spend, weighted budget, the weight coverage, and attainment.

Approach

Select versions that overlap the day with valid_from_ts < day_end AND COALESCE(valid_to_ts, 'infinity'::timestamptz) > day_start. Use strict inequality on both sides so a version that ends exactly at midnight does not also claim the following day, which would make the weights sum above 1.
Clip each version to the day before weighting: start at GREATEST(valid_from_ts, day_start) and end at LEAST(COALESCE(valid_to_ts, 'infinity'::timestamptz), day_end). Weighting on the unclipped interval gives a month-long version a weight of 30 rather than 1.
Derive the weight as EXTRACT(EPOCH FROM (clipped_end - clipped_start)) / 86400.0 and sum daily_budget_usd * weight per line item. Keep SUM(weight) in the output as coverage: it should be 1.0 for a line item live all day and below 1 for one created or ended mid-day, and anything above 1 means the interval predicate is wrong.
If valid_to_ts is unreliable on superseded rows, derive the interval end with LEAD(valid_from_ts) OVER (PARTITION BY line_item_id ORDER BY valid_from_ts) instead, and say which source you trust. Mixing the two produces overlapping intervals that double-count budget.
Aggregate spend in its own CTE at impression grain, filtered to is_billable and bucketed on served_ts in UTC, then join the two pre-aggregated results on line_item_id. Joining the SCD table to impressions before aggregating multiplies every impression by the number of versions.
Compute attainment as spend / NULLIF(weighted_budget, 0), and decide explicitly how to treat versions whose status was paused or ended for part of the day, since including a paused stretch in the denominator reads as underdelivery that never had a chance to happen.

Worked solution (40 min)

Declare day_start and day_end as explicit TIMESTAMPTZ bounds, then select line_item rows where valid_from_ts < day_end AND COALESCE(valid_to_ts, 'infinity'::timestamptz) > day_start.
Add w := EXTRACT(EPOCH FROM (LEAST(COALESCE(valid_to_ts, 'infinity'::timestamptz), day_end) - GREATEST(valid_from_ts, day_start))) / 86400.0 to each surviving row.
Build budget AS (SELECT line_item_id, SUM(daily_budget_usd * w) AS weighted_budget, SUM(w) AS coverage FROM clipped GROUP BY 1).
Build spend AS (SELECT line_item_id, SUM(billed_price_micros_usd) / 1e6 AS spend FROM ad_impression WHERE is_billable AND served_ts >= day_start AND served_ts < day_end GROUP BY 1), entirely independent of the dimension.
FULL OUTER JOIN budget and spend on line_item_id, compute spend / NULLIF(weighted_budget, 0), and inspect the rows where coverage differs from 1.0 as a separate list.

Expected result: Per line item: spend, weighted_budget, coverage and attainment. A budget of $1,000 in force from 00:00 to 18:00 and $2,500 from 18:00 to 24:00 gives weighted_budget = 10000.75 + 25000.25 = $1,375, and spend of $1,210 gives 88.0% attainment at coverage 1.0.

Verify it yourself

For any line item live the whole day, the clipped intervals tile the 24 hours with no overlap and no gap, so SUM(w) is exactly 1.0.
Exactly one row per line_item_id has is_current = TRUE. If more do, the SCD writer is broken and every weight downstream is wrong.
Spend from this query equals a plain aggregate over ad_impression with no line_item join at all. A higher figure is proof the version join fanned out.
A line item with a single version covering the whole day returns attainment identical to the naive spend / daily_budget_usd calculation, which is the degenerate case the weighting must reduce to.

Follow-up

A line item shows 180% attainment. Give three explanations that are not overspend.
How does the same calculation change for lifetime_budget_usd across a multi-week flight rather than a daily budget?
pacing_mode is 'even' but hourly spend is front-loaded and stops at 14:00. Which query distinguishes budget exhaustion from a pacing throttle, and which column settles it?

How do you diagnose a sudden drop in a key product metric, such as Click-Through Rate?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize tasks when working on multiple high-impact data initiatives?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Define the Central Limit Theorem and its relevance to A/B testing.

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

What are the most common experimentation pitfalls you have encountered?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

Measure invalid traffic when only labels are observable

Hard · measurement-quality · proxy-bias · restatement

ad_impression.ivt_status is one of valid, givt, sivt, written by a detection system and frequently rewritten days after delivery (ivt_classified_at_ts). Leadership wants a metric that answers whether invalid traffic is getting worse. True invalid traffic is unobservable; you have labels only. Deliverable: the metric, the direction and cause of its bias in one sentence, the auxiliary measurement that bounds that bias, and the evidence that would make you report that the series is not comparable across a period boundary.

Approach

Separate the estimand from the observable and keep them separate for the rest of the answer. The estimand is the share of delivered impressions that were not genuine human ad opportunities. The observable is the share a classifier labelled as invalid. They differ by the classifier's recall, which is unknown and changes with every detection release.
State the bias direction plainly and put it on the dashboard, not in an appendix: the labelled rate is at most the true rate, so the metric is biased downward, and the bias shrinks whenever detection improves. That means a detection upgrade and a genuine increase in invalid traffic produce the same upward movement in the published number, which is the entire difficulty of the question.
Split the two label families because they behave differently. General invalid traffic is list-based and settles within hours. Sophisticated invalid traffic is behavioural and is reclassified days later, so it restates numbers you already published. Measure the reclassification tail directly — the distribution of ivt_classified_at_ts minus served_ts — and freeze a period only after its 95th percentile has elapsed.
Bound the unknown recall with auxiliary measurement rather than pretending it is known. A seeded canary set of known-invalid traffic injected at a controlled rate yields a recall estimate. A reference slice of supply whose composition is held fixed separates classifier drift from supply-mix drift. An independent second vendor label on a sample yields a disagreement rate. Publish labelled rate, recall estimate and reference-slice rate together; none of the three is interpretable alone.
Define the abstention condition in advance, because the honest answer here is sometimes that you cannot tell. If a detection release landed inside the comparison window and the reference slice moved by more than the change in seeded-canary recall can account for, report the series as not comparable across that boundary instead of publishing a delta.
Prepare the restatement note as part of the metric, not as an incident response: what changed, which periods are affected, and the direction and size of the correction — a published number that moves later damages trust far less than one that moves silently.

Worked solution (45 min)

Compute the labelled rate split by givt and sivt, with the denominator as all delivered impressions including invalid ones, bucketed on served_ts.
Measure the reclassification tail from ivt_classified_at_ts minus served_ts and set the freeze lag at its 95th percentile.
Compute the labelled rate on a reference supply slice held fixed on publisher_id, ad_format and geo_country, so composition cannot move the series.
Estimate recall from the seeded canary set and express the labelled rate as a lower bound on the true rate given that recall.
Write the comparability rule and draft the restatement template.

Expected result: A metric spec: givt and sivt rates reported separately over all delivered impressions, frozen only after the measured 95th-percentile reclassification lag, published alongside a canary-derived recall estimate and a fixed-composition reference-slice rate, with a written statement that the labelled rate is a downward-biased proxy whose bias shrinks as detection improves, and a pre-defined condition under which no period-over-period delta is published at all.

Verify it yourself

Re-running a past month today yields a higher sivt rate than the number published at the time, confirming the reclassification tail is real and the freeze lag is necessary.
The fixed-composition reference slice does not move when the overall supply mix moves, which is the test that it isolates classifier behaviour.
The seeded canary recall estimate changes across a detection release, and the size of that change is compared against the movement in the published rate before any conclusion is drawn.
The denominator includes invalid impressions; excluding them makes the rate a ratio of invalid to valid rather than an invalid share.

Follow-up

The detection vendor ships a detector upgrade mid-quarter. How do you restate the numbers already published?
An advertiser demands a credit citing their own vendor's higher invalid rate. What is your response, and what would change it?
How does this bias interact with the viewability measurement rate on the same impressions?

Weekly eCPM lift that survives no calendar alignment

Medium · seasonality · ecpm · mix shift

Supply leadership reports eCPM up 6.8% week over week and wants to credit a floor-price change shipped ten days ago. eCPM is SUM(billed_price_micros_usd)/1e6 over billable ad_impression rows, divided by their count over 1000, bucketed on served_ts date UTC. The comparison is the trailing 7 days against the prior 7 days; one of those windows contains a public holiday. You also have ad_format and device_type via bid_request_log on auction_id. Decide whether the lift is real and state the size of the real component.

Approach

Align the calendar before comparing anything. Rebuild both windows so they contain the same count of each weekday, and either exclude the holiday or compare it against the same holiday in a prior year rather than against an ordinary weekday.
Plot a 28-day trailing series of daily eCPM and check whether there is a level shift at the ship date at all. A change that is only visible in a two-point comparison usually is not there.
Decompose eCPM the same way as any weighted mean: within-stratum price change versus mix across ad_format and device_type. ctv and instream_video carry much higher prices, so a weekend-heavy window inflates eCPM with no price change anywhere.
Report impression count next to eCPM. A floor raise removes low-priced inventory, so eCPM rises mechanically while revenue can fall; eCPM alone cannot tell you which happened.
Judge the floor change on eligible RPM — viewability-adjusted, invalid-traffic-filtered revenue per thousand eligible bid requests — because that denominator keeps the unsold opportunity the floor created in scope.

Follow-up

The floor change did raise eCPM and cut fill. What number decides whether to keep it, and what would make you reverse it?
How would you build a day-of-week adjustment that survives a moving holiday like Eid or Thanksgiving?
If ctv share is genuinely growing week on week, is the eCPM trend real or not — and does that distinction matter to the decision?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Bring numbers, including the ones you are unsure of

An answer without a quantity is hard to interrogate, so interviewers keep probing until they find one. Come with the baseline, the change, the window it was measured over, and how confident you were. If the effect never got measured, say so and say what you would have measured. Fabricated precision is worse than an honest gap.

Tell me about a project where your initial hypothesis was proven wrong.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Scoping a one-line request that attribution is wrong

Easy · scoping · stakeholder-management · attribution

A sales lead forwards a one-line request: 'attribution is wrong for this advertiser, fix it.' You have read access to conversion_event, attribution_credit and ad_click for the account, and thirty minutes with the account manager. You may not contact the advertiser this week. Deliverable: a one-page scope naming the single question you will answer, the questions you are explicitly not answering, the data you need, and the decision the answer changes. Bring the three clarifying questions you would ask the account manager first.

Approach

The interviewer is probing whether you convert a complaint into a comparison before you start work, so begin by writing the complaint as 'number A versus number B' and leave B blank until the account manager names it: attributed conversions against the advertiser's own order table, attributed CPA against last month, or our report against a second vendor's. Each has a different investigation.
Run the two cheap arithmetic checks that resolve a large share of these before any modelling: confirm SUM(credit_fraction) per conversion_id equals 1.0 within one attribution_model and model_version, and measure the duplication rate by counting dedup_key values that appear with more than one distinct source in conversion_event.
Measure identity coverage for the account: share of conversion_event rows carrying a non-null click_tracking_id and share carrying device_id. A low match rate means the channel is under-credited, which is the opposite complaint and needs the opposite fix.
Decide the scope from what the answer will change. If the decision is budget reallocation, the honest scope is an incrementality read, not an attribution repair. If the decision is an invoice dispute, the scope is a reconciliation against the advertiser's order count.
Write the page with an explicit out-of-scope list: model choice, window changes and anything requiring advertiser data you cannot get this week, each with the condition that would bring it back in scope.

Follow-up

The account manager says the advertiser just wants the numbers to match. What do you tell them is actually achievable, and why is exact agreement not one of the options?
How does your scope change if the discrepancy is 4% rather than 40%?

Disagreeing with a proposal to widen the view window

Medium · disagreement · attribution · windows · influence

A product manager proposes changing the default view window on attribution_credit from 1 day to 7 days, to bring reported performance in line with competitors. Their mock shows reported CPA falling 22%. You believe nothing real changes. You have read access to attribution_credit, conversion_event and ad_impression, and two days before the decision review. Deliverable: the analysis you run and the one-page argument you take into the review, including what you concede and the condition under which you would support the change.

Approach

The interviewer is probing whether you can disagree from inside the other person's numbers rather than from principle, so reproduce their 22% first under the same attribution_model and conversion cohort with view_window_days set to 7, and say out loud that it is correct. The disagreement is now about interpretation, which is a debate you can win.
Partition the newly credited volume, which is the whole argument. Group by conversion_id under both windows and split newly credited conversions into those that already carried credit on another touchpoint, which is reshuffling between channels, and those that carried no credited touchpoint at all, which is genuinely new claiming. Report the split as a percentage.
Test the substantive claim on the newly claimed share: look for holdout evidence that view-through exposure at 1 to 7 day latency produces measurable lift. If no such evidence exists, say so plainly and label that share unvalidated rather than incremental.
Price the cost the proposal does not mention: changing the default changes model_version, which restates history, breaks every published time series, and moves advertiser-facing numbers that some advertisers reconcile against their own order tables.
Offer a conditional yes with a mechanism: support a window set from the observed conversion-latency distribution and corroborated by holdout lift, shipped as a parallel model_version alongside the existing one rather than as a replacement.

Follow-up

The product manager says competitors already report this way and we look worse by comparison. Does that change your answer?
How would you choose a defensible view window from data rather than from convention?