Interview guides · Goodway Group

Goodway Group Data Scientist Interview Questions & Guide 2026

Case practice for Goodway Group data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

Data Scientists at Goodway Group play a crucial role in harnessing data to drive strategic decision-making and enhance customer experiences. By leveraging advanced analytical techniques, you will contribute to building data-driven solutions that optimize advertising strategies and improve the overall effectiveness of marketing campaigns. This role is vital in transforming raw data into actionable insights, directly impacting the company's ability to serve its clients with precision and efficacy.

How much statistics you need depends on the flavour of the seat. Experiment-facing work wants you deep enough to notice that repeated looks at accumulating data inflate the false positive rate of a fixed-sample test; modelling-facing work wants estimation and honest uncertainty intervals.

Focus areas

Reconstruct the funnel from bid request to conversion
Design geo holdouts that survive budget interference
Diagnose pacing: underdelivery, overdelivery, budget exhaustion

Overview

As a Data Scientist, you will engage with complex datasets, collaborate with cross-functional teams, and implement machine learning models that help forecast trends and consumer behaviors. Your work will significantly influence product development, marketing strategies, and operational efficiencies, making this position both challenging and rewarding. The dynamic environment at Goodway Group offers opportunities to innovate and explore the frontiers of data science, making your contributions critical to the organization's success.

The Goodway Group interview process

Goodway Group candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Interviews

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Interviews

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
The counterfactual: what would have been done without your analysis, and why that default was worse
How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Final Interviews

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Where candidates lose points

Treating a bid as if it does not affect the price paid, under a first-price auction.

In a second-price auction the winner pays the runner-up's bid, so bidding true value is close to optimal and bid and price are usefully separable. Under first-price clearing, which is now the common case in open exchanges, the winner pays exactly what it bid, so raising the bid raises both the win probability and the cost of every win that would have been won anyway. A bid-increase experiment can therefore show more impressions, more conversions and worse profit simultaneously, and reading only the volume metrics will call it a success. The related second-order problem is that your own bid increase raises the clearing prices your control cells face, which contaminates the comparison unless the cells are separated by market.

Comparing conversion rates between users who were exposed to an ad and users who were not.

Exposure is the output of a targeting and bidding system that ranks users by predicted conversion propensity, so the exposed group has a higher baseline conversion rate before the ad renders. The measured difference therefore estimates the quality of the targeting model, not the effect of the advertising, and it is biased upward by an amount that grows as the model gets better. The valid control is the set of users the system would have shown the ad to but deliberately did not, captured through a ghost-bid or public-service-announcement holdout that records auction wins without serving the advertiser's creative. Anyone who proposes matching or covariate adjustment to fix this should be asked what unobserved feature the bidder used that the matching model does not have.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Dropping rows with missing values without naming the mechanism

Say whether the values are missing at random, missing by a known process, or missing in a way that depends on the outcome, and handle them accordingly. Deleting incomplete rows silently redefines the population whenever missingness correlates with what you are measuring.

Questions & practice

What is overfitting and how can it be prevented?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

Write a function to implement linear regression from scratch.

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Implement click-through rate and eCPM from written definitions

Medium · metrics · joins · filters · pandas

Two frames. impressions: impression_id, served_ts, line_item_id, creative_id, billed_price_micros_usd, is_billable, viewability_status, ivt_status. clicks: click_id, impression_id (nullable), click_ts, line_item_id, ivt_status, is_billable. Implement exactly these definitions: click-through rate is valid clicks whose parent impression is billable and valid, over billable valid impressions, with both sides bucketed on the parent impression's served_ts UTC date; eCPM is SUM(billed_price_micros_usd)/1e6 over billable impressions divided by their count over 1000, same bucketing. Return one row per (line_item_id, served_date) with impressions, clicks, ctr and ecpm_usd, plus a count of every row you dropped and why.

Approach

Filter impressions to is_billable and ivt_status == 'valid' once, and treat that frame as the single source of the denominator for both metrics. Deriving everything from one filtered population is what guarantees the two metrics describe the same impressions.
Join clicks into that filtered frame on impression_id from the click side, keeping only matched rows, because the definition conditions the numerator on the parent impression being billable and valid. Count unmatched clicks separately and split them into null impression_id (orphan beacons) and non-null but filtered-out parents, since a rising orphan share is a measurement fault rather than a CTR movement.
Take served_ts from the impression side after the join and never click_ts. Bucketing on click_ts puts a click in a different day bucket from its own impression, which distorts both days for any cohort near midnight.
Decide the click fan-out rule explicitly: one impression can carry more than one click row, so either deduplicate to one click per impression or state that you are counting click events. Without that decision CTR is not bounded by 1.
Aggregate the filtered impressions by (line_item_id, served_date) for count and summed micros, aggregate matched clicks on the same key, then outer-join and fill missing click counts with zero so zero-click line-item days survive instead of disappearing from the output.

Worked solution (25 min)

Build imp_f = impressions[is_billable & (ivt_status == 'valid')] and record len(impressions) - len(imp_f) as the filtered count.
Build clk_f = clicks[(ivt_status == 'valid')], then merge clk_f onto imp_f[['impression_id','served_ts','line_item_id']] with how='inner', keeping the impression-side served_ts. Record orphan and unmatched counts.
Deduplicate clicks to one per impression_id (or record that you did not), then groupby(['line_item_id','served_date']).size() on both frames.
Aggregate imp_f for impressions and billed micros on the same key, outer-join the click counts, fillna(0), and compute ctr = clicks/impressions and ecpm_usd = (micros/1e6)/(impressions/1000).

Expected result: A frame keyed on (line_item_id, served_date) with impressions, clicks, ctr, ecpm_usd, accompanied by a dropped-rows note giving filtered impressions, orphan clicks, unmatched clicks and invalid clicks as four separate integers.

Verify it yourself

out.impressions.sum() equals len(imp_f) exactly, and out.clicks.sum() equals the matched deduplicated click count. Any excess is join fan-out.
Pooled eCPM recomputed as total_micros/1e6 divided by total_impressions/1000 equals the impression-weighted mean of the per-row eCPMs. If it matches the unweighted mean instead, the weighting is wrong.
CTR lies in [0, 1] for every row once clicks are deduplicated per impression; a value above 1 means fan-out was not handled.
Re-running with the invalid-traffic filter removed from both sides moves CTR only slightly, while removing it from one side alone moves it by about the invalid share.

Follow-up

One impression has two click rows. Does your CTR exceed 1.0, and which definition would you put on an advertiser-facing report?
How does the answer change if the denominator becomes measured-viewable impressions rather than all valid billable impressions?
A filter change removed the cheapest 10% of inventory and eCPM rose 12%. How do you present that so nobody reads it as a price increase?

How would you optimize the performance of a large dataset query?

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Discuss the trade-offs between different database technologies for storing large datasets.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Daily effective CPM per line item from billable impressions

Easy · aggregation · monetisation · date-bucketing

ad_impression has impression_id, served_ts TIMESTAMPTZ, line_item_id, billed_price_micros_usd BIGINT, is_billable BOOLEAN, ivt_status TEXT (valid, givt, sivt) and currency_code. billed_price_micros_usd is the amount charged for that single impression, in USD micros. Write one query returning, per line_item_id and per UTC calendar day over a supplied date range, the billable impression count, total spend in USD, and effective CPM meaning spend per thousand impressions. Count only rows with is_billable = TRUE and ivt_status = 'valid'. Order by day, then spend descending.

Approach

Apply the two filters positively rather than by exclusion: is_billable = TRUE AND ivt_status = 'valid'. Writing ivt_status <> 'givt' silently keeps the sophisticated-invalid rows, which is the larger and later-arriving category.
Bucket with (served_ts AT TIME ZONE 'UTC')::date rather than date_trunc('day', served_ts). date_trunc on a TIMESTAMPTZ resolves in the session's TimeZone setting, so the same query returns different day boundaries for two analysts on the same warehouse.
Aggregate once: COUNT(*) and SUM(billed_price_micros_usd) grouped by line item and day. Everything else is arithmetic on those two columns.
Collapse the unit conversion into a single divisor. Micros to USD is a division by 1e6, per-thousand is a multiplication by 1000, so eCPM = SUM(billed_price_micros_usd) / (1000.0 * COUNT(*)). Use numeric literals so a BIGINT sum is not divided with integer arithmetic.
Return the impression count in the output, not just the rate. eCPM rises mechanically whenever cheap inventory is filtered out, so the rate is uninterpretable without the volume beside it.

Worked solution (15 min)

Count rows before and after the is_billable and ivt_status filters and note the drop; it is usually a few percent, and a sudden change in that drop is itself the finding.
Bucket the day as (served_ts AT TIME ZONE 'UTC')::date so the result does not depend on anyone's session timezone.
Group by (line_item_id, day) and emit COUNT(*) AS impressions and SUM(billed_price_micros_usd) AS micros.
Derive spend_usd = micros / 1e6 and ecpm_usd = micros / (1000.0 * impressions) in an outer select over that aggregate.
Order by day then spend_usd descending, and reconcile the top row against a line item whose spend you already know.

Expected result: One row per (line_item_id, day) with impressions, spend_usd and ecpm_usd. A line item with 4,000,000 billable valid impressions and 9,000,000,000 micros returns spend_usd = 9000.00 and ecpm_usd = 2.25.

Verify it yourself

Recomputing spend from the rate (ecpm_usd * impressions / 1000) matches the direct sum to the cent; if it does not, the divisor was applied twice.
Summing the daily spend_usd over the range equals an ungrouped total over the same filters.
No day is missing for a line item that delivered that day. A missing day usually means a filter dropped every row, most often a NOT IN comparison against a nullable column.
Shifting the session TimeZone and re-running returns identical day buckets, which proves the bucketing is genuinely UTC-pinned.

Follow-up

eCPM rose 12% week over week while spend was flat. What are the candidate causes, and which column separates a price change from a mix shift?
How do you compute a period-level eCPM across 30 days, and why is the mean of the 30 daily eCPMs a different number?
currency_code is not always USD. What does billed_price_micros_usd mean on those rows, and what would you need before the sum is trustworthy?

How do you prioritize your tasks when working on multiple projects?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How would you approach building a recommendation system for a retail client?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Given a dataset, how would you determine the most significant predictors of a target variable?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Describe your process for conducting an A/B test.

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

State your assumptions explicitly before working the problem.
Work from the decision backwards to the evidence you would need.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Measure invalid traffic when only labels are observable

Hard · measurement-quality · proxy-bias · restatement

ad_impression.ivt_status is one of valid, givt, sivt, written by a detection system and frequently rewritten days after delivery (ivt_classified_at_ts). Leadership wants a metric that answers whether invalid traffic is getting worse. True invalid traffic is unobservable; you have labels only. Deliverable: the metric, the direction and cause of its bias in one sentence, the auxiliary measurement that bounds that bias, and the evidence that would make you report that the series is not comparable across a period boundary.

Approach

Separate the estimand from the observable and keep them separate for the rest of the answer. The estimand is the share of delivered impressions that were not genuine human ad opportunities. The observable is the share a classifier labelled as invalid. They differ by the classifier's recall, which is unknown and changes with every detection release.
State the bias direction plainly and put it on the dashboard, not in an appendix: the labelled rate is at most the true rate, so the metric is biased downward, and the bias shrinks whenever detection improves. That means a detection upgrade and a genuine increase in invalid traffic produce the same upward movement in the published number, which is the entire difficulty of the question.
Split the two label families because they behave differently. General invalid traffic is list-based and settles within hours. Sophisticated invalid traffic is behavioural and is reclassified days later, so it restates numbers you already published. Measure the reclassification tail directly — the distribution of ivt_classified_at_ts minus served_ts — and freeze a period only after its 95th percentile has elapsed.
Bound the unknown recall with auxiliary measurement rather than pretending it is known. A seeded canary set of known-invalid traffic injected at a controlled rate yields a recall estimate. A reference slice of supply whose composition is held fixed separates classifier drift from supply-mix drift. An independent second vendor label on a sample yields a disagreement rate. Publish labelled rate, recall estimate and reference-slice rate together; none of the three is interpretable alone.
Define the abstention condition in advance, because the honest answer here is sometimes that you cannot tell. If a detection release landed inside the comparison window and the reference slice moved by more than the change in seeded-canary recall can account for, report the series as not comparable across that boundary instead of publishing a delta.
Prepare the restatement note as part of the metric, not as an incident response: what changed, which periods are affected, and the direction and size of the correction — a published number that moves later damages trust far less than one that moves silently.

Worked solution (45 min)

Compute the labelled rate split by givt and sivt, with the denominator as all delivered impressions including invalid ones, bucketed on served_ts.
Measure the reclassification tail from ivt_classified_at_ts minus served_ts and set the freeze lag at its 95th percentile.
Compute the labelled rate on a reference supply slice held fixed on publisher_id, ad_format and geo_country, so composition cannot move the series.
Estimate recall from the seeded canary set and express the labelled rate as a lower bound on the true rate given that recall.
Write the comparability rule and draft the restatement template.

Expected result: A metric spec: givt and sivt rates reported separately over all delivered impressions, frozen only after the measured 95th-percentile reclassification lag, published alongside a canary-derived recall estimate and a fixed-composition reference-slice rate, with a written statement that the labelled rate is a downward-biased proxy whose bias shrinks as detection improves, and a pre-defined condition under which no period-over-period delta is published at all.

Verify it yourself

Re-running a past month today yields a higher sivt rate than the number published at the time, confirming the reclassification tail is real and the freeze lag is necessary.
The fixed-composition reference slice does not move when the overall supply mix moves, which is the test that it isolates classifier behaviour.
The seeded canary recall estimate changes across a detection release, and the size of that change is compared against the movement in the published rate before any conclusion is drawn.
The denominator includes invalid impressions; excluding them makes the rate a ratio of invalid to valid rather than an invalid share.

Follow-up

The detection vendor ships a detector upgrade mid-quarter. How do you restate the numbers already published?
An advertiser demands a credit citing their own vendor's higher invalid rate. What is your response, and what would change it?
How does this bias interact with the viewability measurement rate on the same impressions?

Click-through rate fell while click volume stayed flat

Easy · ctr · supply mix · denominator

Platform click-through rate fell from 0.42% to 0.34% day over day, while the raw count of ad_click rows was unchanged. CTR here is valid billable clicks over valid billable impressions, both bucketed on the parent impression's served_ts. You have ad_impression (impression_id, auction_id, served_ts, is_billable, ivt_status, publisher_id, placement_id, creative_id, os_family), ad_click (click_id, impression_id, click_ts, ivt_status, is_billable) and bid_request_log (auction_id, exchange_id, ad_format, device_type). Deliver a one-paragraph verdict: real engagement decline or denominator artefact, with the evidence that decides it.

Approach

Plot the numerator and the denominator as absolute counts for 14 days before computing any ratio. A flat numerator against a step in the denominator localises the change in one plot and rules out creative and audience explanations without a single join.
Split the denominator into its four filter populations by served_ts date — all rows, is_billable only, ivt_status='valid' only, and both. If the billable-and-valid count jumped while total impressions did not, a filter moved, not the traffic.
If total impressions genuinely grew, join ad_impression to bid_request_log on auction_id and cut the increment by exchange_id, publisher_id and ad_format, looking specifically for publisher_id values whose first served_ts falls inside the window.
Recompute CTR within each publisher_id present in both periods. Flat within-publisher CTR alongside a falling pooled rate is a mix shift; the pooled number is describing a different population, not different behaviour.
Check the click side for the mirror error: clicks bucketed on click_ts rather than the parent impression's served_ts shift by roughly one impression-to-click latency, which depresses the most recent day and recovers by itself tomorrow.

Follow-up

The new publisher is 30% of billable impressions and 4% of clicks. What do you need before recommending it be blocked, and what does blocking it do to eligible RPM?
How does your answer change if the invalid-traffic job back-filled sivt labels onto impressions served three days ago?
Would you publish CTR pooled across supply sources at all, and if so with what accompanying number?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Describe a situation where you had to communicate complex data insights to non-technical stakeholders.

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Disclosing an invalid-traffic reclassification that changes invoiced numbers

Hard · restatement · data-integrity · ethics · communication

A measurement vendor reclassifies sophisticated invalid traffic for a six-week period. On re-read, 4.1% of one advertiser's delivered impressions flip from valid to sivt, and their reported CPA rises about 6% across two already-invoiced months. You did not cause it, the contract's make-good language is ambiguous, and a colleague suggests waiting to see whether the advertiser notices. Deliverable: the note you send the advertiser, what you say to finance and the account team beforehand, and the position you take internally on waiting.

Approach

The interviewer is probing whether you will bound an uncomfortable number before negotiating about it, so recompute the six weeks with the restated ivt_status first, splitting givt from sivt, and produce the impact by campaign and by invoiced month: impressions flipped, billed amount on those exact rows, and the CPA delta.
Sequence disclosure so one number exists before anyone speaks externally: finance first, because they own the credit and the invoices; the account team second, because they own the relationship; the advertiser third, with the figure both internal parties have already approved.
Write the note in the restatement shape rather than as an apology: what was reclassified and by whom, which dates, the direction and size, the billing position, and what the advertiser needs to do, which is nothing.
Take the position on waiting explicitly and in commercial terms, not only ethical ones. The reclassification is visible in the vendor's own dashboard, so waiting converts a correction the advertiser receives from you into a discrepancy they discover themselves, which is the version that costs the renewal.
Close it structurally: treat invalid-traffic rates as restatable on read with a published reclassification lag, and subscribe to the vendor's reclassification feed so the next event is detected rather than discovered.

Follow-up

The account team says this disclosure will cost us the renewal. Does that change what you send?
The contract is silent on reclassification make-goods. How would you price one, and what precedent are you setting?
What do you do if finance instructs you not to send the note?

Ranking three teams' requests against one sprint of capacity

Medium · prioritisation · stakeholder-management · delivery

You have three requests and capacity for roughly one and a half. The supply team wants eligible-RPM impact sizing for a floor-price change shipping in two weeks. An advertiser team wants a pacing diagnosis for a $400k line item delivering at 60% of budget. The invalid-traffic team wants help validating a new sophisticated-invalid-traffic classifier before it restates last quarter. Deliverable: your ranking with the reason for each, exactly what you say to the two teams you deprioritise, and the trigger that would reorder the list.

Approach

The interviewer is probing whether you rank on decision structure or on social pressure, so attach two attributes to each request before ranking: the date the decision is actually made, and how expensive it is to reverse afterwards. Seniority of the requester is not one of the inputs.
Look for the request that can be collapsed rather than sequenced. The pacing diagnosis is usually a single grouped query over bid_request_log by no_bid_reason and response_status per hour for that line_item_id; time-boxed to two hours it stops competing for the sprint at all.
Size the other two in revenue at risk per week so the ranking is arguable rather than asserted: the floor change's expected eligible-RPM delta against the spend it touches, and the invoiced spend inside the period the classifier would restate.
Rank on irreversibility where deadlines are close together. A classifier that restates already-invoiced periods is the hardest of the three to unwind once shipped, which can outrank a nearer deadline.
Deprioritise with a date and a fallback rather than an apology: name what you will not do, when you could, the smallest thing you can give them now, and who else could take it.

Follow-up

The underdelivering advertiser escalates to your director. What actually changes in your plan, and what do you refuse to change?
How do you stop the pacing question arriving fresh every sprint?