Interview guides · Skillz

Skillz Data Scientist Interview Questions & Guide 2026

Data scientist prep for Skillz candidates: drill A/B design, power and minimum detectable effect on original exercises.

The 60-second brief

As a Data Scientist at Skillz, you sit at the intersection of competitive mobile gaming, real-time data analytics, and platform optimization. Your primary mission is to leverage the vast amounts of telemetry data generated by millions of players to refine game mechanics, improve matchmaking algorithms, and detect fraudulent behavior. Because Skillz operates a high-stakes, real-money competitive environment, your work directly impacts user retention, platform liquidity, and the overall integrity of the ecosystem.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

Compute expected cost of pity-timer tables
Diagnose churn by progression stage, not calendar day
Price cosmetics without cannibalising existing sinks

Overview

This role is highly product-focused. You will not be working in a silo; you will collaborate closely with product managers and engineers to design experiments that test new features and analyze the impact of changes on core business metrics. You will be expected to move quickly, translating ambiguous business problems into rigorous, data-driven hypotheses. The environment is fast-paced and requires a blend of strong technical execution and the ability to communicate findings to stakeholders who may not have a technical background.

The Skillz interview process

Skillz candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Technical Interviews

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Where candidates lose points

Testing revenue with a difference in means on a heavy-tailed spend distribution.

Player spend is approximately log-normal with a long Pareto-like tail, so the variance of the sample mean is dominated by a few accounts and the central limit approximation converges slowly at realistic sample sizes. A t-test on raw ARPDAU can flip sign when one account buys a large currency pack. Winsorising or capping at a pre-registered percentile, or testing conversion and conditional spend separately, fixes the variance problem, but note the cost: capping biases toward zero exactly when the true effect lives in the tail, so state the cap before you look.

Randomising players individually for a change that acts on a shared pool.

Matchmaking parameters, queue populations, tradeable-item supply and event leaderboards are shared resources: a treated player is matched against a control player, and treated supply lands in the control arm's market. The treatment leaks across arms, so the measured difference understates or reverses the true effect, and the control arm is no longer a clean counterfactual. Cluster by region and mode, or switchback on time slices with a burn-in long enough to clear carryover in the shared state.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Averaging per-user rates to produce a population rate

Decide which quantity you want: the mean of per-user ratios and the ratio of summed numerator to summed denominator are different estimands, and heavy users dominate one but not the other. For a ratio metric, aggregate numerator and denominator separately and use the delta method for its variance.

Questions & practice

If your p-value is 0.06, how do you decide whether to launch a feature?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

How would you explain this result to someone who does not know statistics?
What sample size would you need to detect an effect half this size?

How would you design a product metric to measure the "fairness" of a 2-player matching algorithm?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Three integrity checks over a ledger and purchase join

Medium · data quality · join fan-out · reversals

Given ledger (ledger_id, player_id, occurred_at, currency_code, delta_amount, balance_after, is_reversal, reversed_ledger_id, transaction_id) and txns (transaction_id, player_id, purchase_ts, price_usd_net, status), write check_ledger(ledger, txns) returning violations as rule_code, player_id, ledger_id, detail. Rule 1: balance_after that does not equal the prior balance_after plus delta_amount within (player_id, currency_code) ordered by (occurred_at, ledger_id). Rule 2: is_reversal rows whose reversed_ledger_id is missing, points at a different player or currency, or whose delta_amount is not the exact negative of its target. Rule 3: transaction_ids carried on more than one ledger row. For rule 3 also report completed net revenue computed before and after an inner merge of txns onto ledger.

Approach

Rule 1 is a shifted comparison inside the sorted group: take balance_after.shift() within (player_id, currency_code) ordered by (occurred_at, ledger_id) and flag where balance_after minus the prior value does not equal delta_amount. Compare the first row of each group against delta_amount alone. Keep the arithmetic in int64; a float cast makes large balances differ at the eighth significant digit and floods the rule with phantom violations.
Rule 2 is a self-merge of the reversal rows back onto the ledger by reversed_ledger_id. Use a left merge so a dangling pointer survives as a null and becomes a violation. An inner merge deletes precisely the rows you were asked to find.
Split rule 2 into three codes rather than one: missing target, mismatched player or currency, and a delta that is not the exact negative. Collapsing them throws away the diagnostic that tells you whether this is a bad backfill or a bad writer.
Rule 3 is ledger.groupby('transaction_id').size() over non-null transaction_ids, kept where the count exceeds one. Then compute completed net revenue two ways: from txns alone, and from the same rows after an inner merge onto ledger. The merged figure is inflated by exactly the fan-out multiplicity, and it is the figure that ends up in a revenue deck when someone joins for an item name.
Return one long frame with a rule_code column rather than three separate frames. A monitor wants counts per rule per day, and a wide shape makes that awkward to chart and awkward to alert on.

Worked solution (30 min)

Sort once on (player_id, currency_code, occurred_at, ledger_id) and build the shifted balance comparison in int64, handling the first row of each group as its own case.
Left-merge reversal rows onto the ledger by reversed_ledger_id and emit three distinct rule codes from the merged result.
Group the non-null transaction_ids, keep the fan-out cases, and compute the two revenue totals.
Concatenate all violations into one long frame with rule_code, and return it alongside the two revenue figures.

Expected result: A long frame of (rule_code, player_id, ledger_id, detail) plus two revenue totals, where the merged total exceeds the true total by the sum, over fan-out transactions, of (row_count - 1) times price_usd_net.

Verify it yourself

A synthetic ledger seeded with one deliberate break of each kind returns exactly the expected violations and nothing else.
Rule 1 produces zero violations on any group of length one.
The true revenue total is a function of txns alone and does not change when ledger content changes.
Re-running on identical input is deterministic, which requires the ledger_id tiebreak in the sort.

Follow-up

Rule 1 fires on 0.02 percent of rows, all on one currency, all inside one hour. Incident or backfill, and what distinguishes them in the data?
How do you run rule 1 incrementally each day without re-reading the full history, given that the check is defined over adjacent rows?
Rule 3 flags a transaction with three grant rows that turns out to be legitimate. What produces several ledger entries per purchase, and how do you whitelist it without blinding the rule?

Write a query to rank players by their total lifetime earnings across different game types.

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

How do you optimize a query that is performing slowly on a large dataset of match outcomes?

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Build a churn hazard curve by match ordinal with censoring

Hard · window functions · survival analysis · censoring

dim_player has player_id, install_ts, is_test_account. fct_match_participant has player_id, match_id, started_at, result, is_bot_opponent_match. fct_session has player_id, session_start_ts. Number each player's non-abandoned, non-bot matches by started_at. For ordinals 1 through 20, report the risk set and the hazard: among players whose nth match is at least 14 days old, the share who never started an (n+1)th match and have no session in the 14 days following that nth match. Players whose nth match is more recent than 14 days are censored at n.

Approach

Filter the match stream before numbering it. Onboarding bot matches and abandons otherwise occupy the first few ordinals for most players, which shifts every spike leftward and makes ordinal 1 mean different things for different install cohorts.
Assign the ordinal with ROW_NUMBER() OVER (PARTITION BY player_id ORDER BY started_at, match_id). The match_id tiebreak keeps numbering stable across reruns when two matches share a start timestamp.
Form the risk set at each n as players with match_ordinal = n whose started_at <= now() - interval '14 days'. This censoring filter is the whole exercise. Without it, every player's most recent match is classified as churn purely because the follow-up window has not elapsed, which inflates the hazard, worst at the ordinals where the mass of last-observed matches sits.
Define the event without a second scan of the match table by adding MAX(match_ordinal) OVER (PARTITION BY player_id) as final_ordinal: a player churned at n when n equals final_ordinal. Then add NOT EXISTS against fct_session for a session_start_ts in (nth_match.started_at, +14 days], so a player who kept opening the app but stopped playing matches is not counted as churned.
Aggregate to hazard = churned / risk_set per ordinal, report both columns rather than the ratio alone, and read the curve: expect ordinal 1 to dominate, then look for local maxima where a difficulty step or a progression gate is likely to sit.
Before proposing a cause for a spike, join the churning players at that ordinal to dim_player.current_level and check whether the spike coincides with a specific progression gate rather than with elapsed time.

Worked solution (45 min)

Build the filtered match CTE (non-test players, result <> 'abandon', is_bot_opponent_match = false) and add both ROW_NUMBER as match_ordinal and MAX(match_ordinal) OVER (PARTITION BY player_id) as final_ordinal.
Restrict to match_ordinal <= 20 and started_at <= now() - interval '14 days' to form the risk sets.
Mark the event: match_ordinal = final_ordinal AND NOT EXISTS a session in (started_at, started_at + interval '14 days'].
Group by match_ordinal and output risk_set = COUNT(*) and hazard = AVG(event::int).
Verify the risk set shrinks monotonically, then read the curve for local maxima before writing any interpretation.

Expected result: Twenty rows, ordinals 1 through 20, each with risk_set and hazard in [0,1]. risk_set decreases monotonically in ordinal. Hazard is highest at ordinal 1, since first-match drop-off dominates in this shape of data, then falls away with one or two local spikes.

Verify it yourself

risk_set(n+1) must be less than or equal to risk_set(n) * (1 - hazard(n)); a violation means the censoring filter is applied inconsistently across ordinals.
The churned counts summed over all ordinals must not exceed the number of distinct players appearing in any risk set, because a player can be an event at most once.
If hazard(1) is not the maximum, suspect that abandons or bot matches are still inside the ordinal and are absorbing the real first-match churn.

Follow-up

A spike sits at ordinal 8. Name two explanations the curve alone cannot distinguish, and the query that separates them.
How does the curve change if you index by calendar days since install instead of by matches played, and which framing do you defend to a designer?
Your hazard uses a 14-day inactivity window. What happens to the curve at 7 days and at 30, and how do you choose between them?

If the platform’s daily active user (DAU) metric drops by 10% overnight, what is your framework for metric drop diagnosis?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you balance the trade-off between increasing game difficulty and maintaining player retention?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Design a system to detect collusion or cheating in a competitive mobile game.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Describe how you would handle missing data or null values when aggregating user-level metrics.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How do you determine the required sample size for an A/B test to detect a 1% change in conversion?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Explain the concept of statistical significance to a non-technical product manager.

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

Trade queue time against match quality with an explicit exchange rate

Medium · guardrails · order statistics · selection bias · trade-offs

Matchmaking will widen its skill-rating tolerance in apac ranked to cut queue times. You have fct_match_participant (queued_at, started_at, matchmaking_wait_seconds, game_mode, server_region, result, disconnected, skill_rating_before, skill_rating_after). Define the primary metric and the guardrail, then state the exchange rate you would pre-register: how many seconds of p95 wait reduction justify one percentage point of abandon rate. Explain why p95 matchmaking wait is measured only over queue entries that became matches, and what that conditioning does when the change alters which entries form a match.

Approach

Refuse the framing in which wait time is the goal. The decision metric is daily core-loop players in apac ranked. Wait time and match quality are both inputs to it, and the change is worth shipping only if the net moves.
Make the operational primary p95 matchmaking_wait_seconds by server_region and game_mode over a trailing 24 hours, and pair it immediately with the queue-abandonment rate it excludes: entries that never produced a started_at. If queue entries without a formed match are not logged anywhere, say so out loud, because the primary metric is then unfalsifiable and the first deliverable is the instrumentation rather than the test.
Explain the conditioning precisely. p95 is an order statistic over queue entries that succeeded, so widening tolerance changes the composition of that population, not only its level. A configuration that lets the easy-to-match players through faster while the hardest ones time out can improve p95 and serve fewer people.
Guardrail: match abandon rate, counting rows with result = 'abandon' or disconnected = true over all rows with started_at in the window, cut by region and mode, plus a match-quality measure such as the within-match standard deviation of skill_rating_before aggregated to a daily median.
Derive the exchange rate rather than asserting it. Estimate from historical variation how much one point of abandon rate costs in next-day core-loop players, and how much one second of p95 costs in queue abandonment, then convert both into the same unit and state the threshold before the readout. A defensible answer is a number with its derivation attached.
Handle interference. Queue population is shared, so an individually randomised arm changes the pool the control arm matches into. Randomise by region and mode cluster, or switchback on time slices with a burn-in long enough to clear carryover, and compute standard errors at the slice level.

Worked solution (30 min)

Write the decision metric and the two operational metrics with their windows and cuts.
Check whether unmatched queue entries exist in the data. If fct_match_participant carries rows only for matches that formed, name the missing event and treat its absence as a blocker rather than a caveat.
Define the match-quality guardrail concretely, for example the standard deviation of skill_rating_before within a match, aggregated to a daily median by region and mode.
Derive the exchange rate: cost of one abandon-rate point in next-day core-loop players, cost of one second of p95 in queue abandonment, both from historical variation, then fix the threshold.
Choose the randomisation unit, region-mode cluster or time slice, and state the variance cost you are accepting.

Expected result: A decision metric of daily core-loop players in the affected region and mode; p95 matchmaking wait paired with queue-abandonment rate as the primary; match abandon rate and an intra-match skill-spread guardrail; a pre-registered exchange rate with its derivation; and cluster or switchback randomisation because the queue pool is shared across arms.

Verify it yourself

Your primary is stated as a pair, because an order statistic over successes alone can be improved by producing fewer successes.
Your exchange rate has units on both sides and a derivation, not just a number.
Your design does not randomise individual players into different matchmaking rules inside the same queue.

Follow-up

p95 wait falls 22 seconds, abandon rate rises 0.4 points, and daily core-loop players in apac are flat. Ship or not?
Your exchange rate came from an observational fit of abandon rate against next-day play. What confounds that fit, and how far should it move your threshold?
How long a burn-in would you set on the switchback, and what evidence sets that length?

Net ARPDAU fell nine percent while net revenue stayed flat

Easy · decomposition · denominator · acquisition mix

Net ARPDAU over a trailing 28 days fell from $0.152 to $0.138 week over week, a 9.2% drop, while summed price_usd_net for completed transactions is flat to within 1%. Finance wants to know whether monetisation weakened. You have dim_player (player_id, install_ts, install_country, acquisition_channel, is_test_account), fct_session (player_id, session_start_ts) and fct_iap_transaction (player_id, purchase_ts, price_usd_net, status). Deliver a one-paragraph verdict and the query behind it.

Approach

Before touching segments, recompute both sides of the ratio separately for each of the two weeks, using the metric definition and nothing more: numerator is summed price_usd_net over rows with status='completed', denominator is the count of distinct (player_id, date(session_start_ts)) pairs. status is the row's current state, so a transaction that has since flipped to refunded or chargeback already fails status='completed' and has already left the numerator; subtracting refund value on top of that filter removes it a second time and breaks the numerator-times-denominator rebuild below. A ratio can only move through its numerator or its denominator, and confirming which one moved costs one query and eliminates half the hypothesis space.
Quantify the implied denominator move: if the numerator is flat and the ratio fell by a factor of 0.152/0.138, player-days must have risen by about 10.1%. Check that the observed player-day count matches that figure; if it does not, the numerator is not as flat as reported, and the first suspects are mechanical: a different status filter, a different UTC day boundary between fct_session and fct_iap_transaction, or a window that is not the same length on both sides.
State the one refund caveat that does survive, as a bound rather than a correction. This table exposes only the current status, with no status_changed_at or refund_ts, so both weeks are read as of query time and the older week has had a week longer for purchases to flip to refunded. That biases the recent week's numerator upward relative to the older one, which pushes ARPDAU up, the opposite direction from the observed fall; it therefore cannot manufacture the 9.2%, though it can hide a small real numerator decline. Say so and move on.
Cut the new player-days by install cohort age, acquisition_channel and install_country. Look specifically for players whose install_ts falls inside the window, since an install spike enters the denominator immediately but cannot contribute purchase volume at the same rate.
Compute net revenue per player-day restricted to players who were already active before the window opened. If that figure is flat, monetisation of the existing population did not change and the whole move is denominator composition.
Confirm is_test_account=false is applied on both weeks and that no new automated or QA population entered fct_session, since test traffic creates player-days with structurally zero revenue.
Report the verdict as a decomposition, not a direction: state the player-day growth, the channel and country it came from, and the unchanged per-player-day revenue of the pre-existing base.

Follow-up

The campaign driving the new installs will run for another six weeks. What do you forecast ARPDAU does, and what would you monitor to distinguish a healthy dilution from a bad one?
Which metric would you put beside ARPDAU on the same chart so this question does not get asked again next month?
How would your answer change if the new installs were concentrated in one install_country with a much lower local price point?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

How do you maintain high quality and rigor in your work when the business is pushing for speed?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Disagree with a queue-tolerance change using guardrail data

Medium · guardrails · conditioning on success · switchback · disagreement

A product manager proposes widening matchmaking skill tolerance in one region because p95 matchmaking wait there is 71 seconds against 34 seconds elsewhere. You have fct_match_participant with matchmaking_wait_seconds, skill_rating_before, result, disconnected, server_region and game_mode. Your concern is that the wait percentile is computed only over matches that actually formed, and that wider tolerance raises mismatch and abandon rate. The PM has a deadline and reads your concern as blocking. Make the disagreement in a way that keeps the decision moving, and state the evidence that would change your mind.

Approach

Agree on the problem in the first sentence. The wait gap is real and worth fixing; the disagreement is about the price of this particular fix, which is a far easier conversation than one that sounds like a veto.
Name the measurement defect concretely rather than generally. p95 matchmaking wait is an order statistic over matches that formed, so players who gave up in queue are absent from it entirely. A queue that fails outright reports an excellent p95. Pair it with a queue-abandonment rate measured on queue entries, and with match abandon rate defined over result = 'abandon' or disconnected = true.
Bring a quantity instead of an objection. From current data, bucket matches by absolute skill gap between teams using skill_rating_before and plot abandon rate and disconnect rate per bucket, cut by game_mode and server_region. That converts 'mismatch is bad' into a rate per unit of tolerance, which the PM can trade against seconds of wait.
Propose the design that settles it, and explain why the obvious one is invalid: queue population is shared, so a treated player is matched against a control player and the treatment leaks across arms. Randomise by region and mode cluster, or switchback on time slices with a burn-in long enough for the queue to clear, and analyse at the randomisation unit.
State your falsifier explicitly, for example that if abandon rate is flat across the top two skill-gap buckets in this region, you withdraw the objection.
Give the PM something shippable inside the deadline, such as a bounded tolerance increase in the worst hours only, with the guardrail thresholds pre-registered before launch.

Follow-up

How long a burn-in would you use for a switchback here, and what in the data tells you it is long enough?
Queue-abandonment is not in the tables listed. What would you instrument, and what would you do for this decision in the meantime?
The PM ships it globally without a test. What do you measure afterwards, and what can that measurement honestly support?

Turn 'engagement is down' into a scoped answerable brief

Easy · scoping · release calendar · triage · metric definition

A studio lead messages: engagement is down, can you look into it. You have dim_player, fct_session, fct_match_participant and the release calendar. Daily core-loop players, defined as distinct players per UTC day with at least one match reaching a terminal result other than abandon, is down 6% week over week, and a content release landed nine days ago. You have 30 minutes before a standup. Produce the scoping questions you would ask, the first three cuts you would run, and the one-paragraph brief you send back before doing deeper work.

Approach

Pin the metric and the comparison before touching data. Ask which number the lead actually saw and over what window, because a week-over-week read nine days after a release is measuring post-release decay by construction, and that alone may be the whole answer.
Ask the decision question, not more metric questions: what would the lead do differently if this turns out to be new players versus returning, one platform versus all, one region versus global. Scope follows the decision, and an investigation with no decision attached should be declined or deferred.
Run three cheap cuts that split the space rather than confirm a hunch. First, new versus existing by install cohort age, which separates an activation problem from a retention problem. Second, platform crossed with app_version, where a bad build shows as a concentration of fct_session.ended_reason = 'crash' and truncated duration_seconds. Third, server_region, where an infrastructure incident shows as elevated match abandon rate and p95 matchmaking wait rather than as fewer app opens.
Re-baseline against the matched day in the previous release cycle instead of against last week, so the comparison is not dominated by the release calendar.
Send a brief that states what you confirmed, what you ruled out, the current best explanation with its confidence, and the size of the next block of work with the question it would close.

Follow-up

The crash concentration is on one device_model at one app_version. What do you send, to whom, and how urgently?
How would you tell a genuine drop apart from an instrumentation change that altered which sessions get logged?
If all three cuts come back flat, what is your fourth cut and why that one?