Interview guides · Niantic

Niantic Data Scientist Interview Questions & Guide 2026

SQL practice for Niantic data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

As a Data Scientist at Niantic, you play a crucial role in shaping the future of augmented reality and mobile gaming. Your work directly impacts how users interact with Niantic's innovative products, such as Pokémon GO and Ingress, by leveraging data to enhance user experiences, inform product decisions, and drive business strategies. This position is not just about crunching numbers; it requires a unique blend of technical acumen, creativity, and an understanding of user behavior in a dynamic environment.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

Compute expected cost of pity-timer tables
Diagnose churn by progression stage, not calendar day
Price cosmetics without cannibalising existing sinks

Overview

The contributions made by Data Scientists at Niantic are vital. You will be involved in analyzing large datasets, developing machine learning models, and translating complex data findings into actionable insights. Collaborating closely with cross-functional teams, your insights will help refine game mechanics, improve user engagement, and optimize marketing strategies. This role offers the opportunity to work on exciting projects that require both analytical skills and a passion for gaming and technology.

The Niantic interview process

Niantic candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Take-Home Assignment

Your submission is read asynchronously by someone who cannot ask you a clarifying question, and who will usually skim it before reading it properly. That changes what good looks like. The conclusion belongs near the top with the supporting analysis beneath it, and the code should run start to finish on a clean machine without a manual step you forgot to document. A reviewer forced to reconstruct your reasoning from the order of notebook cells is already discounting the work. The submissions that land are the ones where a busy reader gets the answer immediately and can verify it if they want to.

What to demonstrate

Whether the answer arrives before the methodology, so a reader who stops after the first page still has the recommendation
Whether the code runs end to end from the submitted files, with dependencies and data paths declared rather than assumed
Whether each chart is legible on its own, carrying axis labels and units, and exists to support a claim made in the text

How to prepare

Restructure a past analysis so the opening paragraph holds the recommendation and the number behind it, then check that nothing later in the document quietly contradicts it
Copy your own submission into an empty directory, run it in a clean environment, and fix everything that breaks; hidden local state is caught here or by the reviewer
For every chart, write the one sentence it is meant to prove, and delete the chart if you cannot write that sentence

Phone Interview

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Onsite Interview

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Booking revenue at gross, on purchase day, for cohorts of different ages.

Revenue matures: the platform fee is known immediately, but refunds and chargebacks land over weeks, and later purchases keep accruing to the same cohort. Comparing a two-week-old cohort with a one-year-old cohort therefore compares two different maturities, and any trend chart built that way slopes in the direction of cohort age. Quote cohort revenue at a fixed age, and exclude cohorts that have not reached it rather than extrapolating them.

Testing revenue with a difference in means on a heavy-tailed spend distribution.

Player spend is approximately log-normal with a long Pareto-like tail, so the variance of the sample mean is dominated by a few accounts and the central limit approximation converges slowly at realistic sample sizes. A t-test on raw ARPDAU can flip sign when one account buys a large currency pack. Winsorising or capping at a pre-registered percentile, or testing conversion and conditional spend separately, fixes the variance problem, but note the cost: capping biases toward zero exactly when the true effect lives in the tail, so state the cap before you look.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Optimising accuracy on a heavily imbalanced target

State the base rate first, then choose the metric from the relative cost of a false positive against a false negative: precision and recall at the operating threshold, PR-AUC, or expected cost. At a 1 percent positive rate, predicting the majority class for everyone scores 99 percent accuracy and is worthless.

Questions & practice

Given a dataset, how would you implement a classification algorithm in Python?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

What machine learning algorithms would you consider for a user segmentation project?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Sink to faucet ratio per currency from a signed ledger

Easy · ledger accounting · groupby · metric definition

You are given ledger, a pandas DataFrame with one row per signed currency movement: ledger_id, player_id, occurred_at (UTC timestamp), currency_code, delta_amount (signed int64), flow_type in {faucet, sink, transfer, correction}, source_system, is_reversal (bool). Implement sink_to_faucet(ledger) returning one row per (currency_code, week_start) with faucet_units, sink_units and ratio = sink_units / faucet_units. Sink units are reported as positive magnitudes. Rows with flow_type in {transfer, correction} and rows with is_reversal true are excluded from both sides. A currency-week with zero faucet volume must return a null ratio rather than raise or return infinity.

Approach

Filter first, aggregate second: build one boolean mask for flow_type in ('faucet','sink') and is_reversal == False, apply it once, so the exclusion cannot be applied to one side of the ratio and forgotten on the other.
Derive week_start by flooring occurred_at to the Monday of its ISO week: occurred_at.dt.normalize() minus pd.to_timedelta(occurred_at.dt.weekday, unit='D'). Avoid dt.isocalendar().week on its own, which drops the year and collapses week 1 of two different years into one bucket.
Split the aggregate by sign rather than by abs(): faucet_units = sum of delta_amount where flow_type == 'faucet', sink_units = sum of -delta_amount where flow_type == 'sink'. Taking abs() of the whole column hides a faucet row carrying a negative delta, which is a real data bug you want surfaced.
Pivot to one row per (currency_code, week_start) and divide under a guard so faucet_units == 0 yields NaN, not inf; downstream charts silently drop NaN and silently rescale on inf.
Run the same aggregate once with the is_reversal filter removed and diff the two results. Any currency whose ratio moves materially is telling you that clawbacks or customer-service corrections are being read as genuine economy flow.

Worked solution (20 min)

Apply the single inclusion mask and assert that the filtered frame contains only the two flow types you intend.
Add the week_start column by weekday subtraction and confirm every value is a Monday at 00:00 UTC.
Aggregate faucet and sink magnitudes separately with a groupby on (currency_code, week_start), then join the two results on that key with an outer join so a currency-week with sinks but no faucets survives.
Divide under a where() guard and return the frame sorted by currency_code then week_start.

Expected result: A frame keyed by (currency_code, week_start) carrying faucet_units, sink_units and ratio, where ratio is NaN rather than inf for any week with zero faucet volume, and rows never disappear because one side of the ratio was empty.

Verify it yourself

Recompute one currency-week by hand from a filtered slice and confirm both aggregates match to the unit.
The row count equals the number of (currency_code, week_start) pairs present after filtering, with no week silently dropped by an inner join.
Running with and without the reversal exclusion changes the answer only for currencies that actually carry is_reversal rows.

Follow-up

The definition drops the reversal row but leaves the original entry it undid inside the aggregate. Defend that choice or change it, and say which direction the ratio moves either way.
The soft currency has held a ratio near 0.8 for six weeks. What do you look at next, and what evidence would let you say the economy is nevertheless fine?
How would you report this when a single live-ops event contributed 60 percent of the week's faucet volume?

Write a function to calculate the average of a list of numbers.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Solve a problem involving data manipulation using SQL.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Report per-SKU net revenue alongside granted premium currency

Medium · fan-out · pre-aggregation · revenue

fct_iap_transaction has transaction_id, player_id, sku_id, purchase_ts, price_usd_net, status. fct_currency_ledger has transaction_id (nullable), currency_code, delta_amount, source_system. A single completed purchase can produce several ledger rows with source_system = 'iap_grant': the base pack, a first-purchase bonus and an event bonus are separate entries. For last month's completed transactions, return per sku_id the transaction count, net USD revenue and total premium currency granted. Write it so the revenue figure is not inflated by the grant rows.

Approach

State the grain of each side before writing any join. fct_iap_transaction is one row per transaction; the iap_grant rows are one-to-many against it. Joining them directly produces a result at grant grain, where any SUM of price_usd_net counts each purchase once per grant row.
Pre-aggregate the ledger to transaction grain in a CTE: SUM(delta_amount) FILTER (WHERE currency_code = 'premium') grouped by transaction_id, restricted to source_system = 'iap_grant'. The CTE is now one row per transaction and the join becomes many-to-one.
Left join that CTE onto the filtered transactions so a completed purchase that produced no grant row still appears with a NULL grant total. That combination is a fulfilment bug worth surfacing, not hiding behind an inner join.
Filter transactions on status = 'completed' and purchase_ts inside last month. Because this table mutates in place, a transaction later refunded now carries status = 'refunded' and is already excluded; say that out loud rather than adding a redundant refund subtraction that would double-count the adjustment.
Aggregate to sku_id and add a count of transactions with no matching grant, so the fulfilment gap is legible in the same output instead of needing a second query.

Worked solution (25 min)

Count transactions in the window, then count the iap_grant ledger rows referencing them. That ratio is how badly a naive join would inflate revenue.
Write the grant CTE grouped by transaction_id and verify its row count is at most the transaction count.
Join, filter on status and purchase_ts, and aggregate by sku_id.
Cross-check total net revenue against a standalone SUM over fct_iap_transaction with the same filters and no join at all. The two must be identical.

Expected result: One row per sku_id with transactions (count of completed purchases), net_revenue_usd (sum of price_usd_net, matching the join-free total exactly), premium_granted (sum of premium delta_amount, NULL or zero where the SKU grants no premium currency) and ungranted_transactions.

Verify it yourself

net_revenue_usd summed across all rows must equal a plain SUM(price_usd_net) over the same filtered transactions with no join. Any excess is fan-out.
The grant CTE's row count must be less than or equal to the number of distinct transaction_ids appearing in the ledger window.
SKUs that are not currency packs, such as directly purchased cosmetics, should show premium_granted of NULL or zero rather than a positive figure.

Follow-up

How would you prove the fan-out was really happening, using only row counts and no prior knowledge of the data?
Last month's transactions have not finished refunding. At what cohort age would you publish this table, and what do you show a stakeholder in the meantime?
A currency_pack SKU shows net revenue but zero granted premium currency. Walk me through the next three queries you run.

If given a dataset of player behaviors, how would you identify patterns that inform product development?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize your tasks when faced with multiple deadlines?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you analyze user engagement data to improve a game feature?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Explain how you would approach a problem where a game feature is not performing as expected.

Medium

Approach

Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Estimate a price-tier change without a randomised holdout

Hard · difference-in-differences · synthetic control · clustered inference

A currency-pack price tier rose roughly 9% in six countries on one date. Store rules make two simultaneous prices in one country impossible, so there is no holdout. You have fct_iap_transaction (purchase_ts, price_local, currency_iso, price_usd_gross, price_usd_net, status, refund_ts), dim_player.install_country, 40 untreated countries and two years of history. Deliver an estimator for the effect on net revenue per active player-day, the identifying assumption it rests on, an inference method that is valid with six treated clusters, and the two threats you would test first.

Approach

Build the panel at country-week level with net revenue per active player-day as the outcome, then estimate either difference-in-differences with country and week fixed effects, or a synthetic control that reweights untreated countries to reproduce each treated country's pre-period path. With only six treated units, run synthetic control per treated country and aggregate, because a parallel-trends claim over a small, non-randomly chosen treated set is weak on its own.
State the identifying assumption exactly: absent the price change, treated and control countries would have followed parallel paths in the outcome. Test it with an event-study specification, leads and lags around the change date, and require the pre-period leads to be jointly indistinguishable from zero. Passing that test is evidence for the assumption, not proof of it, and say so.
Get the inference right, because this is where the design usually fails. Cluster-robust standard errors are unreliable below roughly 40 clusters and badly so at six. Use a wild cluster bootstrap over countries, or synthetic control's placebo-in-space distribution, ranking the treated post-to-pre RMSPE ratio against the same statistic computed for every untreated country. With 40 controls the smallest attainable one-sided p-value is 1/41 = 0.024, so an underpowered design cannot be rescued by the test.
Test threat one, FX. price_usd_gross is converted at the purchase date, so a currency move in a treated country is observationally identical to a price effect measured in USD. Re-run the outcome in local currency and again with the FX rate pinned to the pre-period average, and require the sign to survive both.
Test threat two, maturity and calendar confounding. Refunds and chargebacks land over about 60 days, so hold the outcome at a fixed maturity and drop the most recent 60 days rather than annualising them. Then check the release and live-ops calendar: a global patch or event in the same week is only absorbed by the design if it hit treated and control countries equally, which is a claim to verify, not to assume.
Decompose before recommending: units sold, paying conversion, and net revenue per payer. A 9% rise that holds revenue flat by losing units is a different decision from one that raises both, and the aggregate number hides which happened.

Worked solution (40 min)

Aggregate to country-week net revenue per active player-day, restricted to purchases that have reached 60-day maturity.
Fit the event study with leads and lags around the change date and test the pre-period leads jointly.
Estimate the DiD with country and week fixed effects, and separately fit a synthetic control per treated country against the 40 donors.
Compute inference two ways: wild cluster bootstrap over the six treated clusters, and the placebo-in-space rank test on post-to-pre RMSPE ratios.
Re-run the outcome in local currency and at a pinned pre-period FX rate.
Decompose into units, conversion and revenue per payer, and write the recommendation around the decomposition.

Expected result: A DiD or synthetic-control ATT on country-week net revenue per active player-day, identified by parallel counterfactual trends, with inference from a wild cluster bootstrap or a placebo-in-space rank test whose floor is p = 0.024 at 40 donors, reported only after FX and refund maturity have been handled.

Verify it yourself

Event-study leads for the eight weeks before the change should be jointly indistinguishable from zero; a visible pre-trend means the design is not identified and the estimate should not be quoted.
Each treated country's synthetic control should have a small pre-period RMSPE relative to its post-period gap; a poor pre-period fit means that country is reported separately or dropped, decided on fit alone.
The sign of the effect should survive re-running the outcome in local currency and at a pinned FX rate.
Placebo change dates six months earlier should produce effects centred on zero.

Follow-up

Two of the six countries also received a targeted offer in the same month. What do you do with them, and how do you decide without looking at their outcomes first?
Leadership wants one number for price elasticity. What estimand did you actually identify, and for which population?
What would an encouragement design or an instrument buy you here, and what exactly would the instrument have to satisfy for it to be credible?

Abandon rate spiked with a regional incident and a flag change

Hard · compound causes · logging change · segment attribution

Match abandon rate, defined as rows with result = 'abandon' or disconnected = true over all rows with started_at in the window, rose from 4.1% to 7.8% in a week. Two things happened: a two-day server incident in one region, and a client release on one platform. fct_match_participant carries match_id, player_id, started_at, result, disconnected, server_region, game_mode; fct_session carries player_id, session_start_ts, platform, app_version. Attribute the 3.7-point rise across both causes and a residual.

Approach

Split the numerator into its two disjoint terms before any segmentation: rows with result='abandon', and rows with disconnected=true and result <> 'abandon'. The metric is an OR of two signals that can move independently, and a change in flag semantics moves only one of them while a genuine outage moves both.
Isolate the regional incident with a two-dimensional cut of server_region by day. A real incident is bounded in both dimensions: it appears in one region, on the days the incident ran, and disappears afterwards. Compute its contribution as the region's share of weekly matches times its abandon-rate delta times the fraction of the week affected.
Isolate the client change by joining participants to sessions on player and time to attach platform and app_version, then cutting by version. A logging or semantics change produces a step at the version boundary and a weekly aggregate that tracks the version adoption curve, not the calendar. Compute its contribution as the platform's share of matches times the new version's share of that platform's matches times the rate delta on the new version.
Discriminate the flag change from a real disconnect problem using corroborating signals that a semantics change would not move: fct_session ended_reason='crash' rate on the new version, matchmaking wait, and the result='abandon' term on its own. If only the disconnected flag moved, the flag's meaning changed.
Add the two contributions and report the residual explicitly. A residual that is small relative to the two named terms is the honest end of the analysis; a large one means a third cause is present and should be named as unexplained rather than absorbed into either story.
Recommend the metric repair: report the result='abandon' term separately so the series stays comparable across client versions, and either backfill a version-consistent disconnected definition or mark the discontinuity on every chart built on it.

Follow-up

The two causes overlapped in time. How do you keep them from stealing each other's attribution, and what does the interaction term look like?
If the residual were 1.5 points instead of 0.3, what would you look at next?
How do you decide whether to restate the historical series or to annotate the break, and who needs to agree?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

Repeat the depth protocol on the second-ranked area with the same six-problem structure.
Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

Re-run the six prompts from day one under the same clock and compare both correctness and time.
Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

Can you walk us through your experience with SQL and how you have used it in past projects?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Report a contaminated switchback when the ship call is Friday

Hard · switchback · contamination · carryover · decision under uncertainty

A switchback test of a container drop-rate change ran Monday to Sunday on 12-hour slices, randomising the whole region-mode population per slice. On Wednesday a limited-time event went live to everyone, changing both the currency faucet and store traffic for the remaining slices. Treatment and control slices are no longer exchangeable. The change is scheduled to ship Friday and the director wants a number. Give what you say, what you can still salvage from the data, and what you ask for.

Approach

Say the full-week estimate is unusable before saying anything else, and give the reason in one sentence a non-analyst can repeat: the world changed halfway through, so the two halves of the test measure different games and pooling them averages across that difference.
Test salvageability rather than assuming it away. If slice assignment is balanced within the pre-event window and that window holds enough slices for a usable standard error, you have a shorter and noisier but uncontaminated estimate. Report it with the widened interval and an explicit statement of the power you lost. Do not pool across the event boundary, and do not run the post-event slices as a second test with the event as a covariate, because the event was not randomised.
Check carryover independently of contamination, because they fail for different reasons. A drop-rate change leaves players holding containers and currency, so treatment effects persist into the following control slice unless the burn-in exceeds the time it takes a typical player to open what they received. State the burn-in you used and what evidence supports it.
Give the decision a path rather than a blocker. Offer a cluster-randomised staged rollout by region and mode with the guardrail set pre-registered before launch, which gets the change shipped on Friday while still producing a clean estimate.
Fix the cause, not the instance. The live-ops calendar was a discovery here and should be a design input, checked against the planned slice schedule before a test is launched.
Decide in advance what you will do if the director insists on a single number anyway, and prefer publishing the clean partial estimate with its interval over publishing the contaminated one with a caveat.

Follow-up

The pre-event window has six slices. What standard error does that give you, and is it worth reporting at all?
How would you choose the burn-in length for the rerun, using data rather than judgement?
The director ships it anyway on Friday. What do you measure afterwards, and what can that measurement honestly support?

Disagree with a queue-tolerance change using guardrail data

Medium · guardrails · conditioning on success · switchback · disagreement

A product manager proposes widening matchmaking skill tolerance in one region because p95 matchmaking wait there is 71 seconds against 34 seconds elsewhere. You have fct_match_participant with matchmaking_wait_seconds, skill_rating_before, result, disconnected, server_region and game_mode. Your concern is that the wait percentile is computed only over matches that actually formed, and that wider tolerance raises mismatch and abandon rate. The PM has a deadline and reads your concern as blocking. Make the disagreement in a way that keeps the decision moving, and state the evidence that would change your mind.

Approach

Agree on the problem in the first sentence. The wait gap is real and worth fixing; the disagreement is about the price of this particular fix, which is a far easier conversation than one that sounds like a veto.
Name the measurement defect concretely rather than generally. p95 matchmaking wait is an order statistic over matches that formed, so players who gave up in queue are absent from it entirely. A queue that fails outright reports an excellent p95. Pair it with a queue-abandonment rate measured on queue entries, and with match abandon rate defined over result = 'abandon' or disconnected = true.
Bring a quantity instead of an objection. From current data, bucket matches by absolute skill gap between teams using skill_rating_before and plot abandon rate and disconnect rate per bucket, cut by game_mode and server_region. That converts 'mismatch is bad' into a rate per unit of tolerance, which the PM can trade against seconds of wait.
Propose the design that settles it, and explain why the obvious one is invalid: queue population is shared, so a treated player is matched against a control player and the treatment leaks across arms. Randomise by region and mode cluster, or switchback on time slices with a burn-in long enough for the queue to clear, and analyse at the randomisation unit.
State your falsifier explicitly, for example that if abandon rate is flat across the top two skill-gap buckets in this region, you withdraw the objection.
Give the PM something shippable inside the deadline, such as a bounded tolerance increase in the worst hours only, with the guardrail thresholds pre-registered before launch.

Follow-up

How long a burn-in would you use for a switchback here, and what in the data tells you it is long enough?
Queue-abandonment is not in the tables listed. What would you instrument, and what would you do for this decision in the meantime?
The PM ships it globally without a test. What do you measure afterwards, and what can that measurement honestly support?