Interview guides · DraftKings

DraftKings Data Scientist Interview Questions & Guide 2026

Modeling drills for a DraftKings data scientist interview: baselines, regression interpretation, class imbalance, model defense.

The 60-second brief

A Data Scientist at DraftKings plays a pivotal role in driving the analytical and predictive engine of one of the world’s leading digital sports entertainment and gaming companies. Operating at the intersection of technology, sports, and finance, the data science team is responsible for building high-scale predictive models that power live betting markets, optimize user acquisition, detect fraudulent behavior, and personalize the player experience. Your work directly impacts millions of active users and influences real-time business decisions where even a fraction of a percent in model accuracy can translate to millions of dollars in bottom-line revenue.

Nearly every loop contains a round whose deliverable is a recommendation to someone non-technical. Practise stating a conclusion, the confidence attached to it, and the cost of being wrong in each direction, because that triple is the artifact being graded.

Focus areas

Design matchmaking tests that survive interference
Compute expected cost of pity-timer tables
Price cosmetics without cannibalising existing sinks

Overview

At DraftKings, you will not work in an isolated research environment. Instead, you will collaborate closely with product managers, software engineers, and trading operations teams to deploy machine learning models directly into production systems. Whether you are optimizing customer lifetime value (LTV) models, designing real-time recommendation systems, or developing sophisticated pricing algorithms for live sports events, your contributions will have immediate, measurable business impact.

The environment is fast-paced, intellectually demanding, and deeply quantitative. Candidates who thrive here are those who possess not only exceptional mathematical and statistical foundations but also the business acumen to translate complex algorithmic outputs into actionable strategic decisions.

The DraftKings interview process

DraftKings candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Call

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Screening

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Panel Loop

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Treating the account as the person, and the calendar week as neutral time.

Reinstalls, multi-platform play and shared accounts split or merge humans relative to player_id, which inflates install counts and deflates retention denominators in ways that correlate with platform and campaign. Meanwhile daily actives spike on release day and decay over the following week, so a week-over-week read aliases the content calendar. Anchor comparisons to time since release or to matched days in the release cycle, and report identity-sensitive metrics with an explicit statement of which identity key was used.

Reading economy health from average balances.

A faucet change can leave the mean balance flat while the top decile accumulates a stock it has no reason to spend and the median player stays starved, which is the state in which a premium shortcut stops selling. The diagnostic pair is the sink-to-faucet ratio per currency and the full balance percentile curve over time, not the mean. Ledger reversals and customer-service grants must be excluded from both sides or a single large correction will look like a genuine faucet.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Crediting a treatment for regression to the mean

Selecting a group because it is extreme (lowest-engagement users, accounts having their worst month, the bottom decile of a score) moves that group's expected next-period value back toward the average even under no treatment, by exactly as much as the selecting measure is imperfectly correlated with its own later value. Compare against units that met the same selection rule and went untreated, or use two pre-periods so the bounce-back is visible before the intervention starts. A pre-post number on a group chosen for being extreme measures the selection rule, not the treatment.

Questions & practice

Explain the mathematical difference between L1 (Lasso) and L2 (Ridge) regularization, and describe a scenario where you would favor one over the other.

Medium · statistics and probability

Approach

Say what the estimate is of, and over what population it generalises.
Write down the assumption the method needs before you use the method.
Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

How would you design a machine learning model to predict the probability of a specific player scoring a touchdown in an ongoing NFL game?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Rebuild running balances and find the first ledger divergence

Easy · running balance · reconciliation · sort stability

Using the same ledger frame plus balance_after (int64), rebuild each player's running balance per currency_code by accumulating delta_amount in (occurred_at, ledger_id) order, then report the first row per (player_id, currency_code) where your computed balance disagrees with balance_after. You may not use cumsum, groupby().cumsum() or expanding(); accumulate explicitly. Assume each group starts from a balance of zero. Reversals and corrections do count toward the balance. Return player_id, currency_code, ledger_id, computed_balance, balance_after and diff, sorted by absolute diff descending.

Approach

Sort once on (player_id, currency_code, occurred_at, ledger_id). The ledger_id tiebreak is load-bearing: entries routinely share occurred_at to the second, and an unstable order manufactures divergences that move between runs.
Accumulate in one pass over itertuples with a dict keyed by (player_id, currency_code), resetting to zero when the key changes. One pass is O(n) and beats any reshape; the ban on cumsum is about showing the accumulation, not about speed.
Record the first index per key where computed != balance_after and then stop tracking that key. After a break every later row inherits the same offset, so reporting all of them buries the single row that matters.
Keep reversals and corrections in the accumulation. They are genuine balance movements, which is the exact opposite of the rule for flow aggregates, and mixing the two rules up is the common error in this domain.
Sort the output by abs(diff) descending. A one-unit diff is usually a rounding artefact somewhere upstream; a six-figure diff is a lost grant, and the ordering puts the second kind on the first screen.

Worked solution (25 min)

Sort on the four-column key and reset the index so positional recording is unambiguous.
Walk the frame once, carrying a dict of running balances and a set of keys already flagged, appending a record the first time a key disagrees.
Assemble the flagged records into a frame, compute diff = computed_balance - balance_after, and sort by its absolute value descending.
Spot-check by re-accumulating one flagged group by hand from its first row and confirming the break lands on the reported ledger_id.

Expected result: At most one row per (player_id, currency_code), each identifying the earliest ledger_id at which the accumulated balance and the stored balance_after part company, with diff carrying the signed size of the break.

Verify it yourself

A group with no break contributes zero rows, not a row with diff 0.
Re-running on a shuffled copy of the input reproduces the identical output, which is what the ledger_id tiebreak buys you.
Injecting a single corrupted balance_after into a clean group produces exactly one flagged row, at that row's ledger_id.
The sum of delta_amount up to the flagged row, computed independently, equals your computed_balance.

Follow-up

Your check assumes each group starts at zero. How would you distinguish a player whose early history was trimmed by a retention policy from one who genuinely started at zero?
Divergence appears on exactly one currency and only after a specific date. What is your next query?
How would you run this as a daily monitor over hundreds of millions of rows without rebuilding every balance from the beginning of time?

Write a SQL query to calculate the rolling 7-day active user count and average spend per player from a transaction ledger.

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Find cosmetic items with no recorded spend in ninety days

Medium · anti-join · null semantics · catalog

dim_catalog_item has item_id, config_version, item_class, rarity, available_from, is_current. fct_currency_ledger has item_id (nullable, and most rows are match rewards and daily logins carrying no item), flow_type, occurred_at, is_reversal. Return every current cosmetic that became available more than 90 days ago and has no sink ledger row referencing it in the last 90 days, with its rarity and available_from. Your first draft will probably return zero rows on this data. Explain exactly why, then fix it.

Approach

Build the candidate set from dim_catalog_item where is_current = true, item_class = 'cosmetic' and available_from < now() - interval '90 days'. Confirm is_current really yields one row per item_id, because the primary key is (item_id, config_version) and two current versions of one item would duplicate the output.
Build the spend set separately: ledger rows with flow_type = 'sink', occurred_at in the last 90 days, is_reversal = false.
Diagnose the empty result. Most ledger rows carry a NULL item_id, and x NOT IN (a, b, NULL) expands to x <> a AND x <> b AND x <> NULL, whose last conjunct is NULL. The predicate can therefore be FALSE or NULL but never TRUE, so the query returns nothing no matter what the data says.
Fix it and say which form you prefer: NOT EXISTS with a correlated predicate, which is NULL-safe by construction and usually plans as an anti-join; a LEFT JOIN with WHERE spend.item_id IS NULL; or NOT IN with an explicit WHERE item_id IS NOT NULL inside the subquery.
Sense-check the output against the catalog. If almost every cosmetic appears, the sinks are probably recorded against a bundle or container item_id rather than the granted item, and the question needs a different join path through drop_table_id.

Worked solution (25 min)

Count ledger rows with item_id IS NULL over the window first. That number is the whole explanation for the empty result and is worth having before you start debugging.
Write the candidate CTE and confirm its row count equals the count of distinct current cosmetic item_ids older than 90 days.
Write the naive NOT IN version, observe zero rows, then rewrite it as NOT EXISTS.
Join the survivors back to dim_catalog_item for rarity and available_from, ordering by available_from so the oldest dead stock surfaces first.

Expected result: A non-empty list of current cosmetic item_ids, one row each, with rarity and available_from, covering items live for more than 90 days with no sink ledger entry in that window. The naive NOT IN version of the identical query returns exactly zero rows.

Verify it yourself

Row count must be strictly less than the count of current cosmetics older than 90 days. Equality means the anti-join predicate never matches anything.
Spot-check one returned item_id with a direct count over fct_currency_ledger for that item_id in the window and confirm zero sink rows.
Swapping NOT EXISTS for LEFT JOIN with IS NULL must return an identical set. A difference means a join condition leaked from ON into WHERE.

Follow-up

The same NOT IN written against fct_iap_transaction.sku_id returns rows normally. What is different about that column, and would you still reach for NOT EXISTS?
How would you separate an item nobody wants from an item nobody can see, using available_from, available_to and store impression data?
An item shows zero sinks but non-zero grants from source_system = 'container_open'. Is it unsold, or is it simply not directly purchasable?

Walk me through your modeling approach to sports betting: how would you account for real-time, streaming in-game events to update live odds dynamically?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Design a recommendation engine that suggests personalized betting slips to users based on their historical betting patterns and real-time trending events.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Explain how you would implement a custom evaluation metric in Python to optimize a model specifically for sports betting odds precision.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Walk through this specific block of Python code, explain what it output, and identify potential bottlenecks or areas where memory usage can be optimized.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you design and evaluate an A/B test when there are network effects or user spillover across control and treatment groups?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

Find a proxy for perceived fairness in a randomised reward table

Medium · proxy metrics · selection bias · drop tables · survivorship

A container SKU (dim_catalog_item.item_class = 'container') grants items from a drop table with a 1.5% legendary rate and a hard guarantee at the 60th open. Leadership asks you to measure whether players find the system fair. Fairness is not in any table. Using fct_currency_ledger (source_system = 'container_open'), fct_session and fct_match_participant, propose two proxy metrics, state the direction and likely size of the bias in each, and say what decision you would refuse to make from proxies alone.

Approach

Say what fairness is not. It is not the drop rate matching its configuration, and it is not the mean number of opens. Both can be exactly as designed while the experience is resented.
Build the exposure variable first: per player, the run length of consecutive container_open ledger entries since their last legendary grant. Ground it in the reference distribution, because under a 1.5% rate with a guarantee at the 60th open, expected opens is (1 - 0.985^60)/0.015, about 39.7, the median is 46, and roughly 41% of players reach the guarantee. Reaching pity therefore describes a plurality of the population and is useless as a frustration flag. Use the upper tail of the dry streak instead.
Proxy one, behavioural: next-24-hour session count and matches played after an open, bucketed by dry-streak length and compared against that same player's trailing 14-day baseline. Bias: it is computed only over players who kept opening containers, so the most alienated players are missing from every bucket and the estimate is attenuated toward zero, with the attenuation growing as streak length grows, which is exactly where you wanted the signal.
Proxy two, economic: the share of players who stop spending premium currency on containers while continuing to spend it on direct-sale items, within 7 days of a long dry streak. Bias: it cannot separate disappointment from a rational switch to a deterministic purchase, and it is defined only on players who spend at all, a small share of actives, so it reports the sentiment of the group with the least reason to be angry.
Triangulate with a source that is not behavioural. A sampled survey or a support-contact rate keyed to the same dry-streak buckets has different biases from both proxies, so report all three together rather than choosing the one that agrees with you.
Draw the line explicitly. These proxies can support flagging a configuration or region whose behavioural response is far worse than the rest, and they cannot support a claim that the system is or is not perceived as fair, nor a change to the drop rate.

Worked solution (35 min)

Compute the reference distribution for the current table: expected opens, median opens, and the share of players who reach the guarantee, so you know which streak lengths are actually rare.
Build the dry-streak variable from ordered container_open ledger rows per player.
Define proxy one as a within-player comparison of the 24 hours after an open against that player's trailing 14-day baseline, bucketed by dry-streak length.
Define proxy two on premium spend composition within 7 days, containers against direct-sale items.
Write one sentence per proxy naming the population it silently conditions on and the direction that pushes the estimate.
Write the sentence that refuses the decision the proxies cannot support.

Expected result: Two proxies with explicit definitions and explicit biases: a within-player post-streak activity change, attenuated because it conditions on players who kept opening; and a premium-spend composition shift, defined only on spenders and unable to separate disappointment from a preference for deterministic purchases. Plus a non-behavioural triangulating source and a stated refusal to tune the drop rate on proxy evidence alone.

Verify it yourself

Under a 1.5% rate with a guarantee at the 60th open, expected opens is (1 - 0.985^60)/0.015, about 39.7, while the median is 46. The mean sits below the median because the guarantee truncates the long right tail of the geometric. If your reference numbers do not show that ordering, recheck them.
About 41% of players reach the guarantee, so any proxy that treats reaching pity as the frustration event is describing a plurality of your players.
Each proxy in your answer names the population it is conditioned on.

Follow-up

Next-day play falls 6% after a 50-open dry streak. How much of that is the streak, and how much is that a long streak means a long and expensive session that may simply have exhausted a budget?
The guarantee moves from 60 opens to 40. Which of your proxies can read that change within a week, and what happens to your dry-streak exposure variable when it does?

Recent install cohorts look richer than mature ones

Medium · cohort maturity · refund recognition · revenue definition

Net revenue per install at cohort day 30 reads about 6% higher for install cohorts aged 30 to 45 days than for cohorts aged 120 days or more. Revenue is refund-adjusted using refunds recognised to date. Tables: dim_player (player_id, install_ts, acquisition_channel) and fct_iap_transaction (player_id, purchase_ts, price_usd_net, status, status_changed_at, refund_ts). Marketing wants to credit a creative refresh. Decompose the 6% and say what you would change about the chart.

Approach

Separate the two clocks the chart conflates. Purchase accrual is capped at cohort day 30 by the metric definition, so it is the same for every cohort. Refund recognition is not capped: a refund lands weeks after its purchase, so a young cohort has observed only part of the refunds its day-30 purchases will eventually attract. The metric is therefore not comparable across cohort ages even though the accrual window is fixed.
Measure the recognition curve from cohorts old enough to be complete. Using purchases at least 60 days old, compute the share of eventual 60-day refund and chargeback value that had been recognised by each day since purchase. That curve is the instrument for everything that follows.
Apply the curve to size the bias. If r is the eventual share of net value refunded within 60 days and f is the share of that already recognised for a given cohort, measured net over matured net is (1 minus r times f) divided by (1 minus r). Compute f as a weighted average over the cohort's purchase-age distribution, not at the cohort's own age, because a cohort aged 40 days contains purchases from day 0 to day 30.
Subtract the maturity term and decompose what is left by acquisition_channel using the standard mix and within-segment split, since a creative refresh usually moves channel mix as well as channel performance.
Confirm nothing else changed in the pipeline over the comparison span: check store_channel availability, whether any price book or FX conversion changed, and whether platform_fee_usd rates moved, since all three alter price_usd_net without any player behaving differently.
Fix the chart rather than the number: restrict it to cohorts that have passed a fixed maturity, or hold refunds at a fixed observation lag so every cohort is measured with the same refund window.

Follow-up

How long must a cohort be to appear on this chart, and how do you answer the executive who wants a read on last week's cohort?
Would you rather fix the maturity by dropping young cohorts or by using the recognition curve to project them, and what does each choice cost you?
If the creative refresh genuinely raised revenue per install, how would that look different from what you found here?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

Explain the concept of gradient boosting to a non-technical stakeholder, then explain how you would tune its hyperparameters for a latency-sensitive application.

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Turn 'engagement is down' into a scoped answerable brief

Easy · scoping · release calendar · triage · metric definition

A studio lead messages: engagement is down, can you look into it. You have dim_player, fct_session, fct_match_participant and the release calendar. Daily core-loop players, defined as distinct players per UTC day with at least one match reaching a terminal result other than abandon, is down 6% week over week, and a content release landed nine days ago. You have 30 minutes before a standup. Produce the scoping questions you would ask, the first three cuts you would run, and the one-paragraph brief you send back before doing deeper work.

Approach

Pin the metric and the comparison before touching data. Ask which number the lead actually saw and over what window, because a week-over-week read nine days after a release is measuring post-release decay by construction, and that alone may be the whole answer.
Ask the decision question, not more metric questions: what would the lead do differently if this turns out to be new players versus returning, one platform versus all, one region versus global. Scope follows the decision, and an investigation with no decision attached should be declined or deferred.
Run three cheap cuts that split the space rather than confirm a hunch. First, new versus existing by install cohort age, which separates an activation problem from a retention problem. Second, platform crossed with app_version, where a bad build shows as a concentration of fct_session.ended_reason = 'crash' and truncated duration_seconds. Third, server_region, where an infrastructure incident shows as elevated match abandon rate and p95 matchmaking wait rather than as fewer app opens.
Re-baseline against the matched day in the previous release cycle instead of against last week, so the comparison is not dominated by the release calendar.
Send a brief that states what you confirmed, what you ruled out, the current best explanation with its confidence, and the size of the next block of work with the question it would close.

Follow-up

The crash concentration is on one device_model at one app_version. What do you send, to whom, and how urgently?
How would you tell a genuine drop apart from an instrumentation change that altered which sessions get logged?
If all three cuts come back flat, what is your fourth cut and why that one?

Explain a wide revenue interval to a non-technical executive

Easy · uncertainty communication · heavy tails · bootstrap · decision framing

A 14-day test of a new currency-pack price is done. Net ARPDAU is up 4.1% with a bootstrap 95% interval of -1.8% to +10.3%. Paying conversion is up 0.6 percentage points with a much tighter interval. In both arms the top 1% of payers hold 42% of net revenue. An executive with no statistics background asks whether this is a win, yes or no, and has ten minutes. Give the explanation, the recommendation, and the claim you refuse to make.

Approach

Answer the decision first, in one sentence, then explain. An executive who waits ninety seconds for a recommendation stops listening to the reasoning.
Convert the interval into stakes in their units. Translate -1.8% to +10.3% of net ARPDAU into a monthly or annual dollar range at current active player-days, and say which end you would budget against.
Explain the width with a fact they already believe rather than with sampling theory: 42% of net revenue sits on 1% of accounts, so a handful of large purchases move the average. That is a property of the business, not a defect of the test, and it is why a conversion number of the same length is far tighter.
Offer the conversion result as the part of the decomposition that does have enough sample, and be explicit that it is a different claim: more people paid, and the size of the average payment is where the uncertainty lives.
Price more data honestly. The interval narrows roughly with the square root of exposure, so halving its width needs about four times the runtime. Give the calendar date that implies and say whether the decision is worth waiting for.
Name the claim you will not make, so nobody reports the point estimate as fact after you leave the room.

Follow-up

The executive suggests capping the top payers and rerunning the numbers to tighten the interval. What do you say?
Bootstrap intervals can undercover on a very heavy tail. How would you check whether yours is trustworthy at this sample size?
If the interval had been -0.4% to +1.2%, would your recommendation change, and why?