Interview guides · Datadog

Datadog Data Scientist Interview Questions & Guide 2026

Case practice for Datadog data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

A Data Scientist at Datadog sits at the intersection of massive-scale observability, platform engineering, and high-velocity experimentation. Datadog processes trillions of data points daily—monitoring infrastructure logs, application metrics, traces, and security events across cloud environments. Data Scientists build the intelligence layer that transforms this raw, noisy system telemetric data into actionable insights for engineers, data teams, and business leaders worldwide.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

Measure churn only on renewal-eligible accounts
Read NRR on a fixed account cohort
Power experiments for heavy-tailed account revenue

Overview

In this role, your work directly shapes both internal product decisions and customer-facing capabilities. Whether you are developing real-time anomaly detection algorithms for cloud infrastructure, modeling high-volume time-series data under strict computational and memory constraints, or driving causal inference and A/B testing strategy (amplified by integrated tools like Eppo), your output defines how platform features are validated and deployed. You will partner closely with product managers, backend engineers, and solutions teams to ensure that metric frameworks and analytical models scale alongside rapid business growth.

What makes this role unique is the technical rigor required to operate in an environment where standard out-of-core algorithms fail. Dealing with non-stationary time series, high-cardinality system metrics, massive datasets that exceed memory limits, and complex experimentation setups (including holdouts and synthetic controls) requires a strong foundation in both fundamental statistics and efficient algorithms.

The Datadog interview process

Datadog candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Fundamentals

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Onsite Stage

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Building features from data that postdates the prediction time

Check every feature against the timestamp at which the model would actually score, and drop anything computed from a window that includes or follows the label event. For a forecasting use case, split train and test by time rather than at random, and split by entity when the same entity recurs.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Questions & practice

Given an extremely large dataset with a continuous target column, how do you approach modeling when memory bounds limit parameter fitting?

Medium · statistics and probability

Approach

Write down the assumption the method needs before you use the method.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

How would you explain this result to someone who does not know statistics?
What sample size would you need to detect an effect half this size?

How do you design an LLM or ML system evaluation pipeline to monitor output quality and latency degradation in production?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Implement seven-day activation rate from its written definition

Easy · metrics · pandas · activation

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

Approach

Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Worked solution (20 min)

cohort = accounts.loc[~accounts.is_internal & accounts.created_at.between(week_start, week_start + pd.Timedelta(days=7), inclusive='left'), ['account_id','created_at']].drop_duplicates('account_id')
ok = requests[(requests.http_status < 400) & requests.api_key_id.notna() & (requests.traffic_class != 'synthetic_monitor')]
first_ok = ok.groupby('account_id', as_index=False).request_at.min(); m = cohort.merge(first_ok, on='account_id', how='left'); assert len(m) == len(cohort)
activated = (m.request_at - m.created_at) <= pd.Timedelta(hours=168); return {'rate': float(activated.mean()), 'numerator': int(activated.sum()), 'denominator': len(m)}

Expected result: A dict whose denominator is the count of distinct non-internal accounts created in that ISO week, numerator <= denominator, and rate = numerator / denominator; accounts with no qualifying request sit in the denominator with NaT and evaluate to False.

Verify it yourself

len(m) == cohort.account_id.nunique(); the merge must not add rows.
Dropping the synthetic_monitor filter can only raise the numerator, never lower it.
Widening the window from 168h to 336h is monotone: the numerator must not decrease.
An account whose only qualifying request falls at exactly created_at + 168h counts as activated, since the comparison is inclusive.

Follow-up

Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
How would you decide whether 168 hours is the right window rather than 72 or 336?
An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Write a query using SQL window functions to calculate the 7-day rolling average of user session durations across regional clusters.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Given a table of infrastructure log events, find the session ID with the longest continuous error state using sessionization techniques.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Monthly account margin joined as-of the live contract version

Medium · as-of join · fan-out · scd2 · unit economics

fct_usage_daily carries account_id, usage_date, net_amount_cents and cogs_cents. fct_subscription_period carries account_id, plan_tier, term_start_date, term_end_date, booked_at and is_current, with one row per contract version, so several superseded versions can bracket the same usage_date. dim_account is a type 2 dimension keyed on account_id with employee_band, is_internal, effective_from, effective_to and is_current. Return monthly net revenue, allocated COGS and gross margin per account, labelled with the plan_tier in force during that month and the employee_band in force at month end. The reported figure excludes internal accounts, so it cannot equal the raw sum of net_amount_cents; reconcile additively instead, quantifying every bucket you drop so the reported and dropped amounts add back to that raw sum.

Approach

Aggregate fct_usage_daily to one row per (account_id, month) before touching any dimension. Aggregating after the join is what multiplies revenue, and no amount of DISTINCT afterwards recovers the right number.
Resolve the contract as-of the month rather than as-of now. Take fct_subscription_period rows whose term brackets the month, then rank with ROW_NUMBER() OVER (PARTITION BY account_id, month ORDER BY booked_at DESC, subscription_period_id DESC) and keep rank 1. Filtering on is_current instead backdates today's plan over last year's usage and quietly rewrites history.
Resolve dim_account with the half-open interval effective_from <= month_end AND (effective_to > month_end OR effective_to IS NULL). The NULL on the live version has to be spelled out or the current row drops out of every recent month.
Compute margin as (sum(net_amount_cents) - sum(cogs_cents)) / NULLIF(sum(net_amount_cents), 0) at the account-month grain, and report the distribution rather than a blended rate so margin-negative accounts stay visible.
State the month-straddling rule explicitly: either split the month at the amendment date or take the version in force at month end. Both are defensible; an unstated choice is not.
Reconcile additively, not by equality to the unfiltered total. The output deliberately drops two populations: account-months whose as-of dim_account version carries is_internal = true, and account-months with no fct_subscription_period version bracketing the month, which an inner join on the contract removes without saying so. The statement that ties out is reported_net + internal_net + unmatched_net = sum(net_amount_cents) over the same usage_date range. Plain equality to the raw sum would only hold if both dropped buckets were empty, and the internal one never is.
Compute the internal bucket with the identical as-of rule used for the output, evaluating is_internal on the dim_account version in force at that month end. Reading is_internal from the current version instead moves accounts between the two sides of the identity and it stops closing.

Worked solution (30 min)

Build usage_monthly: SELECT account_id, date_trunc('month', usage_date) AS month, sum(net_amount_cents) AS net_cents, sum(cogs_cents) AS cogs_cents FROM fct_usage_daily GROUP BY 1, 2.
Build contract_asof by joining usage_monthly to fct_subscription_period on account_id with the term bracketing the month, then applying the ROW_NUMBER ranking on booked_at DESC and keeping rn = 1.
Join dim_account with the half-open effective_from/effective_to predicate evaluated at month end, and filter is_internal = false on the version selected.
Select account_id, month, plan_tier, employee_band, net_cents, cogs_cents and (net_cents - cogs_cents)::numeric / NULLIF(net_cents, 0) AS gross_margin.
Run the reconciliation as a three-way identity over the same usage_date range: sum(net_cents) in the output, plus sum(net_cents) over account-months whose as-of dim_account version has is_internal = true, plus sum(net_cents) over account-months with no bracketing subscription version, must equal SELECT sum(net_amount_cents) FROM fct_usage_daily for that range, to the cent. Publish the two dropped amounts next to the total rather than leaving them implicit; a non-zero unmatched bucket is a contract-coverage bug to chase, not rounding.

Expected result: One row per account per month with net revenue, allocated COGS, gross margin, and the plan_tier and employee_band that were actually in force, with output row count equal to the number of distinct (account_id, month) pairs that survive both the internal-account filter and the contract-bracket join.

Verify it yourself

Output row count equals COUNT(DISTINCT (account_id, date_trunc('month', usage_date))) on the source fact restricted to the same two filters; counted against the unrestricted fact it is strictly smaller, by exactly the internal and unmatched account-months.
reported_net + internal_net + unmatched_net equals the raw sum from fct_usage_daily over the same usage_date range, to the cent, with all three terms computed from the same usage_monthly CTE.
The reported figure alone is strictly below the raw sum whenever any internal account recorded usage in the window, and equal to it only in a window where none did.
No account-month carries more than one plan_tier or employee_band value.
Accounts with net_cents of zero return NULL margin rather than a division error, and margin-negative accounts appear with a value below zero rather than being dropped.

Follow-up

An account amends mid-month from team to enterprise. Show what your query reports and argue for one attribution rule over the other.
Your monthly total disagrees with the finance figure by a small amount. Where would you look first, and which number do you defend?

Describe a time an experiment yielded contradictory results between primary engagement metrics and guardrail performance metrics. How did you resolve it?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Given a streaming input of numeric system metrics, write a sliding window approach combined with a hashmap to maintain rolling quantile estimates.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you define primary, secondary, and guardrail metrics when evaluating a major platform change?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Given a series of dashboard plots showing a sudden spike in latency across microservices, walk through your root cause analysis methodology.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you determine the correct unit of randomization and sample size required to achieve sufficient statistical significance and power?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

How would you handle interference between treated and control units?
What would you do if you could not randomise at all?

What experimentation pitfalls (e.g., novelty effects, network leakage, sample ratio mismatch) have you encountered, and how did you mitigate them?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.
Say whether units interfere with each other, and switch design if they do.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Build an activation metric tree for a self-serve onboarding rebuild

Easy · metric tree · activation · cohort windows · denominators

The self-serve signup flow is being rebuilt. You have dim_account (account_id, created_at, is_internal, first_workspace_at, is_current) and fct_api_request (account_id, workspace_id, environment, api_key_id, http_status, traffic_class, request_at). Build a three-layer metric tree from signup through to production dependency, name the primary metric for this specific change, and state the fixed measurement window and the denominator for each layer. Two of the three layers share a denominator and the third does not, so say which invariant still holds across the tree once the denominators differ. Say which layer the onboarding team can actually move and which it cannot, and why raw signup counts are not on the tree.

Approach

Fix the cohort once: accounts created in cohort period P with is_internal = false, collapsed to one row per account_id since dim_account is type 2. Say out loud that employee, demo and load-test accounts are removed, otherwise the tree partly measures internal testing.
Layer one, qualified signup, on the cohort denominator: accounts recording at least one successful request (http_status < 400) with traffic_class in ('interactive','batch') within 14 days of created_at. This replaces raw signup counts, which measure registration-form friction and duplicate sign-ups from one organisation rather than demand.
Layer two, activation, also on the cohort denominator: the share of the cohort whose first successful request with a non-null api_key_id arrives within a fixed 168 hours of created_at. Fixed window, not trailing, so that cohort weeks are directly comparable and no cohort is censored.
Layer three, production depth, on a conditional denominator: the share of activated accounts holding a workspace with environment = 'production' whose trailing 7-day count of successful non-synthetic requests exceeds a floor calibrated from the data, for example the volume above which 90-day retention stops rising. A floor picked by intuition makes this a metric about the floor. Conditioning on activation is the right operational choice, because the question it answers is whether accounts that got started go on to depend on the product, but it means this rate is not on the same scale as the two above it: a conditional rate of 0.50 sits perfectly happily underneath a 0.30 activation rate. To place it back on the funnel, multiply it by the activation rate and report that product as the cohort share.
Align the traffic_class filter across all three layers, otherwise the underlying account sets do not nest: an activation definition that admits traffic_class = 'ci' will count accounts that layer one excluded, and the tree stops being a funnel. With mixed denominators the invariant that survives is on account sets and therefore on counts, not on rates. Then name the primary metric for this change, the seven-day activation rate, because the onboarding flow controls the path to first successful call and nothing below it. Production depth is the lagging check that the activation gain was real rather than one sample request.

Worked solution (25 min)

Build the cohort table: one row per account_id, bucketed by the ISO week of created_at, is_internal = false, dim_account collapsed to is_current = true.
For each layer, find the first qualifying request per account from fct_api_request under that layer's filter and keep min(request_at), using EXISTS or a lateral so the cohort stays at one row per account.
Compute layers one and two as shares of the cohort denominator and layer three as a share of the activated accounts, carrying the raw account count beside every rate, and mark any cohort week not yet clear of the longest window (14 days) as not reportable.
Plot the three rates by cohort week, and separately confirm nesting on the counts within each cohort.

Expected result: A three-row tree with explicit numerator filters, fixed windows of 14 days, 168 hours and trailing 7 days, and a stated denominator per layer: layers one and two share the signup-cohort denominator of non-internal accounts, while layer three is conditional on activation and is therefore a rate that cannot be read on the same scale as the two above it. The primary metric is the seven-day activation rate, with qualified signup above it and production-depth adoption below. Cohort weeks inside the longest window are marked not yet reportable, the traffic_class filter is stated to be identical across layers so the account sets nest, and the cohort share of production depth is reported as the activation rate multiplied by the conditional rate.

Verify it yourself

Check nesting on account counts, not on rates: within a cohort the production-depth account set must be a subset of the activated set, which must be a subset of the qualified-signup set. The rates do not obey that ordering, because layer three is conditional on activation, so a conditional rate of 0.50 sitting above a 0.30 activation rate is correct rather than a defect.
The reported cohort share of production-depth accounts must equal the activation rate multiplied by the conditional production-depth rate; a mismatch means the two layers were computed over different cohort definitions or different traffic filters.
Every reported cohort week must be at least 14 days old at query time.
Removing the is_internal filter should visibly change the denominator; if it does not, the flag is not populated and the whole tree rests on an untested assumption.

Follow-up

Activation rises 6 points and the production-depth rate among activated accounts is flat 60 days later. Say what happened to the cohort share of production-depth accounts, and why a flat conditional rate is the better of the two readings available here.
Median time-to-first-successful-call answers the same question with more information. Why is it a worse weekly dashboard metric, and what estimator would make it honest?

Blended gross margin fell while every segment improved

Medium · mix shift · simpsons paradox · as-of join

Blended gross margin across paying accounts fell from 71 percent to 66 percent over two quarters, yet margin improved inside every plan_tier and every deployment_model. You have fct_usage_daily (account_id, sku_code, usage_date, net_amount_cents, cogs_cents), fct_subscription_period (account_id, plan_tier, term_start_date, term_end_date) and dim_account (account_id, deployment_model, is_internal, effective_from, effective_to). Produce an exact decomposition of the five-point move into mix, rate and interaction terms, and name the segment shift that carries it.

Approach

Write blended margin as a revenue-weighted average of segment margins, m = sum over i of w_i * m_i, where w_i is the segment's share of net_amount_cents. Margin is a ratio of sums, so only revenue weights reproduce the blended figure.
Apply the exact three-term decomposition: delta_m = sum(delta_w_i * m_i0) + sum(w_i0 * delta_m_i) + sum(delta_w_i * delta_m_i), read as mix, rate and interaction. It is exact by construction, so the three terms must sum to the observed change with no residual.
Build segments from as-of joins: take the fct_subscription_period version whose term brackets each usage_date and the dim_account version whose effective_from and effective_to bracket it, rather than joining on is_current and backdating today's attributes over last year's usage.
Run the decomposition on more than one segmentation, at minimum plan_tier, deployment_model and sku_code, since the mix that actually moved may not be the one anybody already suspected.
Once the driving segment is identified, determine whether the shift came from acquisition landing new accounts in a lower-margin segment or from existing accounts migrating, because those have different owners and different remedies.

Follow-up

The interaction term is sometimes large. What does it mean operationally, and when would you prefer a log-mean decomposition that distributes it across the other two terms?
Revenue-weighted average margin is one construction. What changes if leadership wants margin weighted by account count, and which question does each version answer?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ownership when the result is inconvenient

Sometimes the honest read is that the initiative did not work, and the person who commissioned the analysis was hoping otherwise. Interviewers want to know whether you softened it. Prepare the case where you delivered an unwelcome result, how you presented the uncertainty without hiding behind it, and what the team did next.

How do you handle non-stationary time series when attempting to establish baseline statistics for system latency?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

State the measured impact of your own work honestly

Hard · impact · causal inference · self-assessment

You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.

Approach

The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.

Follow-up

Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
What would have had to be true for the naive covered versus uncovered comparison to be valid?
If the holdout costs the team real renewals, how do you justify asking for it, and to whom?

Walk through an analysis you later discovered was wrong

Easy · data quality · error ownership · metering

Six weeks ago you reported that consumption fell 9 percent in the last week of the month, and a team spent a sprint investigating the cause. The fall was an artefact: rows in fct_usage_daily land late and are restated in place, and you queried before the tail had settled. Describe how you found the error, what you told the people who acted on it, and the control you put in place so this class of mistake cannot reach a dashboard again. Be specific about how the settling window was measured.

Approach

The interviewer is probing whether you self-report errors before someone else finds them, and whether your fix is structural rather than a promise to be more careful. Say plainly that the number was wrong and that a sprint was spent on it, before describing any diagnosis.
Establish the artefact quantitatively instead of asserting that data lands late. For each usage_date, compare the total as of first_written_at against the settled total and read the settling time off that curve, for example 97 percent of final by day three and 99.5 percent by day five.
Correct the record the same day, in the channel the original number went out in, to the same audience. The cost of the wasted sprint belongs in the correction, not in a footnote.
Make the fix structural: exclude a trailing lag window from every reportable figure, and make the reporting view return no rows inside that window rather than returning partial ones. A dashboard that shades unsettled days still gets read as a decline.
State what generalises. Any fact table restated in place has this failure mode, so the guard belongs at the source rather than on the one dashboard that embarrassed you. A strong answer ends with the class of error closed; a generic one ends with a lesson learned.

Follow-up

How did you choose the completeness threshold behind the lag window, and what would make you recalibrate it?
What did you say to the team that lost the sprint, and what did they say back?
Is there a legitimate case for showing the unsettled tail at all, and to whom?