DigitalOcean Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at DigitalOcean, you will play a crucial role in analyzing and interpreting complex datasets to drive actionable insights that enhance the company's product offerings and strategic decisions. Your work will directly impact how DigitalOcean serves its users, optimizing processes and informing product development based on data-driven findings. As a member of a dynamic team, you will navigate a range of projects, from statistical modeling to machine learning solutions, all aimed at improving the overall user experience and operational efficiency.

SQL is seldom the hardest round and is often the one that eliminates people. The working bar is usually window functions, correct deduplication, and joins that do not silently fan out rows, rather than obscure syntax.

Focus areas

• Strip CI, retry and synthetic traffic first
• Measure churn only on renewal-eligible accounts
• Analyse at the account grain, cluster errors

Overview

The role of a Data Scientist at DigitalOcean is particularly compelling due to the company's commitment to simplifying cloud infrastructure for developers around the globe. You will engage with products that serve developers, startups, and businesses, making your contributions vital in shaping how technology is accessed and utilized. The blend of technical challenge and real-world application offers a unique opportunity to make a significant impact on both the business and its customers.

The DigitalOcean interview process

DigitalOcean candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

• Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
• Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
• Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

• Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
• Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
• Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Interviews with Managers

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Technical Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Case Studies

Underneath the business framing, this round is usually asking whether you can turn a fuzzy goal into a quantity that could be computed from data such a business would plausibly hold. That means a metric with a stated numerator, denominator, eligibility rule and time window, plus an honest account of the conditions under which it would mislead you. Answers come apart when a candidate names a familiar metric and never defines it, because every follow-up then lands on an ambiguity that was left open and the candidate has to invent the definition under pressure.

What to demonstrate

• Whether a named metric arrives with its denominator, eligibility rule and window attached rather than assumed
• Whether the measure follows from the mechanism you proposed, or is a recognisable metric retrofitted to it afterwards
• Whether you name a guardrail that would reveal the gain came from somewhere you did not want it to come from
• Whether you can say what data the plan requires and what you would settle for if that logging were never implemented

How to prepare

• Take five metrics you reach for by reflex and write each as one sentence containing numerator, denominator, eligibility rule and time window. The ones you cannot finish are the ones that will fail under follow-up.
• For a product you use daily, write the measurement plan you would propose for a change to it: primary metric, one guardrail, the unit of analysis, and the table the numbers would come from.
• Practise the substitution question. For three metrics you like, write what you would measure instead if the event you depend on were not being logged.

Behavioral Questions

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

• Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
• The counterfactual: what would have been done without your analysis, and why that default was worse
• How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

• Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
• Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
• Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Where candidates lose points

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Randomising an experiment at the user level when users share an account

Two problems fire at once. Colleagues in one workspace see each other's work and talk to each other, so a treated user changes the behaviour of a control user in the same account, which violates the no-interference assumption and biases the estimate toward zero. Separately, outcomes within an account are strongly correlated, so the effective sample size is roughly n / (1 + (m - 1) * rho) for m users per account and intra-class correlation rho, not n. With rho around 0.3 and twenty users per account that is a design effect near 6.7, meaning a user-level confidence interval is about two and a half times narrower than it should be and results cross significance thresholds on noise alone. Randomise the account and cluster the standard errors.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

Write a function to calculate the mean and standard deviation of a list of numbers.

Medium · statistics and probability

Approach

1. Quantify uncertainty explicitly rather than reporting a point estimate alone.
2. Say what the estimate is of, and over what population it generalises.
3. Write down the assumption the method needs before you use the method.

Follow-up

• How would you explain this result to someone who does not know statistics?
• Which assumption here is most likely to be violated in practice?

Explain how you would optimize a machine learning pipeline for efficiency.

Medium · machine learning and modelling

Approach

1. Pick an evaluation metric that matches the cost of each error type, not a default.
2. Set a baseline first, so any model has something honest to beat.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• How would you choose the decision threshold, and who owns that choice?
• What would you monitor after launch to know the model is still valid?

What types of machine learning algorithms are you most comfortable with, and why?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Permutation-test a consumption experiment randomised at account level

Hard · permutation-test · heavy-tails · cuped · experiment-readout

An experiment randomised 900 accounts into two arms. You have one row per account: account_id, arm, consumption_28d (billable units after launch) and consumption_pre (the 28 days before). Consumption is heavy-tailed and the largest account is several percent of the total. Write a permutation test from scratch: winsorise at the pooled 99th percentile as a pre-registered rule, use the difference in arm means of the winsorised outcome as the statistic, and obtain a two-sided p-value from 20,000 relabellings of the account-level arm vector. Report the observed effect, the p-value, and the same test on a CUPED-adjusted outcome.

Approach

1. Be precise about what the permutation test needs. Under the sharp null of no effect for any account, the outcomes are exchangeable across arm labels, and the test is valid for ANY statistic T(outcomes, labels) provided the identical function is applied to the observed labels and to all 20,000 relabellings. The pooled 99th percentile is a function of the outcome vector alone, so recomputing it inside the loop returns the same number 20,000 times: that is wasted CPU, not a bias, and hoisting it out is an optimisation rather than a correctness fix. Say plainly that capping at all changes the estimand from mean consumption to mean capped consumption; it is not a neutral cleaning step.
2. The mistake that does invalidate the test is an asymmetry between the observed statistic and the permuted ones, and the easiest way to create it is to derive the cleaning rule from the observed arm labels and then freeze it — winsorise each arm at its own observed 99th percentile, hold those two caps fixed, and permute. The observed value is then computed with caps matched to its own partition while every relabelling is scored with caps belonging to a different one, so the null distribution no longer answers the question the p-value claims to answer. A per-arm cap recomputed consistently inside every permutation is a valid test, but it estimates a contrast whose two sides are capped at different thresholds, so prefer the pooled cap on estimand grounds and pre-register it.
3. Permute the account-level arm vector, because the account is the randomisation unit. Relabelling anything finer — users, workspaces, requests — generates a null distribution narrower than the design actually supports and returns p-values that are anti-conservative.
4. Vectorise the null: tile the treatment indicator into a (B, n) matrix and permute along axis 1 with rng.permuted(..., out=...). The statistic is a difference of means, so the treated sum alone determines it and the whole null is one matrix-vector product. Use the two-sided p-value (1 + count(|stat_perm| >= |stat_obs|)) / (B + 1); the plus-one on each side is not cosmetic, it keeps the p-value away from exactly zero and keeps the test valid at finite B.
5. For CUPED, fit theta = cov(y, x) / var(x) on the pooled data and use that same theta for the observed statistic and every relabelling. Pooled theta, like the pooled cap, carries no label information, so where in the loop you compute it is again only a performance question; fitting theta within arms is what goes wrong, because the adjusted outcome then depends on the labels and an observed-label fit frozen across all 20,000 relabellings breaks the match between observed and permuted statistics. x must be measured entirely before launch, which consumption_pre is. Expected variance reduction is about 1 - corr(y, x)^2; measure the achieved reduction from the two null distributions rather than asserting it.

Worked solution (45 min)

1. cap_y = np.quantile(df.consumption_28d, 0.99); y = np.minimum(df.consumption_28d.to_numpy(float), cap_y); cap_x = np.quantile(df.consumption_pre, 0.99); x = np.minimum(df.consumption_pre.to_numpy(float), cap_x)
2. t = (df.arm == 'treatment').to_numpy(); n1 = int(t.sum()); n0 = len(t) - n1; obs = y[t].mean() - y[~t].mean()
3. rng = np.random.default_rng(11); L = np.tile(t.astype(np.int8), (20_000, 1)); rng.permuted(L, axis=1, out=L); s1 = L @ y; stats = s1/n1 - (y.sum() - s1)/n0
4. p = (1 + int(np.sum(np.abs(stats) >= abs(obs)))) / (20_000 + 1)
5. theta = np.cov(y, x, ddof=1)[0,1] / np.var(x, ddof=1); y_adj = y - theta*(x - x.mean()); repeat steps 2 to 4 on y_adj and compare stats.std(ddof=1) between the two runs.

Expected result: A two-sided p-value strictly between 1/20001 and 1, an observed effect expressed in capped billable units per account, and a CUPED null whose standard deviation is smaller than the unadjusted one by roughly sqrt(1 - corr(y, x)^2). The CUPED point estimate stays close to the unadjusted one, since the adjustment removes variance rather than shifting the effect.

Verify it yourself

• The permutation null is centred near zero: |mean(stats)| is a small fraction of std(stats).
• An A/A run, relabelling arm at random before the test and repeating over seeds, gives roughly uniform p-values with about 5% below 0.05.
• std(stats_cuped) < std(stats) whenever corr(y, x) is materially non-zero, and the ratio matches sqrt(1 - r^2) to within simulation noise.
• Moving the pooled cap computation inside the permutation loop reproduces the same p-value bit for bit at the same seed — the direct demonstration that a pooled cap carries no label information — while setting the cap to infinity reproduces the uncapped test exactly, so the winsorisation switch is independently verifiable.

Follow-up

• The p-value is 0.04 with the cap and 0.31 without it. What do you report, and what did you pre-register?
• Colleagues in a shared workspace can see the treated behaviour. How does that change the design and the estimate?
• How many accounts would you need to detect a 5% lift given this outcome's distribution?

Monthly account margin joined as-of the live contract version

Medium · as-of join · fan-out · scd2 · unit economics

fct_usage_daily carries account_id, usage_date, net_amount_cents and cogs_cents. fct_subscription_period carries account_id, plan_tier, term_start_date, term_end_date, booked_at and is_current, with one row per contract version, so several superseded versions can bracket the same usage_date. dim_account is a type 2 dimension keyed on account_id with employee_band, is_internal, effective_from, effective_to and is_current. Return monthly net revenue, allocated COGS and gross margin per account, labelled with the plan_tier in force during that month and the employee_band in force at month end. The reported figure excludes internal accounts, so it cannot equal the raw sum of net_amount_cents; reconcile additively instead, quantifying every bucket you drop so the reported and dropped amounts add back to that raw sum.

Approach

1. Aggregate fct_usage_daily to one row per (account_id, month) before touching any dimension. Aggregating after the join is what multiplies revenue, and no amount of DISTINCT afterwards recovers the right number.
2. Resolve the contract as-of the month rather than as-of now. Take fct_subscription_period rows whose term brackets the month, then rank with ROW_NUMBER() OVER (PARTITION BY account_id, month ORDER BY booked_at DESC, subscription_period_id DESC) and keep rank 1. Filtering on is_current instead backdates today's plan over last year's usage and quietly rewrites history.
3. Resolve dim_account with the half-open interval effective_from <= month_end AND (effective_to > month_end OR effective_to IS NULL). The NULL on the live version has to be spelled out or the current row drops out of every recent month.
4. Compute margin as (sum(net_amount_cents) - sum(cogs_cents)) / NULLIF(sum(net_amount_cents), 0) at the account-month grain, and report the distribution rather than a blended rate so margin-negative accounts stay visible.
5. State the month-straddling rule explicitly: either split the month at the amendment date or take the version in force at month end. Both are defensible; an unstated choice is not.
6. Reconcile additively, not by equality to the unfiltered total. The output deliberately drops two populations: account-months whose as-of dim_account version carries is_internal = true, and account-months with no fct_subscription_period version bracketing the month, which an inner join on the contract removes without saying so. The statement that ties out is reported_net + internal_net + unmatched_net = sum(net_amount_cents) over the same usage_date range. Plain equality to the raw sum would only hold if both dropped buckets were empty, and the internal one never is.
7. Compute the internal bucket with the identical as-of rule used for the output, evaluating is_internal on the dim_account version in force at that month end. Reading is_internal from the current version instead moves accounts between the two sides of the identity and it stops closing.

Follow-up

• An account amends mid-month from team to enterprise. Show what your query reports and argue for one attribution rule over the other.
• Your monthly total disagrees with the finance figure by a small amount. Where would you look first, and which number do you defend?

Collapse retried requests into logical operations per account

Medium · window functions · deduplication · null semantics

fct_api_request carries request_id, account_id, api_key_id, idempotency_key (null when the caller supplied none), is_retry, request_at and http_status. For one ISO week, count logical operations rather than HTTP calls: requests sharing (account_id, api_key_id, idempotency_key) collapse to the earliest one, while every request with a null idempotency_key is its own operation. Return per account the raw request count, the logical operation count and the ratio between them, and state which direction a rising 5xx rate pushes that ratio.

Approach

1. Split the stream before deduplicating. GROUP BY and PARTITION BY treat NULLs as equal to one another, while the = operator does not, so a single grouping pass folds every null-key request in an account into one 'operation' and can cut the count by orders of magnitude.
2. Deduplicate the keyed rows with ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) and keep rn = 1. Including request_id in the ORDER BY makes the result deterministic when two rows share a millisecond, which they will.
3. UNION ALL the null-key rows back in unchanged; they need no dedup and must not pass through the partition.
4. Aggregate per account: count() over the raw week, count() over the union, and the ratio of the two. Report the ratio, not the difference, so accounts of very different sizes are comparable.
5. Cross-check against is_retry, which flags a resend of the same idempotency key: raw minus logical should sit close to the count of is_retry rows, and a large gap means clients are resending without a key and the dedup is under-counting duplicates.
6. Name the inversion explicitly: 5xx responses provoke client retries, so amplification rises exactly when reliability falls, and any engagement metric built on raw request counts will show growth during an outage.

Worked solution (25 min)

1. Filter to the ISO week with request_at >= :week_start AND request_at < :week_start + interval '7 days', and record the raw count per account as the baseline.
2. Build keyed_dedup: SELECT ... ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) AS rn FROM the week WHERE idempotency_key IS NOT NULL, then keep rn = 1.
3. Build unkeyed: the same week's rows WHERE idempotency_key IS NULL, taken as-is.
4. UNION ALL the two, then GROUP BY account_id selecting the logical count; join back to the raw counts and compute raw::numeric / logical.
5. Validate on one account by comparing raw - logical against COUNT(*) FILTER (WHERE is_retry) for that account and explaining any gap.

Expected result: One row per account with raw_requests, logical_operations and an amplification ratio at or above 1.0, where logical_operations counts each distinct (account_id, api_key_id, idempotency_key) once and each null-key request once.

Verify it yourself

• logical_operations is never greater than raw_requests, and the two are equal for accounts whose keys are all distinct.
• For accounts with no null idempotency_key, raw_requests - logical_operations is close to the count of is_retry rows.
• Deliberately grouping on the key without the null split reproduces a visibly lower count, confirming the NULL collapse is the mechanism.
• Re-running the dedup returns the same surviving request_id set, proving the ORDER BY tiebreak made it deterministic.

Follow-up

• Which of these two counts belongs in the billable-units metric, and which in an engagement metric?
• An account's amplification ratio jumps from 1.05 to 3.0 in a day. Name three causes and the query that separates them.

If you were tasked with improving a product feature based on user feedback data, how would you proceed?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Fix the population and the time window before naming any metric.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Given a dataset, how would you approach the problem of predicting user churn?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Fix the population and the time window before naming any metric.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

How do you prioritize your work when managing multiple projects?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Fix the population and the time window before naming any metric.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Describe your process for designing an experiment to test a new feature.

Medium · a/b testing and experimentation

Approach

1. Say whether units interfere with each other, and switch design if they do.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• How would you handle interference between treated and control units?
• What would you conclude if the result is positive but the test is underpowered?

Can you describe a project where you applied statistical methods to derive insights?

Medium

Approach

1. Work from the decision backwards to the evidence you would need.
2. Clarify what is being asked and what a complete answer would contain.
3. State your assumptions explicitly before working the problem.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Set a stopping rule for a daily-read experiment dashboard

Medium · peeking · sequential-testing · multiple-comparisons

An experiment dashboard recomputes a two-sided t-test every morning for 21 days across one primary metric and eleven guardrails. The team ships whenever anything crosses p < 0.05 and stops the test that day. Quantify the false-positive rate this procedure actually carries, from repeated looks and from the twelve metrics separately, then specify the readout protocol you would put in its place: what is checked daily, what is checked at pre-declared points, what controls the family-wise or false-discovery rate, and what the stopping rule is.

Approach

1. Separate the two inflations, since they compound and have different remedies. Repeated fixed-horizon testing on accumulating data: at roughly 20 equally spaced looks with nominal alpha 0.05, the probability of crossing at least once under the null is about 0.25, and about 0.19 at 10 looks (Armitage, McPherson and Rowe, 1969). Multiplicity: twelve independent metrics at 0.05 give 1 - 0.95^12 = 0.46 of at least one false positive, somewhat lower under correlation but not much lower when metrics share a denominator.
2. Point out that the stopping rule is worse than either figure alone. Stopping the moment anything crosses is optional stopping on the maximum over twelve metrics and twenty-one days, so the two inflations multiply rather than average, and the realised error rate is above 0.5 for a null effect.
3. Rebuild around one pre-registered primary metric that alone decides ship or no-ship. Secondary metrics are demoted to hypothesis generation, reported with intervals and no significance marks, and are not permitted to trigger a ship on their own under any circumstances.
4. For the primary, choose deliberately between two valid options and say what each costs. An alpha-spending boundary with a small number of pre-declared interim analyses, where an O'Brien-Fleming shape spends very little alpha early so the final look retains nearly the full amount; or an always-valid confidence sequence, which permits unlimited looks at the price of a wider interval at every fixed time, typically requiring on the order of 1.2 to 1.5 times the sample to match the fixed-horizon width. Whichever is chosen goes into the power calculation before launch, not after.
5. Control the guardrail family separately from the primary. Benjamini-Hochberg at q = 0.10 when a missed regression costs more than an occasional false alarm; Holm when any single false positive triggers an expensive rollback. Say which regime you are in rather than reaching for a correction by habit.
6. Distinguish diagnostics from readouts and write both lists down. Sample ratio mismatch, exposure counts, error rates and pipeline health are checked every single day and are not peeking, because their only permitted action is to abort the test, never to declare a win. The abort conditions belong next to the stopping rule in the same document.

Worked solution (25 min)

1. Record the repeated-testing inflation for 21 daily looks against the nominal 0.05, about 0.25 at 20 looks, either from published tables or by simulating a random walk under the null.
2. Compute the independence bound 1 - 0.95^12 = 0.46, then re-estimate by simulation using the observed correlation matrix of the twelve metrics.
3. Pick the sequential method and size the test for it, adding the confidence-sequence width penalty to the power calculation or placing two interim looks under an O'Brien-Fleming boundary.
4. Write the one-page protocol: primary metric, stopping rule, interim dates, guardrail family and its FDR level, abort conditions, and the list of daily diagnostics that explicitly cannot trigger a ship.

Expected result: About 0.25 type-I error from twenty daily looks and about 0.46 from twelve independent metrics, compounding under a stop-on-first-crossing rule to more than half. Replaced by one primary metric under a pre-declared sequential boundary or confidence sequence, guardrails under Benjamini-Hochberg at q = 0.10, and daily diagnostics whose only permitted action is to abort.

Verify it yourself

• 1 - 0.95^12 = 0.4596 exactly under independence. A simulation using the real correlation matrix must come out below that; if it comes out above, the simulation is wrong.
• A valid sequential boundary is at least as wide as the fixed-horizon boundary at every look, including the final one. A boundary that is ever narrower is not controlling anything.
• A sample ratio mismatch check run daily needs no multiplicity correction, because its action is to stop and fix rather than to claim an effect.

Follow-up

• Your confidence sequence never crosses in 21 days, but the point estimate is exactly the size you powered for. What do you tell the team, and what do you do next?
• The eleven guardrails are heavily correlated with each other. Does that make Benjamini-Hochberg conservative or anti-conservative here, and does it change your q?
• Someone asks for a directional read on day five. What can you honestly give them without compromising the stopping rule?

Weekly active organisations fell nine percent over one week

Easy · seasonality · denominators · segmentation

A dashboard reports the weekly active organisation ratio on a trailing seven-day window ending each Wednesday. This week it reads nine percent below last week. You have fct_api_request (account_id, environment, traffic_class, http_status, request_at) and dim_account (account_id, billing_country, account_status, is_internal, is_current). Nothing was released. Decide whether usage actually fell, and hand back a corrected series plus a one-paragraph explanation that a non-analyst can repeat without you in the room.

Approach

1. Count the holiday-free business days inside each window before comparing them. A trailing seven-day window spans exactly five weekdays wherever it ends, so its business-day count can only fall to four or fewer when a public holiday lands inside it and can never reach six, while usage in this domain follows a hard five-to-two weekday cycle. Two windows holding different numbers of business days are not comparable whatever the product did.
2. Count distinct active accounts per calendar day for the last ten weeks and overlay the two windows. A calendar problem shows as a small number of weekdays sitting at weekend level, not as every day being uniformly lower.
3. Cut the daily series by billing_country and index each country-day to that country's trailing same-weekday median, which isolates a regional public holiday from a product change.
4. Check the denominator on its own: the metric divides by accounts whose account_status was in trial, free or active_paid for the whole week, so a batch suspension or status backfill moves the ratio with no change in the numerator at all.
5. Report the series with each window's business-day count and the holiday dates annotated beside it, and state the residual week-over-week change that survives once the calendar effect is removed. Compare against earlier windows holding the same number of business days rather than dividing by business days, since distinct-account counts are sublinear in window length and dividing would over-correct.

Follow-up

• Distinct account counts are sublinear in the number of days in the window. Why does losing one of five business days reduce the count by noticeably less than twenty percent?
• How would you make this metric comparable across countries with different holiday calendars without hand-maintaining a holiday table forever?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

Describe a time when you had to work collaboratively with cross-functional teams. What challenges did you face?

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Quantify the outcome, including what you would not claim credit for.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• What would you do differently if you ran that project again?
• What did you decide not to do, and why?

Scope an open-ended request to predict account churn

Medium · scoping · churn modelling · operating point

A customer success director asks for a list of accounts about to churn. You know only that the team has six people and that contracts are annual. Available data is fct_subscription_period, fct_usage_daily, fct_api_request, fct_support_ticket and dim_account. Before writing any code, produce the questions you need answered, a proposed definition of about to churn, and the shape of the artefact you would hand back, including the operating point that turns a score into a decision.

Approach

1. The interviewer is probing whether you convert a vague request into a decision with a capacity constraint attached. A candidate who starts talking about model families has already failed the exercise.
2. Pin the event and the horizon first. Churn is only possible at term_end_date, so the population is accounts renewing in the next 60 to 90 days, not the whole base. Ask explicitly whether contraction and downgrade count as churn or only full non-renewal, because the three have different base rates and different interventions.
3. Pin the action and the capacity. Six people times a realistic number of meaningful interventions per week gives k, and k is what the list is ranked to. Evaluate on precision at k rather than a global AUC over accounts that will never be contacted.
4. Audit leakage before choosing features. Every feature needs a timestamp proving it existed before the prediction date. A downgrade amendment, a churn reason code, and a ticket opened after the renewal conversation started are all leaks that will make the offline number look excellent and the live list useless.
5. Ask for the counterfactual now rather than later. Coverage is assigned deliberately, so without a held-out slice agreed at the start the intervention can never be evaluated, and you will be asked for its impact in nine months regardless.
6. Propose the smallest artefact that closes the loop: a weekly ranked list sized to capacity with two or three inspectable reasons per row, plus a stated policy for accounts below the line.

Follow-up

• The director insists all accounts are in scope, not only those renewing soon. How do you answer without simply refusing?
• Historical non-renewals number about 30 a year. At what point do you tell them a model is the wrong tool and a rules list is better?
• Which candidate features would you drop purely because you cannot date them?

State the measured impact of your own work honestly

Hard · impact · causal inference · self-assessment

You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.

Approach

1. The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
2. Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
3. Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
4. Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
5. State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
6. Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.

Follow-up

• Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
• What would have had to be true for the naive covered versus uncovered comparison to be valid?
• If the holdout costs the team real renewals, how do you justify asking for it, and to whom?