System2 Data Scientist Interview Questions & Guide 2026

The 60-second brief

The role of a Data Scientist at System2 is pivotal in harnessing data to drive strategic decisions and enhance the user experience. As a Data Scientist, you will be tasked with analyzing complex datasets, developing predictive models, and translating data insights into actionable recommendations that influence product development and business strategy. Your work will directly contribute to optimizing our offerings and ensuring that our decisions are data-driven and aligned with user needs.

Nearly every loop contains a round whose deliverable is a recommendation to someone non-technical. Practise stating a conclusion, the confidence attached to it, and the cost of being wrong in each direction, because that triple is the artifact being graded.

Focus areas

• Size an experiment before anyone launches it
• Pick a randomisation unit that respects interference
• Decompose a metric move by segment and mix

Overview

At System2, the Data Scientist role is not just about crunching numbers; it involves collaborating across teams to identify opportunities for innovation and improvement. You will work closely with product managers, engineers, and other stakeholders to tackle challenging problems, such as user engagement forecasting and investment analysis. This role is crucial in shaping our products and strategies, making it both exciting and impactful.

Candidates can expect to engage in a variety of projects, ranging from statistical analysis to machine learning applications, all aimed at enhancing our service offerings. The complexity and scale of the data you will work with will provide a stimulating environment where your analytical skills can thrive.

The System2 interview process

System2 candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

HR Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

• Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
• Whether your reason for wanting the role points at the work itself rather than the company's reputation
• Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

• Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
• Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
• Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Case Study

A case round is decided by whether you leave the interviewer with a recommendation, not by how much analysis you narrate on the way there. The prompt is open on purpose, so the first job is to convert it into a decision someone could act on: ask what would be done differently depending on the answer. From there name the quantity that would settle it, state the assumptions you need, and commit. Candidates who cover more ground than anyone expected and still end on "it depends" score below candidates who scoped narrowly and said what they would do.

What to demonstrate

• Whether the version of the question you choose to answer is genuinely narrower than the prompt and still worth answering
• Whether the recommendation arrives as an action with a number attached, rather than as a summary of what you looked at
• Whether assumptions are stated at the moment you rely on them, instead of collected into a disclaimer at the end
• Whether you notice when a branch you are exploring would not change the decision either way

How to prepare

• Take six open prompts and write only the scoping move for each: the one-sentence question you would actually answer and the decision it feeds. Give yourself three minutes per prompt and stop there.
• Put a five-minute warning into every practice case and force a closing statement that names the action, the result that would justify it, and the result that would reverse it.
• Record one case and count how long you talked before naming a measurable quantity. Past roughly five minutes, what you are calling scoping is narration.

Where candidates lose points

Treating last-touch attribution as the causal value of a channel

The attribution label on dim_user is the output of a rule that assigns full credit to whichever touch happened to be recorded last inside a lookback window, and that rule systematically rewards channels that sit close to the conversion, especially branded search and retargeting, which largely intercept demand that already existed. Reallocating spend on those labels moves budget toward the channels that are best at being last, which is why attributed return on ad spend often improves while total signups do not. Nothing in the touchpoint data can settle this, because the counterfactual of not running the channel was never observed. The credible reads are a geo holdout or a scheduled pause, sized in advance on the total-signups metric rather than on the attributed one, and the honest framing in the meantime is that the label describes correlation with conversion and not incremental contribution.

Watching an experiment daily and stopping when it crosses significance

A fixed-sample test controls type I error at one pre-declared look. Checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate to roughly 0.15 to 0.20 for ten looks, and it rises further with more frequent checks, because the p-value takes a random walk that will eventually dip below the threshold under the null. The usual defences are a fixed horizon declared before launch, group-sequential boundaries such as O'Brien-Fleming that spend alpha across a planned number of looks, or always-valid confidence sequences that are correct under continuous monitoring. Compounding it, the effect size reported conditional on having crossed the threshold is biased away from zero, and the bias is larger the lower the power was, so an underpowered test that 'won' typically overstates the lift it found.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

What steps would you take to improve the accuracy of a predictive model?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

Describe a machine learning model you have implemented and its impact.

Medium · machine learning and modelling

Approach

1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Permutation test for a difference in conversion rates

Medium · permutation test · resampling · inference

Write a two-sided permutation test from scratch for a difference in conversion rates, using no scipy hypothesis function. Input: a DataFrame with unit_id, variant in {control, treatment} and converted in {0,1}, one row per randomisation unit. Compute the observed difference in proportions, then build the null distribution by reshuffling the variant labels while holding each arm's size fixed. Report the p-value as (1 + the count of permuted statistics at least as extreme in absolute value) / (B + 1) with B at least 10,000, and return the permutation distribution.

Approach

1. Name the null being tested: the sharp null that each unit's outcome is the same under either label. That is what licenses permuting labels, and it is stronger than the null of equal means, which matters when someone asks whether the test is valid under unequal variances.
2. Extract converted to a single numpy array of 0s and 1s and record n_treatment. Every permutation is then just a reshuffle of one array, and the treatment mean is the mean of the first n_treatment entries of the shuffled array.
3. Vectorise the B permutations with rng.permuted on a tiled 2-D array, or with argsort of a (B, n) random matrix. A Python loop calling np.random.shuffle B times is correct but roughly an order of magnitude slower and often runs past the time limit.
4. Use the +1 correction in both numerator and denominator. Without it a p-value of exactly 0 is reportable, which is false: the observed labelling is itself one of the permutations, so the smallest attainable p-value is 1/(B+1).
5. Compare the resulting p-value against a two-proportion z-test as a sanity check. At these sample sizes they should agree closely; a large divergence means the statistic or the shuffle is wrong, not that the permutation test found something subtle.

Worked solution (25 min)

1. y = df['converted'].to_numpy(); n_t = (df['variant'] == 'treatment').sum(); obs = y[treat_mask].mean() - y[~treat_mask].mean().
2. Build the null: for B draws, shuffle y and take the mean of the first n_t entries minus the mean of the rest.
3. p = (1 + (np.abs(null) >= abs(obs) - 1e-12).sum()) / (B + 1), with the small tolerance so exact ties count as at least as extreme.
4. Return obs, p and the null array; plot or describe the null to confirm it is centred at 0.

Expected result: The observed difference, a p-value in [1/(B+1), 1], and a null distribution whose mean is approximately 0 and whose standard deviation approximately matches the analytic pooled standard error sqrt(p_pool*(1-p_pool)*(1/n1 + 1/n2)).

Verify it yourself

• The null distribution's mean is within a few Monte Carlo standard errors of 0; if it is centred on the observed difference, the code bootstrapped instead of permuted.
• The null's standard deviation agrees with the analytic pooled standard error to within a couple of percent.
• The p-value agrees with a hand-computed two-proportion z-test p-value to within about 0.01 at these sample sizes.
• Setting converted identical in both arms by construction yields a p-value near 1, and a planted large effect yields the floor of 1/(B+1).

Follow-up

• The arms are 200 and 20,000 units. Does the permutation test stay valid, and what happens to its resolution at B = 10,000?
• Give a 95 percent confidence interval for the difference. Can you get it from this permutation distribution, and if not, what would you run instead?
• The randomisation unit is user_id but the outcome is per session. What breaks, and what is the fix?

Seven-day activation rate by weekly signup cohort

Easy · cohorts · activation · date-bounds

dim_user holds user_id, account_created_at_utc, is_internal. fct_event holds user_id, occurred_at_utc, is_core_action. A user is activated when core-action events fall on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). Return, for the last twelve complete weekly signup cohorts, the cohort week, cohort size, activated users and the activation rate. Exclude is_internal users. Every signup in the cohort week stays in the denominator, including users who never returned.

Approach

1. Start from dim_user as the denominator spine with is_internal = FALSE and DATE_TRUNC('week', account_created_at_utc) as the cohort key. Driving the query from the event table instead would silently condition on having events and delete the entire non-activating population.
2. Join fct_event on user_id with is_core_action = TRUE and a per-user bound, occurred_at_utc >= u.account_created_at_utc AND occurred_at_utc < u.account_created_at_utc + interval '7 days'. The bound is correlated to each user's own signup timestamp, not a single global date range.
3. Aggregate per user with COUNT(DISTINCT occurred_at_utc::date) >= 2, then LEFT JOIN that back onto the spine and COALESCE the flag to FALSE so non-activators contribute a zero rather than vanishing.
4. Restrict the published cohorts to those whose week ended at least eight days ago. A cohort younger than that has not finished its seven-day window, so its rate is mechanically low and reads as a decline.
5. Roll up by summing the numerator and denominator per cohort week, and state the two-distinct-days threshold next to the number since it is a choice that re-bases the whole history if changed.

Worked solution (20 min)

1. Write the cohort spine and confirm its total equals the count of non-internal signups in the date range.
2. Write the per-user distinct-active-days CTE with both interval bounds and inspect a handful of users manually.
3. LEFT JOIN, COALESCE the flag, aggregate to cohort week.
4. Apply the eight-day publication lag and drop the incomplete cohort.
5. Re-run with a closed upper bound (<= +7 days) and note how many users change state, to show the boundary is doing work.

Expected result: Twelve rows, one per cohort week, each with cohort_size, activated and rate. Cohort sizes sum to the total non-internal signup count across those twelve weeks. No cohort is younger than eight days.

Verify it yourself

• activated is never greater than cohort_size in any row.
• A user with three core actions on a single day is not counted as activated.
• Removing the LEFT JOIN in favour of an inner join raises every rate, which demonstrates the denominator was being conditioned on survival.
• Cohort sizes sum to an independent COUNT(*) over dim_user for the same range.

Follow-up

• Why two distinct days rather than one event? What happens to the published history if someone changes it to three?
• Invited seats and SSO-provisioned users get an account_created_at_utc at provisioning and may never sign in. Should they be in this denominator?
• The rate rose 3 points this week. What do you check before believing it?

Pair flow starts to completions within a thirty-minute bound

Medium · event-streams · conditional-aggregation · flow-funnel

fct_event holds event_id, flow_id, flow_instance_id, event_name, occurred_at_utc, surface, app_version. Using only rows where event_name is 'flow_started', 'flow_completed' or 'error_shown', compute the daily core-flow completion rate by surface and app_version. Numerator: distinct flow_instance_id whose flow_completed occurs within 30 minutes of its flow_started with no error_shown for the same instance in between. Denominator: distinct flow_instance_id with a flow_started that day. Report rows with a NULL flow_instance_id as a separate coverage count, not inside either side.

Approach

1. Collapse the stream to one row per flow_instance_id using conditional aggregation: MIN(occurred_at_utc) FILTER (WHERE event_name = 'flow_started') AS started, MIN(...) FILTER (WHERE event_name = 'flow_completed') AS completed, MIN(...) FILTER (WHERE event_name = 'error_shown') AS first_error. This makes the 'first' semantics explicit and avoids a self-join entirely.
2. Apply the predicates on that single row: completed IS NOT NULL AND completed <= started + interval '30 minutes' AND (first_error IS NULL OR first_error > completed). Note the last clause encodes 'in between' literally, so an error surfaced after a successful completion does not disqualify the attempt.
3. Bucket by started::date, surface and app_version. Splitting by app_version is not decoration: a completion-rate regression is nearly always confined to one client build, and a pooled daily rate hides it behind the installed base.
4. Count NULL flow_instance_id rows as a separate coverage figure. They cannot be attributed to an attempt at all, so they belong in neither numerator nor denominator, and their share tells you how much of the rate is unmeasurable.
5. For any roll-up to week or to all surfaces, re-sum the numerator and denominator rather than averaging the daily rates.

Follow-up

• occurred_at_utc is client-supplied. What do you do with an instance whose completion timestamp precedes its start?
• flow_instance_id is missing on 4 percent of starts for one app_version. What can and cannot you conclude about that build?
• An instance starts at 23:55 and completes at 00:04. Which day owns it, and what does the alternative do to the daily series?

How do you prioritize tasks when working on multiple projects?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would you approach forecasting sales for a new product?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Fix the population and the time window before naming any metric.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Analyze the following data and propose potential investment opportunities.

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Given a dataset on user behavior, how would you identify key trends?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Fix the population and the time window before naming any metric.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

What is A/B testing, and how is it conducted?

Medium · a/b testing and experimentation

Approach

1. Say whether units interfere with each other, and switch design if they do.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• How would you handle interference between treated and control units?
• What would you do if you could not randomise at all?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

1. Say what you would check first and why it is the highest-information step.
2. State your assumptions explicitly before working the problem.
3. Clarify what is being asked and what a complete answer would contain.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Measure a paid channel without a user-level experiment

Hard · difference-in-differences · geo holdout · incrementality · clustered inference

Finance needs the incremental contribution of paid social before next year's budget. It carries 18% of last-touch-attributed signups in dim_user.last_touch_channel, but the ad platform will not randomise users and unexposed users are unobservable. The ad platform targets at market level, your warehouse resolves a market label onto fct_session, and 40 markets are eligible with 12 months of weekly signups derivable from dim_user. Design the read, size it on total signups rather than attributed ones, and state the identifying assumption and the check that would falsify it.

Approach

1. Say why the attributed number cannot answer the question. Last-touch assigns full credit to whichever touch was recorded nearest the conversion, which systematically rewards channels that intercept demand that already existed, and the counterfactual of not running the channel is nowhere in the touchpoint data.
2. Choose a geo holdout: randomise markets into a paused arm and a spend-as-usual arm, and measure total signups per market. The metric has to be the total, because the question is whether demand is created or merely relabelled, and attributed signups fall to zero in a paused market by construction.
3. Size on markets, since the market is the randomisation unit and therefore the variance unit. Forty markets give at most twenty per arm, so form matched pairs on twelve months of pre-period weekly signups and randomise within pair, which removes the between-market level differences that would otherwise dominate the residual variance.
4. Fix the identifying assumption in plain words: absent the pause, treated and control markets' signup trends would have moved in parallel. Falsify it by running the same estimator on placebo cut dates inside the pre-period; a non-zero placebo estimate or visibly diverging pre-trends kills the design as specified.
5. Estimate with market and week fixed effects and cluster standard errors on market. Below about forty clusters the cluster-robust error is biased downward, so use a wild cluster bootstrap or randomisation inference over the pair assignment. If only one or two markets can be paused, switch to synthetic control: weight untreated markets to match the treated market's pre-period path and use in-space placebos for inference.

Worked solution (40 min)

1. Aggregate weekly signups per market for twelve months from dim_user with is_internal = FALSE, and drop markets whose history is too short or too volatile to pair.
2. Rank markets on pre-period mean and trend, form twenty matched pairs, and randomise one of each pair into the paused arm.
3. Derive the MDE from the between-pair residual standard deviation of weekly signups, not the within-market one, because power is set by the number of pairs; express it as a percentage of total signups and compare it with the effect finance cares about.
4. Run placebo estimates at three pre-period cut dates before committing; re-pair or abandon if any is significant.
5. After the pause, estimate difference-in-differences with market and week fixed effects, report a wild cluster bootstrap interval, and convert the point estimate into incremental cost per signup against the spend withheld.

Expected result: A matched-pair geo holdout across twenty pairs, read on total signups with market and week fixed effects and market-clustered inference, yielding an incrementality ratio of incremental signups to attribution-credited signups. The identifying assumption is parallel counterfactual trends, checked with pre-period placebo cuts. The MDE is set by twenty pairs rather than by millions of sessions, and is usually far coarser than the requester expects.

Verify it yourself

• Pre-period placebo estimates should centre on zero with intervals covering it. A systematically non-zero placebo means the pairing failed and the main estimate inherits that bias.
• Treated plus control market signups in the pre-period must reconcile with the total the finance dashboard reports, otherwise the market label is silently dropping sessions.
• An incrementality ratio above 1 implies the channel drives conversions it never touched. That is possible through view-through effects the model never sees, but it demands a stated mechanism rather than a shrug.

Follow-up

• Campaign commitments mean the pause starts in different weeks in different markets. What does staggered timing do to a two-way fixed effects estimate when effects vary across cohorts?
• How long do you pause, given that paid social has a delayed conversion tail, and how do you keep that tail out of the pre-period?
• The estimate says the channel is 40% incremental. What happens to the last-touch number on the dashboard after that?

Separate a definition edit from a behaviour change

Easy · definition-change · metric-versioning · cohorts

Seven-day activation rate is documented as core actions on at least 2 distinct UTC dates inside the first seven days after account_created_at_utc, and every cohort published through last week was computed on that definition. The dashboard now reads 38% for the signup cohort that started Monday, against 31% for the cohort before it. Two things landed in that week: an onboarding change, and a commit to the metric's SQL that you have not read yet and that came with no documentation update. You have dim_user, fct_event and the metric's version history. Deliver two numbers: how many of the 7 points are behaviour and how many are the definition. Then say which published cohorts are no longer comparable.

Approach

1. Diff the metric SQL across the commit boundary and enumerate every changed predicate individually: the distinct-day threshold, the is_internal exclusion, the is_core_action source, the window bounds, and the identity key. A single commit routinely changes more than one, and each has its own sign.
2. Dual-run both definitions over the same 26 cohort weeks. The definition effect is the level shift between the two series computed on identical data, which isolates it from anything that happened in the world.
3. Read the behaviour effect only from the old-definition series, comparing the new cohort to its own trailing cohorts. The new-definition series cannot answer the behaviour question because it has no pre-period.
4. Explain the mechanism of the changed predicate rather than only its size. Lowering a distinct-day threshold from 2 to 1 admits every user who acted once in a single session, which is a large and low-intent population, so a several-point jump is the expected magnitude and not a surprise.
5. State the restatement plan: pick one definition, backfill the full history on it, and annotate the break date. A series carrying two definitions is not a time series.

Follow-up

• The onboarding team wants the 1-day threshold kept because it is easier to move. What is your argument, and what would change your mind?
• How would you make a definition change visible to every consumer of this metric without relying on people reading a changelog?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

Tell me about a time you faced a significant challenge in a project. How did you handle it?

Medium · behavioural and stakeholder questions

Approach

1. Name the disagreement or constraint, and how you resolved it with evidence.
2. Close with what you would do differently, concretely.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• What would you do differently if you ran that project again?
• What did you decide not to do, and why?

Quantify your own impact without claiming the topline you touched

Hard · self-assessment · attribution · communication

You are writing the impact section of your own review. Over the year you ran four experiments, one of which shipped and three of which were flat; you corrected the definition of gross monthly revenue churn so that cancellation is recognised at period_end_utc; and you built a self-serve funnel dashboard. Weekly active accounts rose 14% over the same period. Your reviewer knows the data well. Write the three impact claims you would defend, stating for each what you contributed, what evidence supports it, and what portion of the outcome you are not claiming.

Approach

1. Recognise what is being probed: whether you apply to your own work the causal standard you would apply to somebody else's roadmap claim. Nearly everyone who would reject 'accounts that do Y retain better' will write 'I drove a 14% increase' without noticing it is the same error with a friendlier subject.
2. Sort the work by the kind of evidence it can carry. The shipped experiment is the only item with a randomised estimate, so it is the only one where an effect size is defensible, and you claim the interval rather than the point estimate.
3. Claim the three flat experiments as decisions prevented and price them. Features not built, or built differently, on evidence, with the engineering weeks reallocated as the number somebody else can verify. A defensible null is a delivered decision and should be written as one.
4. Claim the definition fix as correctness, not as improvement. The old figure was overstated by a specific percentage and appeared in a specific set of recurring documents; the impact is the change it produced in the forecast built on top of it, not a change in churn itself.
5. Claim the dashboard on usage and displacement: distinct weekly users of it, and the ad-hoc request count for six months before against six months after. If the request log does not exist, record the claim as unverified rather than estimating it upward.
6. Disclaim the 14% explicitly and once. State that it cannot be separated from seasonality, other teams' launches and a pricing change, and bound your own contribution from above using the shipped experiment's interval converted into headline units.

Follow-up

• Your shipped experiment's interval was +0.2pp to +1.4pp on activation. How much of the 14% can that account for, and how do you say so without undercutting yourself?
• A peer in the same cycle claims the full 14%. What, if anything, do you do about it?
• If you could only keep two of your three claims, which do you drop, and why that one?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

1. Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
2. Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
3. Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
4. Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
5. Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

• The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
• Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
• Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?