Interview guides · Engine

Engine Data Scientist Interview Questions & Guide 2026

Engine data scientist prep on data pitfalls: selection bias, target leakage, and checking an outlier's provenance before you drop it.

The 60-second brief

As a Data Scientist at Engine, you will play a pivotal role in transforming data into actionable insights that drive strategic business decisions. The importance of this role cannot be overstated; your analyses will influence product development, enhance user experiences, and ultimately contribute to the overall success of the organization. You will engage with complex datasets, utilizing advanced statistical techniques and machine learning models to uncover trends and patterns that can inform critical business strategies.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

Separate release-calendar spikes from underlying trend
Define a qualified stream and defend it
Split voluntary churn from involuntary payment failure

Overview

In this role, you will collaborate closely with cross-functional teams, including product managers, engineers, and marketing specialists, to solve challenging problems and optimize processes. Whether it’s improving product features based on user behavior analysis or developing predictive models to enhance customer engagement, your contributions will be integral to Engine's mission of delivering innovative solutions at scale. Expect to work on projects that require both technical acumen and creative problem-solving, making your work not only impactful but also intellectually stimulating.

The Engine interview process

Engine candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Interviews

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Where candidates lose points

Treating the account as the person, or the profile as the person.

A household account carries several people, profiles frequently are not switched, and shared-screen, car and speaker playback often lands on a default profile with no user behind it. Personalisation trained on a profile therefore learns a mixture, retention regressions attribute one member's behaviour to another, and a per-account taste statistic describes a household composite. The practical consequence is that apparent personalisation wins can be device or context effects, so any identity-level claim needs a stated unit and an acknowledgement of what that unit actually aggregates.

Comparing consumption week over week across the release calendar and the rights calendar.

A major release, a season drop or a live event produces a spike that dwarfs almost any treatment effect, and the effect is not confined to the new title because it pulls attention from everything else in the same window. Separately, licensed content leaves the catalogue when its window expires, so consumption falls with no product change and the drop is attributed to whatever shipped that week. Both need to be handled by an explicit control: a comparison period chosen for calendar equivalence, a covariate for scheduled releases, or a pre-registered rule for excluding a window, decided before the numbers are seen.

Naming a model class before naming the deployment constraints

Set out the latency budget, the label delay, the retraining cadence, the interpretability requirement and the number of labelled examples, then pick the model that fits them. A boosted-tree answer to a problem where each decision must be explained to the affected user is a well-executed answer to the wrong question.

Building features from data that postdates the prediction time

Check every feature against the timestamp at which the model would actually score, and drop anything computed from a window that includes or follows the label event. For a forecasting use case, split train and test by time rather than at random, and split by entity when the same entity recurs.

Questions & practice

Explain how you would optimize a machine learning algorithm for better performance.

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

What algorithm would you use for a classification problem and why?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Sessionise playback heartbeats into streams under an idle-gap rule

Hard · sessionisation · gaps-and-islands · pandas

heartbeats has profile_id, device_id, content_version_id, heartbeat_ts, playhead_seconds, is_paused and reported_at; clients emit one every 30 seconds while in the foreground. Collapse it into stream rows keyed on (profile_id, device_id, content_version_id) under a 300-second idle gap. Emit started_at, ended_at, max_position_seconds and played_seconds, where played_seconds counts advancing playback only and so excludes pause, seek and rebuffer time. Flag streams whose reported_at trails heartbeat_ts by more than an hour. Then re-run at a 60-second gap and report what moved.

Approach

Sort by the key and heartbeat_ts, take the per-row time delta inside the key with groupby().diff(), mark a new stream where that delta is null or above 300 seconds, and cumulative-sum the flag into a stream ordinal. This is gaps-and-islands, and it is O(n log n) after the sort with no per-group Python loop.
Compute played_seconds from consecutive heartbeats as min(max(playhead_delta, 0), wall_delta), summed inside the stream. The lower clamp discards a backward seek; the upper clamp is what stops a forward seek to the end of a forty-minute episode from crediting forty minutes of playback to one thirty-second interval. Note what this makes the column mean: a rewind and replay inside one stream counts the replayed interval twice, because played_seconds measures time spent advancing the playhead, not distinct coverage of the item.
Accept that the first heartbeat of each stream has no predecessor and contributes nothing, which undercounts by up to one heartbeat interval per stream. Quantify that bias as stream count times 30 seconds against the total instead of leaving it unsaid.
Take ended_at as the last heartbeat_ts, but keep it null when that heartbeat sits within one idle gap of the data cutoff: such a stream may still be running, and giving it an ended_at manufactures a completed stream.
For the offline flag, work on reported_at minus heartbeat_ts rather than on event_date, and report the distinct dates the flagged streams belong to, because that list is the recompute list.
Re-run at 60 seconds and report both the stream count and the change in total played_seconds: the count can only rise, and the total falls by exactly the contributions that now straddle a split.

Worked solution (40 min)

Sort by (profile_id, device_id, content_version_id, heartbeat_ts); compute wall_delta as the grouped heartbeat_ts diff in seconds and pos_delta as the grouped playhead_seconds diff.
new_stream = wall_delta.isna() OR wall_delta above 300; stream_ordinal = new_stream grouped-cumsum; group on the key plus stream_ordinal from here on.
contrib = np.minimum(np.maximum(pos_delta, 0), wall_delta), forced to 0 on the first row of each stream; played_seconds is contrib summed per stream.
Aggregate started_at as min(heartbeat_ts), ended_at as max(heartbeat_ts), max_position_seconds as max(playhead_seconds), and late_offline as max(reported_at minus heartbeat_ts) above 3600 seconds.
Null ended_at where it falls within 300 seconds of the data cutoff.
Re-run with the gap as a parameter at 60 seconds, and diff the two summaries on stream count and total played_seconds.

Expected result: One row per stream whose played_seconds never exceeds ended_at minus started_at in seconds, since each interval contributes at most its wall_delta and the wall_deltas inside a stream telescope to exactly that span. At a 60-second gap the stream count is greater than or equal to the 300-second count, and total played_seconds is lower by exactly the sum of the contributions that now straddle a split.

Verify it yourself

Assert played_seconds at or below (ended_at minus started_at) in seconds for every stream; any violation means the clamp against wall_delta was dropped.
No stream spans two content_version_ids or two device_ids. Group the heartbeats back by stream ordinal and check nunique equals 1 on both.
A hand-built three-heartbeat case with a backward seek (playheads 600, 300, 330 at 30-second spacing) yields played_seconds of 30, not 60 (the wall clock) and not -270 (the raw playhead deltas).
The distinct event_date partitions of the late_offline streams are exactly the days that need recomputing, and that set should be small and old rather than spread evenly across the window.
Do not assert played_seconds against an item's runtime. The heartbeats input carries no duration_seconds, and the bound does not hold even if you join one: a backward seek followed by a rewatch advances the playhead over the same interval twice inside one stream, and both passes are genuine playback. max_position_seconds is the column that compares against a joined runtime, and then only within the segment-overshoot tolerance of a second or two.

Follow-up

A client dies mid-playback and stops reporting. What does its stream look like under this rule, and what does that do to mean stream duration?
Which of these numbers determines a per-stream payout, and what does moving the idle gap do to somebody's money?
The same profile plays the same track on a phone and a TV at once. Your key separates them. When would you want it not to?

Describe a coding project you worked on and the challenges you faced.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

What factors would you consider when choosing a database for a specific application?

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Rebuffer ratio by point of presence, excluding start failures

Easy · ratio metrics · delivery quality · conditional aggregation

fct_stream carries cdn_pop, device_type, network_type, played_seconds, rebuffer_seconds and end_reason (enum including 'playback_error'). For streams with started_at in the trailing 24 hours, return one row per (cdn_pop, device_type, network_type) with two figures: rebuffer_ratio = SUM(rebuffer_seconds) / SUM(played_seconds), computed only over rows with played_seconds > 0; and start_failure_rate = the share of all rows in the cell with played_seconds = 0 and end_reason = 'playback_error'. Restrict output to cells with at least 500 streams and order by rebuffer_ratio descending.

Approach

Do it in one pass with conditional aggregation: SUM(rebuffer_seconds) FILTER (WHERE played_seconds > 0) over SUM(played_seconds) FILTER (WHERE played_seconds > 0), or the equivalent SUM(CASE WHEN ...) on engines without FILTER. Two separate scans risk the numerator and denominator disagreeing about which rows are in the cell.
Wrap the denominator in NULLIF(..., 0) so a cell in which every stream failed before first frame returns NULL rather than raising a division error — and so it is visibly absent rather than silently zero.
The failure rate's denominator is every row in the cell, including the zero-play ones. That is the whole point of the pair: the rows excluded from the ratio have to be counted somewhere, or a point of presence that fails before first frame reports a perfect ratio.
Apply the 500-stream minimum with HAVING COUNT(*) >= 500 before ordering, otherwise a cell with three streams and one stall tops the list at a ratio no one should act on.

Worked solution (15 min)

Write the cell aggregate exposing the raw components: COUNT(*), the two filtered sums, and the count of zero-play playback errors.
Derive both rates in an outer SELECT so the raw counts stay visible when a cell looks wrong.
Recompute the ratio ungrouped over the same filtered rows and confirm it equals the played_seconds-weighted mean of the cell ratios.

Expected result: One row per (cdn_pop, device_type, network_type) cell with at least 500 streams, carrying stream_count, rebuffer_ratio >= 0 (NULL where no row had played_seconds > 0) and start_failure_rate in [0, 1].

Verify it yourself

The ungrouped ratio equals the played_seconds-weighted mean of the per-cell ratios, not their unweighted mean.
Every cell whose start_failure_rate is exactly 1.0 has a NULL rebuffer_ratio.
COUNT(*) summed across all cells before the HAVING equals the unfiltered row count for the 24-hour window.

Follow-up

Two points of presence show the same rebuffer ratio but one has twice the start-failure rate. Which do you escalate, and what does the second number tell you that the first cannot?
How would you decide whether a cell's ratio moved beyond what day-to-day variation explains, given that cell sizes differ by orders of magnitude?

How would you approach a data analysis problem where the data is incomplete?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Given a dataset, how would you identify the key factors influencing a specific outcome?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Walk me through your thought process when designing a data-driven solution for a business challenge.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize your tasks when working on multiple projects?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a data pipeline for real-time analytics?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Design a switchback around a shared popularity feature

Hard · interference · switchback · spillover · clustered design

A ranker change boosts items using a global popularity feature computed from every account's streams over a trailing 24 hours. An account-randomised A/B is proposed. Treated accounts' plays feed that feature, so control accounts are shown items the treatment arm made popular. Design an experiment that identifies the effect on qualified hours. Specify the randomisation scheme, slice length, burn-in, the analysis unit and its standard error, and how you would measure the interference you cannot remove. The surface refreshes hourly and 74 percent of qualified hours fall between 17:00 and 24:00 local.

Approach

Name the violation: SUTVA fails because one account's assignment changes another's exposure through the shared feature. The account-randomised estimate is attenuated by an unknown factor, so a null cannot distinguish no effect from complete spillover.
Rule out cluster designs by reading the feature's partition key rather than by reaching for geography. Popularity computed globally has no containing cluster smaller than the whole population: a treated account in one territory moves the same feature value a control account in another territory reads. The general rule is that any cluster or saturation scheme must be randomised at or above the unit the shared feature is computed over, which here leaves only time.
Choose a switchback: randomise time, put the entire population on one arm per slice, and the shared feature becomes internally consistent within a slice. The price is that only outcomes resolving inside a slice are measurable, so this design reads hours and cannot read retention.
Set slice length against carryover rather than convenience. A 24-hour popularity window means each slice inherits up to 24 hours of the previous arm's influence, so burn-in must be at least 24 hours and a one-hour slice is meaningless. The engineering trade worth proposing is shortening the feature window for the duration of the test.
Block assignment against the daypart. With 74 percent of hours inside a seven-hour evening band, unblocked random slices will hand one arm more evenings than the other; randomise within blocks of complete days and verify each arm's realised share of evening and weekend hours.
Measure the residual spillover with a saturation ladder built on the same slice schedule, since time is the only unit above a global feature. Run some slices at intermediate treated shares and read control accounts' outcomes as a function of the slice's treated share: the slope is the spillover, and it is identified precisely because the feature is global.
Analyse at the slice, not the account. With tens of slices per arm, use a permutation test over the actual slice-assignment mechanism rather than a t-test on millions of correlated account rows, and allow for correlation between adjacent slices.

Worked solution (45 min)

Read the feature definition to confirm its partition and window rather than assuming: global versus per-territory decides whether cluster randomisation is even available, and whether a saturation design can be run on anything but time.
Cost the design under the native 24-hour window: a 24-hour burn-in forces 48-hour slices, so 28 days gives 672 / 48 = 14 slices, seven per arm, each contributing 24 measured hours. Write down explicitly that this is unreadable for anything but a very large effect.
Propose shortening the feature window to 4 hours for the test. A 4-hour burn-in then allows 8-hour slices: 672 / 8 = 84 slices over 28 days, each contributing 4 measured hours.
Allocate the 84 slices across treated shares rather than spending all of them on the two extremes: 24 slices at 0 percent treated, 24 at 100 percent, and 12 each at 25, 50 and 75 percent. The primary contrast is 0 versus 100 with 24 slices an arm instead of 42, which widens its MDE by sqrt(42 / 24) = 1.32. That 32 percent is the price of being able to measure the spillover at all rather than assuming it away.
Block the slices into consecutive day pairs so every treated share receives the same number of 17:00 to 24:00 bands and the same number of weekend bands, and verify the realised balance before unblinding.
Compute the slice-level outcome as qualified hours per active account inside each slice's measured portion, then estimate the 0 versus 100 contrast with a permutation test over 10,000 re-randomisations of the blocked assignment.
Regress control accounts' qualified hours per active account on the slice's treated share across the 0, 25, 50 and 75 percent rungs. The slope is the spillover the switchback exists to avoid, measured on the same schedule and the same metric.

Expected result: A blocked switchback rather than an account-randomised A/B. Under the native 24-hour feature window the design yields only seven slices per arm over 28 days and is not readable; shortening the feature window to 4 hours for the test gives 8-hour slices with a 4-hour burn-in and 84 slices over 28 days, analysed at the slice level with a permutation test. Twenty-four of those slices go to each extreme and 36 to intermediate treated shares, widening the primary contrast's MDE by a factor of 1.32 in exchange for a direct spillover estimate. The spillover is read as the slope of control accounts' hours against the slice's treated share, and it is measured on a time ladder rather than on territory clusters because a globally computed feature admits no saturation unit smaller than the whole population. The limitation stated up front is that this measures within-slice consumption and cannot measure retention.

Verify it yourself

An A/A switchback over the same slice schedule must reject at close to the nominal rate. Rejection above nominal means the burn-in is too short or adjacent slices are more correlated than the test allows.
Split each measured window into halves: after adequate burn-in the two halves must give the same estimate, and a gradient between them means carryover is still bleeding through.
Each arm's share of qualified hours inside 17:00 to 24:00 and its share of weekend hours must both sit within one percentage point of 50 percent across the realised schedule.
Control accounts' hours must be flat against the slice's treated share if the shared feature carries no spillover, and a positive slope across the 0, 25, 50 and 75 percent rungs is the spillover measured directly. Read a flat line only to the width of its confidence interval: 12 slices a rung is a diagnostic, not a precise estimate, so report the interval and say what magnitude of spillover it still admits rather than calling the null zero.
The ladder must sit on the same unit the feature is computed over. A saturation design run on territory clusters while popularity is global is uninformative by construction, because every cluster reads the same feature value whatever its local treated share, so a null there says nothing about spillover.

Follow-up

A stream starts at 07:55 and ends at 08:40 across a slice boundary. Write the rule, and say which direction your choice biases the estimate.
The change is also expected to move 30-day retention. Can this design say anything about that, and if not, what is the least-bad alternative?
Bound the attenuation in the account-randomised test instead: with 50 percent of accounts treated, what share of the popularity feature is treatment-derived, and what does that imply about the observed effect relative to the true one?

Involuntary churn share jumped after a dashboard rebuild

Medium · definition-change · churn · late-resolving-data

Involuntary share of gross churn — fct_subscription_period rows with renewal_outcome = 'failed_involuntary' over rows with renewal_outcome in ('cancelled_voluntary','failed_involuntary'), keyed on period_end_ts month — reads 31% for last month against 19% for the three months before. Billing reports no change in decline rates and dunning_attempts per failed period is unchanged. Using fct_subscription_period (period_end_ts, renewal_outcome, payment_status, dunning_attempts, cancel_requested_ts, billing_provider), determine whether involuntary churn actually rose. Deliverable: the corrected month-on-month series and the specific change that produced the step.

Approach

Check the month's maturity before anything else. Periods ending recently still have retries in flight, and a row reading 'failed_involuntary' today becomes payment_status = 'retried_paid' next week. Recompute every one of the last twelve months as of a fixed age — period_end_ts plus the maximum retry window — instead of as of today.
If the step survives age alignment, hunt for a definition change: diff the model or query against the version that produced 19%, and check specifically whether the month is now keyed on cancel_requested_ts rather than period_end_ts, and whether 'refunded' or 'chargeback' rows have entered one side of the ratio.
Settle it with one query rather than a narrative. Run the old definition and the new definition on the same month. Two numbers from one month under two definitions is decisive; two numbers from two months under one definition is not.
If a definition did change, restate history rather than leaving a step in the chart: run the new definition back across all twelve months and publish both series with the cutover marked, so nobody re-discovers this in a quarter.
Cut the horizon-closed series by billing_provider before concluding nothing moved. Card-direct, app-store and carrier billing have different retry mechanics and different success rates, so a mix shift between them moves the ratio with every provider's own rate flat.

Follow-up

How would you set the dunning horizon from the data instead of taking billing's stated maximum?
Under cancel-at-period-end, the cancellation request and the entitlement end fall in different months. Which timestamp keys this metric, and what does the other choice do to the curve?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Describe a time when you used statistical analysis to solve a real-world problem.

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

State the impact of your last year without inflation

Hard · self-assessment · attribution · counterfactual reasoning

You are asked what your work was worth over the last year. Two candidates are on the table. First: a dunning-schedule change you analysed, shipped in March with no holdout, after which the involuntary share of gross churn fell three points. Second: a metric-definition change you drove, which nobody can attribute revenue to. Deliverable: the impact claim you make for each, the counterfactual attached to each, and which one you lead with in a performance review — with the reason.

Approach

The probe is whether you apply the same causal standard to your own work as to a product experiment. Attach the counterfactual before the claim: involuntary share of gross churn responds to card-updater coverage, billing_provider mix, retry schedules and the dunning horizon you read it at, any of which can move three points with the schedule change contributing nothing.
Say which evidence would separate those, then say plainly which you actually have. A staggered rollout by billing_provider, a holdout, or at minimum a flat pre-period series would each support a different strength of claim; with none of them, the honest claim is a contribution to a favourable movement, stated as such.
Check the classification itself before claiming the movement is real: the split between cancelled_voluntary and failed_involuntary must be computed only after the dunning horizon has closed for every period in the month, or retries still in flight are counted as churn and the share reads too involuntary in one direction and corrects in the other.
Price your contribution rather than the outcome. The value of analysis that changed a decision is the decision's delta multiplied by the probability the decision would not have been taken without it, and being explicit about that second factor is most of what honesty means here.
Make the definition-change claim concrete rather than apologetic: name the decisions that would have been taken on the wrong number, the reports it reconciled, the recurring argument it closed, with dates. Then lead with whichever claim survives questioning, not whichever carries the larger number, because an inflated first claim makes the second unbelievable.

Follow-up

Your manager writes up the churn improvement and credits it to you. Do you correct it, and to whom?
How would you have designed the March rollout so that attribution was possible, at what cost in delay?
Name something you worked on last year that had no impact, and say what you learned from that rather than from the wins.

Defend a flat result on a flagship discovery launch

Medium · defending findings · statistical power · experiment readout

A new ranker ran to 50 percent of accounts for four weeks. Your read: qualified hours per active account-week is +0.4 percent with a 95 percent interval of [-0.9 percent, +1.7 percent]. Meanwhile the share of qualified streams with start_source = 'algorithmic_slate' rose six points and the share with start_source = 'search' fell five. The team reads the slate shift as the win and wants to ship. The launch review is Friday. Deliverable: a five-minute verbal position and the one table you put on the screen.

Approach

The probe is whether you can hold a position under social pressure without overclaiming in the other direction. Separate the two claims explicitly: the start-source shift is well measured and real; the hours effect the launch was justified on is not distinguishable from zero at this sample size. Those are different statements and only one is contested.
Do the power arithmetic before the meeting, using the observed per-account variance on a right-skewed hours metric. Arrive able to say "at this n we could not have detected less than X percent", which is a fact, instead of "it didn't work", which is an opinion the room can simply disagree with.
Show the substitution as arithmetic, not interpretation: decompose treatment hours by start_source and show slate hours rising by approximately what search hours fell. A start-source mix shift with no change in total is exactly what displacement looks like, and the decomposition sums to the total so there is nothing to argue about.
Refuse the symmetric overclaim. The interval's upper bound is commercially meaningful, so "the ranker does nothing" is not supportable either; say that out loud, because it is what makes the rest of your position credible.
Offer the falsifiable next step with a price: a longer read powered for the hours effect, or CUPED on a pre-period consumption covariate measured before assignment, with the required account-weeks stated. Put the disagreement on the pre-registered metric, not on the team's judgement.

Follow-up

The PM argues start-source mix is a leading indicator of retention. How would you test that claim, and how long would it take?
Suppose the interval had been [+0.1 percent, +1.9 percent]. Does your position change, and by how much?
The decision goes against your read and the ranker ships. What do you do on Monday?