Interview guides · Abm Industries

Abm Industries Data Scientist Interview Questions & Guide 2026

Data scientist interview prep for Abm Industries: original practice on experiment design, SQL and metric reasoning, with worked solutions.

The 60-second brief

As a Data Scientist at Abm Industries, you will play a pivotal role in leveraging data to drive strategic decision-making and enhance operational efficiency. This position is essential for translating complex datasets into actionable insights that support various teams across the organization, from operations to marketing. Your work will directly impact how Abm Industries utilizes data to optimize services, improve customer satisfaction, and streamline processes, ultimately contributing to the company's success in a competitive landscape.

Product-sense cases reward reasoning from a mechanism to a testable prediction. Reciting every metric you can name reads as pattern matching; naming the single quantity that would move if your explanation were true reads as thinking.

Focus areas

Diagnose rebuffering by device, network and POP
Define a qualified stream and defend it
Split voluntary churn from involuntary payment failure

Overview

In this role, you will be involved in a variety of projects, including predictive modeling, data analysis, and machine learning applications. You will collaborate with cross-functional teams to identify opportunities for data-driven improvements and develop solutions that enhance the overall performance of the business. The complexity and scale of the data you will work with are significant, providing you with the chance to engage in innovative problem-solving and contribute to meaningful outcomes that affect both internal stakeholders and external clients.

The Abm Industries interview process

Abm Industries candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Interviews

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Live Coding

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Case Studies

Underneath the business framing, this round is usually asking whether you can turn a fuzzy goal into a quantity that could be computed from data such a business would plausibly hold. That means a metric with a stated numerator, denominator, eligibility rule and time window, plus an honest account of the conditions under which it would mislead you. Answers come apart when a candidate names a familiar metric and never defines it, because every follow-up then lands on an ambiguity that was left open and the candidate has to invent the definition under pressure.

What to demonstrate

Whether a named metric arrives with its denominator, eligibility rule and window attached rather than assumed
Whether the measure follows from the mechanism you proposed, or is a recognisable metric retrofitted to it afterwards
Whether you name a guardrail that would reveal the gain came from somewhere you did not want it to come from
Whether you can say what data the plan requires and what you would settle for if that logging were never implemented

How to prepare

Take five metrics you reach for by reflex and write each as one sentence containing numerator, denominator, eligibility rule and time window. The ones you cannot finish are the ones that will fail under follow-up.
For a product you use daily, write the measurement plan you would propose for a change to it: primary metric, one guardrail, the unit of analysis, and the table the numbers would come from.
Practise the substitution question. For three metrics you like, write what you would measure instead if the event you depend on were not being logged.

Behavioral Questions

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Where candidates lose points

Counting plays without a qualification threshold, or changing the threshold without restating history.

Playback arrives as heartbeats, so a play only exists once you decide what counts, and the common 30-second convention is not a neutral analytics choice: in music it is also the boundary at which a play becomes payable, which makes the warehouse definition a payout definition. The threshold interacts violently with content length, so a catalogue of three-minute tracks and one of forty-minute episodes move in opposite directions when you change it, and a skip-heavy surface can add plays while adding no hours. Any metric mixing pre-threshold and post-threshold counts, or pooling short-form and long-form on a per-stream basis, moves by double digits for reasons that have nothing to do with the product.

Comparing consumption week over week across the release calendar and the rights calendar.

A major release, a season drop or a live event produces a spike that dwarfs almost any treatment effect, and the effect is not confined to the new title because it pulls attention from everything else in the same window. Separately, licensed content leaves the catalogue when its window expires, so consumption falls with no product change and the drop is attributed to whatever shipped that week. Both need to be handled by an explicit control: a comparison period chosen for calendar equivalence, a covariate for scheduled releases, or a pre-registered rule for excluding a window, decided before the numbers are seen.

Dropping rows with missing values without naming the mechanism

Say whether the values are missing at random, missing by a known process, or missing in a way that depends on the outcome, and handle them accordingly. Deleting incomplete rows silently redefines the population whenever missingness correlates with what you are measuring.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Questions & practice

Describe a machine learning project you have worked on and the outcomes.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

How would you implement a linear regression model in Python?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Permutation test for hours per account across two ranker arms

Medium · permutation-test · heavy-tails · experiment-analysis

arm_hours holds one row per account: account_id, arm in {control, treatment}, qualified_hours over a seven-day window. Roughly 40 thousand accounts per arm, about 38 percent of them at zero hours, and the non-zero tail is long. Without calling a library test function, write a permutation test on the difference in mean hours with 10,000 relabellings. Then run it as two parts: the difference in the share of accounts with any hours, and the difference in mean hours among accounts with hours. Report all three and say which belongs in the readout.

Approach

Shuffle the labels, not the data. Draw a permutation of the arm indicator over accounts, which is the unit that was randomised, and hold the hours vector fixed.
Make each replication O(n): precompute the grand sum and the arm sizes, so a shuffled difference is the treated subset sum over n_t minus (grand sum minus that subset sum) over n_c. Ten thousand replications then take seconds instead of a minute.
Use the two-sided p-value (1 + count of permuted absolute differences at or above the observed) divided by (B + 1). The plus one is not cosmetic: it makes the p-value valid rather than optimistic, and it means the smallest reportable value here is 1/10001, not zero.
For the two-part version, run the same machinery on the 0/1 indicator for the rate, then on the non-zero subset for the conditional mean, and say plainly that conditioning on a post-treatment outcome breaks the randomisation, so the conditional arm is descriptive rather than causal.
Report the rate test and the overall mean test as the result, with the conditional mean as colour, and give the effect size in hours beside each p-value, because at 80 thousand accounts almost anything is detectable.

Worked solution (30 min)

Pull hours into a float array and arm into a boolean array; record n_t, n_c, the grand sum and the observed difference of means.
Loop B = 10,000 times, taking np.random.permutation(n)[:n_t] as the treated index set and accumulating hours[idx].sum().
Convert each shuffled treated sum into a difference of means, compare its absolute value against the observed, count, and apply the plus-one form.
Repeat with hours replaced by (hours > 0).astype(float) for the rate test.
Subset to hours above zero, re-run for the conditional mean, and assemble a three-row result carrying effect, p-value and the unit each is defined on.

Expected result: Three rows: the mean-hours difference with a permutation p-value bounded below by 1/10001; the streaming-rate difference in percentage points with its own p-value; and the conditional mean difference among non-zero accounts, labelled descriptive because it conditions on a post-treatment outcome.

Verify it yourself

The permutation distribution of the mean difference is centred on zero within its own Monte Carlo error. A centre away from zero means the labels were not actually being reshuffled between replications.
The permutation p-value on the rate agrees to two decimal places with a two-proportion z-test, which is a fair cross-check because a 0/1 indicator is not heavy-tailed.
Doubling B leaves the p-value stable to within roughly 1 over sqrt(B). A p-value that moves a lot between runs is not yet resolved.
Multiplying every hours value in both arms by 60 leaves all three p-values identical under the same seed and scales only the effect sizes, since a permutation test is invariant to a common positive rescaling.

Follow-up

The permutation p-value on the mean is 0.03 and the rate test is flat. What is the most likely explanation, and does it change the decision?
How would CUPED on pre-period hours change your power here, and what would disqualify a covariate?
Accounts are households. Does that affect the validity of this test, or only its interpretation?

Paid retention triangle with monthly and annual curves kept apart

Medium · cohorts · retention · subscriptions

fct_subscription_period has subscription_period_id, account_id, period_index, billing_interval, period_start_ts, period_end_ts, payment_status, cancel_requested_ts and renewal_outcome. dim_account has account_id, first_paid_ts and is_test_account. Build a paid-retention triangle: label non-test accounts with the calendar month of first_paid_ts, and for month offsets 0 through 6 report the share of the cohort still holding a period with payment_status in ('paid','retried_paid') that contains the instant first_paid_ts + offset months. The anchor is each account's own first payment, not the first day of the offset calendar month. Report monthly and annual billing_interval as separate curves. Exclude cohorts not yet mature at offset 6.

Approach

Take the cohort key from dim_account.first_paid_ts, not from MIN(period_start_ts). The period table includes trial periods as zero-amount rows with payment_status = 'trial_no_charge', so the minimum start shifts trialling accounts a cohort early and inflates the youngest cohort's offset-0 denominator.
Anchor every offset on that same first_paid_ts; the calendar month is only the row label on the triangle. Anchoring survival on the first day of the offset month instead breaks offset 0 for everyone who did not first pay on the 1st — an account that first paid on the 20th holds no paid period covering that month's 1st, so its offset-0 cell reads as churned and the curve rises from offset 0 to offset 1. A retention chart whose first step goes up is almost always this. Two conventions to write down while you are here: PostgreSQL clamps timestamp + INTERVAL '1 month' at month end, so an account first paid on the 31st is tested on the 28th or 30th at some offsets; and the whole construction assumes first_paid_ts falls inside its own first paid period, which fails if the platform charges shortly before period_start_ts, in which case anchor on that period's period_start_ts instead.
Build the offsets 0..6 as an explicit ordered set and cross-join it to the cohort list. A grid built by aggregating survivors alone loses any (cohort, offset) cell with no survivors, so a collapsing cohort reads as a missing row instead of a zero.
Test survival as interval containment, not date equality: EXISTS a period for that account with payment_status in ('paid','retried_paid') where first_paid_ts + offset months falls inside [period_start_ts, period_end_ts). One annual row contains seven consecutive anniversaries on its own; an equality join against period_start_ts finds it at offset 0 and nowhere else.
Partition everything by billing_interval and never pool. An annual account has had no opportunity to churn before day 365, so pooling makes an annual-heavy cohort read as retentive when what it actually is, is un-renewed. billing_interval lives on the period and not on the account, so take the cohort's interval from the period containing first_paid_ts and hold it fixed for all seven offsets; an account that switches monthly to annual at offset 4 otherwise appears in both curves and is counted twice in the denominators.
Apply maturity last, and against the anchor rather than the label: keep a cohort only when the last day of that cohort month plus six months is at or before the last fully closed day in the data, so every account in it has actually reached its offset-6 anniversary. Otherwise the newest cohorts print 0% at the far offsets for a reason that is entirely calendar.

Follow-up

An account migrates plan mid-cohort with renewal_outcome = 'migrated_plan'. Retained or churned, and what does your choice do to the revenue story told beside this chart?
A period fails and a retry succeeds as a new row. Does your survival test find the account at that offset, and should it?
Can you produce a curve for annual accounts that is comparable to the monthly one before twelve months have elapsed, and what do you give up?

Home-row impressions that never converted, and an empty result

Medium · anti-join · null semantics · attribution

fct_impression has impression_id, profile_id, content_version_id, surface, slate_position, rendered_at and viewport_visible_ms. fct_stream has stream_id, profile_id, content_version_id, started_at, is_qualified and a nullable impression_id, populated only when the play started from a rendered slate. For surface = 'home_row' impressions rendered yesterday with viewport_visible_ms > 0, return content_version_id and the number of impressions that produced no qualified stream within 30 minutes. A colleague's draft filters WHERE impression_id NOT IN (SELECT impression_id FROM fct_stream) and returns zero rows on data that plainly contains misses. Explain why, then write the correct query.

Approach

Name the mechanism rather than the symptom. fct_stream.impression_id is nullable, so the subquery returns a set containing NULL. In SQL's three-valued logic x NOT IN (..., NULL) evaluates to UNKNOWN whenever x matches nothing else, never to TRUE, so the WHERE clause admits no rows. This is correct engine behaviour on correct data.
Fix with NOT EXISTS, which is a correlated existence test and is NULL-safe by construction, or with LEFT JOIN ... WHERE s.impression_id IS NULL. Adding IS NOT NULL to the subquery also works but leaves the same landmine armed for whoever edits the query next.
Put is_qualified = true and the 30-minute window inside the join or the correlated predicate, never in an outer WHERE over a LEFT JOIN. An outer filter on a right-side column turns a LEFT JOIN back into an inner join and returns the exact complement of the set you were asked for.
Match on impression_id, not on (profile_id, content_version_id). The second is a many-to-many between two event tables and will count one stream against every impression of that title for that profile.
Keep viewport_visible_ms > 0. A row that was never scrolled into view is not a recommendation that failed; it is a recommendation that was never made, and including it depresses every title's conversion by whatever share of the row went unseen.

Worked solution (20 min)

Prove the mechanism in one line: SELECT COUNT(*) FROM fct_stream WHERE impression_id IS NULL. A non-zero count is the entire explanation and should be shown before the rewrite.
Rewrite as NOT EXISTS, with is_qualified and started_at BETWEEN rendered_at AND rendered_at + INTERVAL '30 minutes' inside the correlated predicate.
Compute eligible and converted impression counts per content_version_id separately; the requested answer must be their difference.
Spot-check one high-count content_version_id by listing its impressions alongside any streams sharing those impression_ids.

Expected result: Rows of (content_version_id, unconverted_impressions) where, for every title, unconverted + converted equals the count of eligible home_row impressions with viewport_visible_ms > 0 rendered yesterday.

Verify it yourself

The NOT IN version returns zero rows while NOT EXISTS returns a non-empty result on the identical filtered data.
converted + unconverted reconciles to the eligible impression count for every content_version_id, with no title missing from either side.
Every impression counted as converted has a matching fct_stream row with is_qualified = true and started_at inside the 30-minute window.

Follow-up

Two home rows on the same screen render the same title, and the stream carries one impression_id. Should the other impression count as a miss? Defend whichever rule you pick.
Would NOT EXISTS and LEFT JOIN ... IS NULL produce the same plan here, and is there a case where you would prefer one over the other?
What attribution window between rendered_at and started_at do you allow, and why does an unbounded window overstate conversion rather than merely add noise?

How would you approach a dataset with thousands of features?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Present a case where you derived insights from ambiguous data.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Describe how you would evaluate the success of a new data-driven initiative.

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Given a specific business problem, outline the steps you would take to analyze it.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

What are the differences between supervised and unsupervised learning?

Medium

Approach

Work from the decision backwards to the evidence you would need.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer would contain.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Decide a rights renewal on retention, not on plays

Medium · metric design · catalogue economics · displacement · causal comparison

A licensed bundle from one rights_holder_id has rights_expires_at in 60 days and renewal is expensive. Finance wants a number that decides it. You have dim_content_version (content_version_id, content_id, rights_holder_id, license_type, rights_expires_at, available_territories), fct_stream (account_id, content_version_id, played_seconds, is_qualified, started_at, play_territory) and fct_subscription_period (account_id, period_end_ts, renewal_outcome, net_amount_usd). Define the metric you would put in the renewal memo, the comparison that identifies it, and the assumption it rests on. Plays on the bundle is not an acceptable answer.

Approach

Reject the play count with a reason, not a slogan: plays measure what the bundle drew, and the renewal question is what would be lost if it went, which differs by whatever those accounts would have watched instead. A bundle can be the third most played thing in the catalogue and still be worth nothing at the margin if its audience substitutes freely.
Define the account-level exposure measure the analysis runs on: share of an account's qualified hours in the trailing 90 days falling on content_version_id rows whose rights_holder_id is the bundle's, which turns a catalogue fact into an account attribute that can be related to renewal_outcome.
State the metric for the memo: incremental retained subscription revenue, estimated as the difference in the rate of renewal_outcome = 'renewed' between exposed accounts and matched unexposed accounts, multiplied by net_amount_usd per period, summed over exposed accounts, and compared directly against the renewal cost.
Name the identification problem instead of burying it: exposure is chosen, so heavy users of this bundle are heavier users generally and would renew at a higher rate with or without it. Match on pre-period total qualified hours, tenure, plan_tier and billing_interval, and state that residual confounding on taste remains and biases the estimate upward.
Use the natural experiment the rights data hands you: available_territories means the window has already lapsed somewhere, so a difference-in-differences on renewal rates between lapsed and still-licensed territories identifies the effect under parallel trends, and territory is not randomly assigned so the assumption must be argued with pre-period trend lines rather than asserted.
Quantify displacement, because it is the whole question: for exposed accounts, measure qualified hours on everything outside this rights_holder_id before and after an expiry that already happened, relative to the matched group, and report what fraction of the lost hours reappeared elsewhere in the catalogue.

Worked solution (35 min)

Define exposure as a continuous share and then a threshold, and show one distribution cut: what fraction of accounts have more than 10 percent of trailing-90-day qualified hours on this rights_holder_id.
Write the matched comparison: exposed versus unexposed accounts matched on pre-period qualified hours decile, tenure bucket, plan_tier and billing_interval, with renewal_outcome as the outcome and the matching variables listed explicitly.
Write the difference-in-differences on territories where rights_expires_at has already passed, with the pre-period trend check named as the test of the parallel-trends assumption.
Compute displacement for the already-lapsed territories: exposed accounts' qualified hours outside the bundle, before versus after, relative to matched accounts elsewhere, expressed as a substitution fraction.
Assemble the memo number: incremental renewal rate times net_amount_usd per period times exposed accounts, with the substitution fraction applied and the upward-bias caveat stated in the same paragraph.

Expected result: Metric: incremental retained subscription revenue over the renewal term, identified by matched exposed-versus-unexposed renewal rates and corroborated by a difference-in-differences across territories whose window already lapsed, net of measured substitution into the rest of the catalogue, compared against renewal cost. Stated assumptions: matching removes selection on volume but not on taste, biasing upward; the territory comparison requires parallel pre-trends.

Verify it yourself

The memo number is a revenue figure comparable to the renewal cost, not an hours or plays figure requiring a further conversion the reader has to invent.
A substitution fraction appears explicitly, and setting it to 1 makes the incremental value go to roughly zero, which is the correct behaviour.
The territory analysis has a pre-period trend check rather than an assertion of comparability.
The direction of the residual confounding bias is stated, and it is upward, which means the honest recommendation is more conservative than the point estimate.

Follow-up

The difference-in-differences shows a renewal effect only in two of eleven lapsed territories. What do you report, and does that change the recommendation?
How would your answer change if the bundle's license_type were 'exclusive_license' rather than 'non_exclusive_license'?
Exposed accounts substitute 70 percent of the lost hours within four weeks. What number goes in the memo now?

Involuntary churn share jumped after a dashboard rebuild

Medium · definition-change · churn · late-resolving-data

Involuntary share of gross churn — fct_subscription_period rows with renewal_outcome = 'failed_involuntary' over rows with renewal_outcome in ('cancelled_voluntary','failed_involuntary'), keyed on period_end_ts month — reads 31% for last month against 19% for the three months before. Billing reports no change in decline rates and dunning_attempts per failed period is unchanged. Using fct_subscription_period (period_end_ts, renewal_outcome, payment_status, dunning_attempts, cancel_requested_ts, billing_provider), determine whether involuntary churn actually rose. Deliverable: the corrected month-on-month series and the specific change that produced the step.

Approach

Check the month's maturity before anything else. Periods ending recently still have retries in flight, and a row reading 'failed_involuntary' today becomes payment_status = 'retried_paid' next week. Recompute every one of the last twelve months as of a fixed age — period_end_ts plus the maximum retry window — instead of as of today.
If the step survives age alignment, hunt for a definition change: diff the model or query against the version that produced 19%, and check specifically whether the month is now keyed on cancel_requested_ts rather than period_end_ts, and whether 'refunded' or 'chargeback' rows have entered one side of the ratio.
Settle it with one query rather than a narrative. Run the old definition and the new definition on the same month. Two numbers from one month under two definitions is decisive; two numbers from two months under one definition is not.
If a definition did change, restate history rather than leaving a step in the chart: run the new definition back across all twelve months and publish both series with the cutover marked, so nobody re-discovers this in a quarter.
Cut the horizon-closed series by billing_provider before concluding nothing moved. Card-direct, app-store and carrier billing have different retry mechanics and different success rates, so a mix shift between them moves the ratio with every provider's own rate flat.

Follow-up

How would you set the dunning horizon from the data instead of taking billing's stated maximum?
Under cancel-at-period-end, the cancellation request and the entitlement end fall in different months. Which timestamp keys this metric, and what does the other choice do to the curve?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Your part, not the team's

Most data work is done by groups, so an interviewer has to work out which piece was yours. An answer that runs on 'we' for several minutes gets interrupted with a question about what you personally did, and by then the answer sounds defensive even when it is true. Mark your own contribution as you go, and name the parts that belonged to someone else instead of leaving them ambiguous. Keep a few specifics back as well, like the name of the metric or who actually objected, so a probe can be answered with something you had not already said.

Can you provide an example of how you influenced a team decision?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Describe a time when you had to work under pressure to meet a deadline.

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

Walk through an analysis you shipped that was wrong

Medium · error ownership · data quality · postmortem

Describe a case where you delivered a result that was later shown to be wrong, and it had already been acted on. Cover how the error surfaced, whether you or someone else found it, what the wrong number caused, and what you changed afterwards. Prepare an example whose root cause was a definition, a denominator or a join — not a transcription slip. The interviewer will push on the mechanism, not the apology. Deliverable: a four-minute account that ends with a specific control now running in a pipeline.

Approach

The probe is whether your account has a mechanism in it. Choose an error that generalises — a denominator that silently changed population, a join that fanned rows, a metric partitioned on event_date while offline playback arrived days late and landed in the wrong partition — rather than one that only teaches you to check your typing.
State the blast radius factually and early: which decision was taken, how long the number stood, what it cost. A candidate who softens this is answering a different and easier question, and the interviewer can hear the substitution.
Explain how it surfaced without adjusting who found it. The generalisable detail is why your own checks did not catch it, which is a statement about your checks rather than about your luck.
Name the control you added and where it now lives: a row-count assertion after the fan-out join, a reconciliation that recomputes a closed day after late-arriving offline playback and alerts above a threshold, a denominator assertion inside the query. A fix that lives in a pipeline is different in kind from a resolution to be more careful.
Close with whether the control has fired since, or how you tested that it would. That single sentence is what separates a fix from an intention, and interviewers ask for it when candidates do not offer it.

Follow-up

Why didn't your own review catch it? Be specific about what you did check.
What class of error would that control still not catch, and what would you add next?
Have you found an error in someone else's published analysis since? How did you raise it?