Interview guides · Audible

Audible Data Scientist Interview Questions & Guide 2026

Audible Data Scientist interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Data Scientist at Audible, you operate at the intersection of massive-scale user data and the world’s most compelling audio storytelling. This role is not merely about running models; it is about driving the product strategy that keeps millions of listeners engaged with content. You will tackle complex challenges related to content discovery, personalization, and the health of the subscription ecosystem.

Coding rounds for this role are usually data-manipulation shaped rather than data-structure shaped: group-bys, joins, time windows, ranking within a partition. Confirm the format before spending a week on graph traversal.

Focus areas

Split voluntary churn from involuntary payment failure
Separate release-calendar spikes from underlying trend
Correct discovery slates for position bias

Overview

You will work closely with cross-functional partners in product, engineering, and marketing to translate fuzzy business problems into rigorous analytical frameworks. Whether you are optimizing a recommendation algorithm, designing a new A/B test for a feature launch, or diagnosing a sudden shift in key performance metrics, your work directly influences the daily experience of the Audible user base. The environment is fast-paced, intellectually demanding, and relies heavily on your ability to synthesize technical insights into actionable business recommendations.

The Audible interview process

Audible candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Deep Dives

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Case Studies

A case round is decided by whether you leave the interviewer with a recommendation, not by how much analysis you narrate on the way there. The prompt is open on purpose, so the first job is to convert it into a decision someone could act on: ask what would be done differently depending on the answer. From there name the quantity that would settle it, state the assumptions you need, and commit. Candidates who cover more ground than anyone expected and still end on "it depends" score below candidates who scoped narrowly and said what they would do.

What to demonstrate

Whether the version of the question you choose to answer is genuinely narrower than the prompt and still worth answering
Whether the recommendation arrives as an action with a number attached, rather than as a summary of what you looked at
Whether assumptions are stated at the moment you rely on them, instead of collected into a disclaimer at the end
Whether you notice when a branch you are exploring would not change the decision either way

How to prepare

Take six open prompts and write only the scoping move for each: the one-sentence question you would actually answer and the decision it feeds. Give yourself three minutes per prompt and stop there.
Put a five-minute warning into every practice case and force a closing statement that names the action, the result that would justify it, and the result that would reverse it.
Record one case and count how long you talked before naming a measurable quantity. Past roughly five minutes, what you are calling scoping is narration.

Coding Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Behavioral Rounds

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

Whether you can state the other side's argument accurately before you explain why you disagreed
What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Final Panel Interviews

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Reading recommendation quality from click-through or play-through on logged slates.

Slate position drives interaction independently of relevance, and the ranker that produced the log also decided what got exposure, so logged data is confounded by the policy you are trying to evaluate. Inverse-propensity estimators fix this only when the logging propensities are known and every item had a non-zero chance of being shown; without exploration slots that positivity condition fails silently and the estimate is confidently wrong. The visible symptom is a feedback loop: offline metrics improve, consumption concentrates further into the head of the catalogue, and the online test does not reproduce the offline gain.

Collapsing cancellation and payment failure into one churn number.

Involuntary churn from expired or declined payment instruments is a large and volatile share of gross churn, and it responds to retry schedules, card-updater coverage and billing provider, not to anything in the product. It also resolves late, so a period that looks involuntary today can be a successful retry next week, and reading the split before the dunning horizon closes overstates it. Compounding this, cancel-at-period-end means the cancellation request and the entitlement end are different timestamps on different rows, so a churn curve keyed on cancel_requested_ts and one keyed on churn_ts disagree by a full billing period.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Questions & practice

Permutation test for hours per account across two ranker arms

Medium · permutation-test · heavy-tails · experiment-analysis

arm_hours holds one row per account: account_id, arm in {control, treatment}, qualified_hours over a seven-day window. Roughly 40 thousand accounts per arm, about 38 percent of them at zero hours, and the non-zero tail is long. Without calling a library test function, write a permutation test on the difference in mean hours with 10,000 relabellings. Then run it as two parts: the difference in the share of accounts with any hours, and the difference in mean hours among accounts with hours. Report all three and say which belongs in the readout.

Approach

Shuffle the labels, not the data. Draw a permutation of the arm indicator over accounts, which is the unit that was randomised, and hold the hours vector fixed.
Make each replication O(n): precompute the grand sum and the arm sizes, so a shuffled difference is the treated subset sum over n_t minus (grand sum minus that subset sum) over n_c. Ten thousand replications then take seconds instead of a minute.
Use the two-sided p-value (1 + count of permuted absolute differences at or above the observed) divided by (B + 1). The plus one is not cosmetic: it makes the p-value valid rather than optimistic, and it means the smallest reportable value here is 1/10001, not zero.
For the two-part version, run the same machinery on the 0/1 indicator for the rate, then on the non-zero subset for the conditional mean, and say plainly that conditioning on a post-treatment outcome breaks the randomisation, so the conditional arm is descriptive rather than causal.
Report the rate test and the overall mean test as the result, with the conditional mean as colour, and give the effect size in hours beside each p-value, because at 80 thousand accounts almost anything is detectable.

Worked solution (30 min)

Pull hours into a float array and arm into a boolean array; record n_t, n_c, the grand sum and the observed difference of means.
Loop B = 10,000 times, taking np.random.permutation(n)[:n_t] as the treated index set and accumulating hours[idx].sum().
Convert each shuffled treated sum into a difference of means, compare its absolute value against the observed, count, and apply the plus-one form.
Repeat with hours replaced by (hours > 0).astype(float) for the rate test.
Subset to hours above zero, re-run for the conditional mean, and assemble a three-row result carrying effect, p-value and the unit each is defined on.

Expected result: Three rows: the mean-hours difference with a permutation p-value bounded below by 1/10001; the streaming-rate difference in percentage points with its own p-value; and the conditional mean difference among non-zero accounts, labelled descriptive because it conditions on a post-treatment outcome.

Verify it yourself

The permutation distribution of the mean difference is centred on zero within its own Monte Carlo error. A centre away from zero means the labels were not actually being reshuffled between replications.
The permutation p-value on the rate agrees to two decimal places with a two-proportion z-test, which is a fair cross-check because a 0/1 indicator is not heavy-tailed.
Doubling B leaves the p-value stable to within roughly 1 over sqrt(B). A p-value that moves a lot between runs is not yet resolved.
Multiplying every hours value in both arms by 60 leaves all three p-values identical under the same seed and scales only the effect sizes, since a permutation test is invariant to a common positive rescaling.

Follow-up

The permutation p-value on the mean is 0.03 and the rate test is flat. What is the most likely explanation, and does it change the decision?
How would CUPED on pre-period hours change your power here, and what would disqualify a covariate?
Accounts are households. Does that affect the validity of this test, or only its interpretation?

Simulate concurrent-stream refusals on a shared family account

Medium · simulation · probability · queueing

A family account has four profiles and max_concurrent_streams = 2. Across an evening window of six hours, each profile independently attempts playback as a Poisson process at 0.75 attempts per hour. Durations are lognormal with a median of 24 minutes and sigma 0.6 on the log scale. An attempt arriving while two streams are already active is refused and abandoned, not retried or queued. Estimate the share of attempts refused and the mean refusals per account-evening, each with a 95 percent interval, then repeat with the cap at three.

Approach

Simulate event-driven rather than on a time grid: draw exponential inter-arrivals per profile at rate 0.75 per hour, pool and sort the arrival times, and carry a short list of active end times.
At each arrival, discard end times at or before that instant, then refuse if two remain. Only a served attempt draws a duration and pushes arrival plus duration; drawing a duration for a refused attempt and letting it hold a slot models a queue instead of a refusal.
Start the window empty and do not warm it up, because the question is about an evening that begins with nobody streaming. Say explicitly that this makes the simulated share read below the steady-state value.
Size the replications from a pilot: run 2,000 evenings, read the observed share, then solve 1.96 times the standard error to at most 0.002 using the between-evening standard deviation of the per-evening refusal share.
Re-run with the cap at three by changing one constant and report the pair, because the decision this feeds is whether the cap is what the refusals are about.

Follow-up

What changes if a refused attempt retries after two minutes instead of being abandoned?
Erlang-B gives a closed form for this. Where does it agree with your simulation and where does it not?
How would you find these refusals in fct_stream, given that a refused attempt produces no stream row at all?

Attribute streams to prior impressions without merge_asof

Medium · pandas · joins · attribution

You have impressions (impression_id, profile_id, content_version_id, rendered_at, surface, slate_position, viewport_visible_ms) and streams (stream_id, profile_id, content_version_id, started_at). Attribute each stream to the most recent impression of the same profile and content version with rendered_at at or before started_at, a gap of at most 30 minutes, and viewport_visible_ms above zero; break ties on rendered_at by the smaller slate_position. pd.merge_asof and any groupby-apply over individual rows are off limits. Return streams with impression_id added, plus the attributed share by surface.

Approach

Filter impressions to viewport_visible_ms above zero first. A row that never scrolled into view is not an exposure, and leaving it in lets an unseen impression win the as-of match over a seen one.
Stack both frames into one long frame with a ts column and an is_stream flag, then sort by (profile_id, content_version_id, ts, is_stream, negative slate_position) so that at an identical timestamp impressions sort before the stream, and among tied impressions the smallest slate_position sorts last.
Forward-fill the candidate impression_id and rendered_at with groupby(['profile_id','content_version_id']).ffill(). The grouping is what stops the fill leaking across profiles; a global ffill on a sorted frame is the usual wrong answer.
Apply the 30-minute window as a post-filter on the filled candidate, and let a stream that fails it go unattributed rather than falling back to an older row. Every candidate precedes the stream, so the gap grows monotonically as you walk backwards: if the nearest in-view impression is more than 1800 seconds old, no earlier one is inside the window either, and null is the correct answer. The window cannot be pushed onto the impressions frame beforehand in any case, because "within 30 minutes" is defined against a particular stream's started_at and one impression is a candidate for many streams. Dropping impressions rendered more than 30 minutes before the earliest stream is a safe prune, but it is an optimisation, not the rule.
Slice the stream rows back out and compute the attributed share by surface, stating that unattributed streams (resume, direct link, radio seed) are not a surface and belong in their own bucket.

Follow-up

The same item was rendered in two slates inside the window. Your tie-break picks one. What does that do to per-surface credit, and what would a fractional rule change?
How would you validate this against the impression_id already carried on fct_stream, and what would a systematic disagreement tell you?
At 2 million impressions and 500 thousand streams this fits in memory. What changes at 200 million?

Write a query using SQL window functions to calculate the running total of user listening hours per month.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

How would you identify the top 5 most engaged users per genre using subqueries or window functions?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Pick one canonical version per work without losing remaster plays

Medium · deduplication · window functions · dimension grain

dim_content_version has content_version_id, content_id (stable across versions of the same work), title, content_type, duration_seconds, catalogue_added_ts, is_active_version and updated_at. A remaster, re-edit, dub or re-encode is a new row sharing content_id, and more than one row per content_id can carry is_active_version = true. Produce exactly one canonical row per content_id, preferring is_active_version = true, then the latest catalogue_added_ts, then the highest content_version_id as a deterministic tie-break. Then join fct_stream (content_version_id, played_seconds, is_qualified, started_at) and report qualified hours per content_id for last month.

Approach

ROW_NUMBER() OVER (PARTITION BY content_id ORDER BY is_active_version DESC, catalogue_added_ts DESC NULLS LAST, content_version_id DESC), keeping rn = 1. The third sort key exists only so two runs return the same row; without it the winner among ties is whatever the engine happened to produce, and the result diffs against yesterday for no reason.
Spell NULLS LAST out. In PostgreSQL, DESC defaults to NULLS FIRST, so a version with no catalogue_added_ts would win the canonical slot ahead of every dated one. Other engines order NULLs differently, which is exactly why the clause belongs in the query rather than in your head.
Aggregate the streams first, at content_id grain, by joining fct_stream to dim_content_version on content_version_id and grouping on content_id. Every version's plays belong to the work.
Only then attach the canonical attributes, with a join from the aggregate to the rn = 1 CTE on content_id. Inner-joining fct_stream to the canonical row instead discards every play of every non-canonical version, which is most of the catalogue's remasters and all of its dubs.
Report the count of content_ids carrying more than one is_active_version row as a data-quality line beside the result, since that condition is what made the dedup necessary and its size decides whether this is a query fix or a pipeline fix.

Worked solution (25 min)

Size the problem before solving it: count content_ids with more than one row, and separately those with more than one is_active_version = true row.
Write the ROW_NUMBER CTE with explicit NULLS LAST, keep rn = 1, and assert COUNT(*) equals COUNT(DISTINCT content_id) in the dimension.
Aggregate fct_stream to content_id through the version dimension, filtering is_qualified and last month, as SUM(played_seconds)/3600.
Join the canonical attributes on and compare the grand total against the same aggregate computed at version grain.

Expected result: One row per content_id with at least one qualified stream last month, carrying title and duration from the canonical version and qualified_hours summed over all versions of the work. The grand total equals the ungrouped total over the same stream filter.

Verify it yourself

COUNT(*) from the canonical CTE equals COUNT(DISTINCT content_id) in dim_content_version.
Rerunning the CTE returns an identical set of content_version_ids, confirming the tie-break made it deterministic.
Total qualified_hours matches the version-grain total to the second; any shortfall is the dedup-then-join bug.

Follow-up

Which grain should hours be reported at, content_id or content_version_id, and who is the consumer of each?
Rights expire per version, so one version's window can be closed while another is live. What does your canonical pick then mean for an availability report built on the same CTE?

How do you balance short-term conversion goals with long-term customer satisfaction?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

A key engagement metric has dropped suddenly; walk me through your process for diagnosing the root cause.

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you design a recommendation system for Audible to improve long-term user retention?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

We are launching a new feature; how would you define success and choose the primary metric?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Design an A/B test for a change in the subscription checkout flow.

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Decide the analysis before seeing data, including how long it runs and when you look.
Say whether units interfere with each other, and switch design if they do.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

How would you handle a situation where your experiment results show a positive impact on conversion but a negative impact on long-term retention?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Say whether units interfere with each other, and switch design if they do.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

Decide a rights renewal on retention, not on plays

Medium · metric design · catalogue economics · displacement · causal comparison

A licensed bundle from one rights_holder_id has rights_expires_at in 60 days and renewal is expensive. Finance wants a number that decides it. You have dim_content_version (content_version_id, content_id, rights_holder_id, license_type, rights_expires_at, available_territories), fct_stream (account_id, content_version_id, played_seconds, is_qualified, started_at, play_territory) and fct_subscription_period (account_id, period_end_ts, renewal_outcome, net_amount_usd). Define the metric you would put in the renewal memo, the comparison that identifies it, and the assumption it rests on. Plays on the bundle is not an acceptable answer.

Approach

Reject the play count with a reason, not a slogan: plays measure what the bundle drew, and the renewal question is what would be lost if it went, which differs by whatever those accounts would have watched instead. A bundle can be the third most played thing in the catalogue and still be worth nothing at the margin if its audience substitutes freely.
Define the account-level exposure measure the analysis runs on: share of an account's qualified hours in the trailing 90 days falling on content_version_id rows whose rights_holder_id is the bundle's, which turns a catalogue fact into an account attribute that can be related to renewal_outcome.
State the metric for the memo: incremental retained subscription revenue, estimated as the difference in the rate of renewal_outcome = 'renewed' between exposed accounts and matched unexposed accounts, multiplied by net_amount_usd per period, summed over exposed accounts, and compared directly against the renewal cost.
Name the identification problem instead of burying it: exposure is chosen, so heavy users of this bundle are heavier users generally and would renew at a higher rate with or without it. Match on pre-period total qualified hours, tenure, plan_tier and billing_interval, and state that residual confounding on taste remains and biases the estimate upward.
Use the natural experiment the rights data hands you: available_territories means the window has already lapsed somewhere, so a difference-in-differences on renewal rates between lapsed and still-licensed territories identifies the effect under parallel trends, and territory is not randomly assigned so the assumption must be argued with pre-period trend lines rather than asserted.
Quantify displacement, because it is the whole question: for exposed accounts, measure qualified hours on everything outside this rights_holder_id before and after an expiry that already happened, relative to the matched group, and report what fraction of the lost hours reappeared elsewhere in the catalogue.

Worked solution (35 min)

Define exposure as a continuous share and then a threshold, and show one distribution cut: what fraction of accounts have more than 10 percent of trailing-90-day qualified hours on this rights_holder_id.
Write the matched comparison: exposed versus unexposed accounts matched on pre-period qualified hours decile, tenure bucket, plan_tier and billing_interval, with renewal_outcome as the outcome and the matching variables listed explicitly.
Write the difference-in-differences on territories where rights_expires_at has already passed, with the pre-period trend check named as the test of the parallel-trends assumption.
Compute displacement for the already-lapsed territories: exposed accounts' qualified hours outside the bundle, before versus after, relative to matched accounts elsewhere, expressed as a substitution fraction.
Assemble the memo number: incremental renewal rate times net_amount_usd per period times exposed accounts, with the substitution fraction applied and the upward-bias caveat stated in the same paragraph.

Expected result: Metric: incremental retained subscription revenue over the renewal term, identified by matched exposed-versus-unexposed renewal rates and corroborated by a difference-in-differences across territories whose window already lapsed, net of measured substitution into the rest of the catalogue, compared against renewal cost. Stated assumptions: matching removes selection on volume but not on taste, biasing upward; the territory comparison requires parallel pre-trends.

Verify it yourself

The memo number is a revenue figure comparable to the renewal cost, not an hours or plays figure requiring a further conversion the reader has to invent.
A substitution fraction appears explicitly, and setting it to 1 makes the incremental value go to roughly zero, which is the correct behaviour.
The territory analysis has a pre-period trend check rather than an assertion of comparability.
The direction of the residual confounding bias is stated, and it is upward, which means the honest recommendation is more conservative than the point estimate.

Follow-up

The difference-in-differences shows a renewal effect only in two of eleven lapsed territories. What do you report, and does that change the recommendation?
How would your answer change if the bundle's license_type were 'exclusive_license' rather than 'non_exclusive_license'?
Exposed accounts substitute 70 percent of the lost hours within four weeks. What number goes in the memo now?

Revenue per engaged account slid with no price change

Medium · mix-shift · segmentation · revenue

Net revenue per active account-month fell 7% across two closed months. No list price changed and no discount campaign ran. You have fct_subscription_period (account_id, period_start_ts, period_end_ts, plan_tier, billing_interval, net_amount_usd, currency_code, is_promotional, payment_status), dim_account (signup_country, plan_tier, billing_provider, first_paid_ts) and the qualified-stream denominator from fct_stream. Decide whether revenue per paying account fell or the denominator changed, and where. Deliverable: a decomposition across numerator and denominator naming the two largest contributing segments and whether each is reversible.

Approach

Split the ratio before splitting any segment. Revenue per engaged account equals revenue per paying account multiplied by paying accounts over engaged accounts. A growing free_ad_supported tier moves only the second factor and looks identical to a pricing problem in the headline.
Cut the numerator by signup_country and currency_code and recompute at both period-of-record and fixed exchange rates. If local list prices are unchanged, a simultaneous USD fall across many countries is FX, and the fixed-rate series says so in one line instead of a meeting.
Decompose revenue per paying account into within-segment change and mix over (plan_tier x billing_interval x signup_country) cells. Annual periods recognise pro rata across months, so a monthly-to-annual shift changes recognised revenue per account-month without changing what anybody paid over a year — a real accounting move with no economic content.
Treat the denominator's composition as its own problem. Engaged accounts are distinct accounts with a qualified stream, so an acquisition push into a low-price market or into the free tier adds denominator months before it adds revenue and dilutes the ratio mechanically for a knowable number of months.
Report the two segments carrying most of the move with their weight change and within-segment change shown side by side, and label which reverse on their own and which do not. That distinction is the actionable part of the answer; the 7% by itself is not.

Follow-up

The ratio dilutes for some months after an acquisition push. How many, and how do you present the metric so a marketing success is not read as a regression every time?
If the shift into annual billing is permanent, is the 7% real? What does the promotional-share guardrail say about it?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

Give an example of a project where you took full ownership and delivered impact despite significant ambiguity.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Arbitrate three requests for the same analyst week

Easy · prioritisation · stakeholder management · scoping

Three requests land on the same Monday for the same week of your time. Growth wants trial-to-paid conversion recut by signup_channel before a budget meeting on Thursday. The playback team wants rebuffer ratio cut by cdn_pop, device_type and network_type after a bitrate ladder change went out Friday. Catalogue wants a displacement estimate for a title that arrived on a rights window last month. You have roughly four working days. Deliverable: your ordering with the criterion behind it, what you tell the two people who do not get the week, and one thing you ship to all three by Tuesday.

Approach

The probe is your criterion, not your ordering. Rank by how reversible the pending decision is and how fast the evidence decays, and say so; ranking by requester seniority is the default answer and reads as one.
Put the playback request first on those grounds: a bitrate ladder change is live and degrading real playback now, and rebuffer ratio is defined on a trailing 24 hours, so the evidence is perishable and the query is small. Cheap and perishable beats expensive and durable.
Ask each requester what they do differently at each plausible answer. The trial-to-paid recut is a defined tree metric — trailing 28 days, read at trial length plus 21 days so dunning retries have resolved — so it is largely a filter plus an hour confirming that unattributed stays its own line rather than being reallocated pro rata.
Treat the displacement estimate as the only genuinely new analysis and scope it rather than squeezing it: it needs a matched comparison group and a pre-registered window, because consumption of the rest of the catalogue moves on the release calendar regardless. Defer it with a date and a stated shape, never silently.
Reply to all three in one thread with the ordering and the criterion visible, so a stakeholder who disagrees argues with the criterion instead of escalating about their position in the queue.

Follow-up

The catalogue lead escalates to your manager on Tuesday. Walk me through that conversation.
How does the ordering change if the bitrate ladder change was already rolled back on Saturday?
All three deadlines are genuine and none can move. What do you actually cut?

Disagree with a product manager about a completion metric

Medium · disagreement · metric design · influence

A product manager proposes making duration-normalised completion rate the team's primary metric for the quarter, and wants a plain global unweighted version "so it's simple enough to put on a wall." You expect the unweighted version to move several points on catalogue mix alone, with no change in how satisfying anything was. You have one meeting, they own the roadmap, and you will work with them for years. Deliverable: the argument, the evidence you bring, and the fallback you accept if they still want the simple version.

Approach

The probe is whether you can lose the decision without damaging either the metric or the relationship. Bring the failure already reproduced: recompute the unweighted global rate over the last two quarters and point at the weeks it moved several points where the only input that changed was which content people played.
Decompose the variance instead of objecting. Attribute the movement in the unweighted rate to between-cell mix — (content_type, duration decile) — versus within-cell movement. A decomposition is an argument; an assertion that mix matters is a preference.
Concede the real cost honestly: the mix-weighted version is harder to explain and harder to recompute. Bring the mitigation rather than dismissing the objection — weights fixed from a stated reference month, published once, so anyone can reproduce the number without redoing the weighting.
Offer the fallback deliberately and in advance: ship both, with the unweighted rate labelled a diagnostic and the weighted one as the decision metric, and pre-agree the one observation that would mean the simple version has misled the team.
Leave the decision with them, with the consequence written down before it happens. That is what makes a later correction a shared prediction coming true rather than a retrospective argument about who was right.

Follow-up

They ship the simple version and it moves four points during a heavy release week. How do you raise it without saying you told them so?
Is there a case where the unweighted rate is genuinely the right primary metric?
Suppose you are wrong and the mix effect turns out to be small. What would you change about how you argued this?