Iheartmedia Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at Iheartmedia, you sit at the intersection of massive media scale and sophisticated data-driven decision-making. You are responsible for transforming complex datasets—ranging from listener behavior and audio consumption patterns to advertising effectiveness—into actionable insights that drive the future of digital audio and live entertainment.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

• Split voluntary churn from involuntary payment failure
• Define a qualified stream and defend it
• Diagnose rebuffering by device, network and POP

Overview

Your impact is felt across the organization, influencing product development, content strategy, and advertising technology. You will work closely with engineering and product teams to build models that enhance user personalization and optimize monetization strategies. This role is ideal for those who thrive in a fast-paced environment where data is not just a support function, but a core component of the business strategy.

Be prepared to articulate how your technical work directly impacts the bottom line, as Iheartmedia values data scientists who understand the business context behind the numbers.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

• Breadth across SQL, experimentation and product reasoning
• Ability to state assumptions before choosing a method

How to prepare

• Drill the practice exercises below and time yourself
• Prepare three quantified stories about decisions you drove

Where candidates lose points

Reading recommendation quality from click-through or play-through on logged slates.

Slate position drives interaction independently of relevance, and the ranker that produced the log also decided what got exposure, so logged data is confounded by the policy you are trying to evaluate. Inverse-propensity estimators fix this only when the logging propensities are known and every item had a non-zero chance of being shown; without exploration slots that positivity condition fails silently and the estimate is confidently wrong. The visible symptom is a feedback loop: offline metrics improve, consumption concentrates further into the head of the catalogue, and the online test does not reproduce the offline gain.

Collapsing cancellation and payment failure into one churn number.

Involuntary churn from expired or declined payment instruments is a large and volatile share of gross churn, and it responds to retry schedules, card-updater coverage and billing provider, not to anything in the product. It also resolves late, so a period that looks involuntary today can be a successful retry next week, and reading the split before the dunning horizon closes overstates it. Compounding this, cancel-at-period-end means the cancellation request and the entitlement end are different timestamps on different rows, so a churn curve keyed on cancel_requested_ts and one keyed on churn_ts disagree by a full billing period.

Dropping rows with missing values without naming the mechanism

Say whether the values are missing at random, missing by a known process, or missing in a way that depends on the outcome, and handle them accordingly. Deleting incomplete rows silently redefines the population whenever missingness correlates with what you are measuring.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Questions & practice

Explain the trade-offs between different classification models in a high-latency environment.

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Set a baseline first, so any model has something honest to beat.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

Five invariant checks over a playback fact table

Easy · data-quality · pandas · validation

fct_stream arrives with stream_id, started_at, ended_at (nullable), played_seconds, max_position_seconds, completion_ratio, is_qualified, event_date, and duration_seconds joined from the content dimension (null for live events). Write one function returning a tidy frame with a row per rule: rule name, eligible rows, violations, violation share, one example stream_id. The five rules are duplicate stream_id; played_seconds exceeding wall-clock ended_at minus started_at; max_position_seconds exceeding duration_seconds; completion_ratio disagreeing with max_position_seconds divided by duration_seconds; and event_date not equal to the UTC date of started_at.

Approach

1. Give every rule its own eligible-row mask before its violation mask. Rules 2 to 4 are undefined where ended_at or duration_seconds is null, and a comparison against NaN evaluates False, so a mostly-null column otherwise reports a clean bill of health.
2. Compare the recomputed completion_ratio with a tolerance (absolute difference above 1e-6), never with ==. It is stored as a float division and exact equality fails on rows that are correct.
3. Allow a small tolerance on rules 2 and 3 too: a final segment can carry the playhead a second or two past duration_seconds, and heartbeat timestamps come off client clocks. State the tolerance you chose rather than burying it.
4. Return violations and eligible rows as separate columns so the share has a stated denominator, and carry one example stream_id per rule so the output is actionable rather than a number.
5. Read rule 5 as a signal rather than corruption: event_date drifting from the UTC date of started_at is the signature of offline playback uploaded after the partition closed, and its distinct dates are the recompute list.

Worked solution (20 min)

1. Write a helper taking (name, eligible_mask, violation_mask) and returning one result row, which forces every rule to declare its eligible population.
2. Rule 1: duplicated('stream_id', keep=False), eligible is all rows. Rule 5: event_date not equal to started_at.dt.date, eligible is all rows.
3. Rules 2 to 4: eligible masks are ended_at.notna() and duration_seconds.notna(); violation masks are played_seconds above wall_seconds plus 2, max_position_seconds above duration_seconds plus 2, and the absolute completion_ratio difference above 1e-6.
4. Take one example stream_id per rule with .loc[mask].head(1), guarding the empty case so a clean rule does not raise.
5. Concatenate the five rows and sort by violation share descending.

Expected result: A five-row frame with columns rule, eligible_rows, violations, violation_share and example_stream_id, where eligible_rows for rules 2 to 4 is strictly below the table length whenever ended_at or duration_seconds contains nulls.

Verify it yourself

• Injecting one row whose played_seconds is ten times its wall clock raises exactly rule 2's count by one and leaves the other four unchanged.
• Setting duration_seconds to null on that same row moves it out of the eligible counts for rules 3 and 4, rather than into their violation counts.
• violation_share is violations divided by that rule's eligible_rows, not by len(df); the two differ for rules 2 to 4.
• Rule 1's violation count is even when duplicates arrive in pairs, because keep=False flags both copies.

Follow-up

• Rule 5 fires on 0.4 percent of rows, all from one app_version, all with reported_at days after started_at. Is that a bug, and what do you do about the daily numbers already published?
• How would you turn these into a blocking check in the pipeline without failing the load every time one client version misbehaves?
• Which of the five would you expect to fire on live events specifically, and how do you keep them out of the denominator?

Attribute streams to prior impressions without merge_asof

Medium · pandas · joins · attribution

You have impressions (impression_id, profile_id, content_version_id, rendered_at, surface, slate_position, viewport_visible_ms) and streams (stream_id, profile_id, content_version_id, started_at). Attribute each stream to the most recent impression of the same profile and content version with rendered_at at or before started_at, a gap of at most 30 minutes, and viewport_visible_ms above zero; break ties on rendered_at by the smaller slate_position. pd.merge_asof and any groupby-apply over individual rows are off limits. Return streams with impression_id added, plus the attributed share by surface.

Approach

1. Filter impressions to viewport_visible_ms above zero first. A row that never scrolled into view is not an exposure, and leaving it in lets an unseen impression win the as-of match over a seen one.
2. Stack both frames into one long frame with a ts column and an is_stream flag, then sort by (profile_id, content_version_id, ts, is_stream, negative slate_position) so that at an identical timestamp impressions sort before the stream, and among tied impressions the smallest slate_position sorts last.
3. Forward-fill the candidate impression_id and rendered_at with groupby(['profile_id','content_version_id']).ffill(). The grouping is what stops the fill leaking across profiles; a global ffill on a sorted frame is the usual wrong answer.
4. Apply the 30-minute window as a post-filter on the filled candidate, and let a stream that fails it go unattributed rather than falling back to an older row. Every candidate precedes the stream, so the gap grows monotonically as you walk backwards: if the nearest in-view impression is more than 1800 seconds old, no earlier one is inside the window either, and null is the correct answer. The window cannot be pushed onto the impressions frame beforehand in any case, because "within 30 minutes" is defined against a particular stream's started_at and one impression is a candidate for many streams. Dropping impressions rendered more than 30 minutes before the earliest stream is a safe prune, but it is an optimisation, not the rule.
5. Slice the stream rows back out and compute the attributed share by surface, stating that unattributed streams (resume, direct link, radio seed) are not a surface and belong in their own bucket.

Follow-up

• The same item was rendered in two slates inside the window. Your tie-break picks one. What does that do to per-surface credit, and what would a fractional rule change?
• How would you validate this against the impression_id already carried on fct_stream, and what would a systematic disagreement tell you?
• At 2 million impressions and 500 thousand streams this fits in memory. What changes at 200 million?

Rebuffer ratio by point of presence, excluding start failures

Easy · ratio metrics · delivery quality · conditional aggregation

fct_stream carries cdn_pop, device_type, network_type, played_seconds, rebuffer_seconds and end_reason (enum including 'playback_error'). For streams with started_at in the trailing 24 hours, return one row per (cdn_pop, device_type, network_type) with two figures: rebuffer_ratio = SUM(rebuffer_seconds) / SUM(played_seconds), computed only over rows with played_seconds > 0; and start_failure_rate = the share of all rows in the cell with played_seconds = 0 and end_reason = 'playback_error'. Restrict output to cells with at least 500 streams and order by rebuffer_ratio descending.

Approach

1. Do it in one pass with conditional aggregation: SUM(rebuffer_seconds) FILTER (WHERE played_seconds > 0) over SUM(played_seconds) FILTER (WHERE played_seconds > 0), or the equivalent SUM(CASE WHEN ...) on engines without FILTER. Two separate scans risk the numerator and denominator disagreeing about which rows are in the cell.
2. Wrap the denominator in NULLIF(..., 0) so a cell in which every stream failed before first frame returns NULL rather than raising a division error — and so it is visibly absent rather than silently zero.
3. The failure rate's denominator is every row in the cell, including the zero-play ones. That is the whole point of the pair: the rows excluded from the ratio have to be counted somewhere, or a point of presence that fails before first frame reports a perfect ratio.
4. Apply the 500-stream minimum with HAVING COUNT(*) >= 500 before ordering, otherwise a cell with three streams and one stall tops the list at a ratio no one should act on.

Worked solution (15 min)

1. Write the cell aggregate exposing the raw components: COUNT(*), the two filtered sums, and the count of zero-play playback errors.
2. Derive both rates in an outer SELECT so the raw counts stay visible when a cell looks wrong.
3. Recompute the ratio ungrouped over the same filtered rows and confirm it equals the played_seconds-weighted mean of the cell ratios.

Expected result: One row per (cdn_pop, device_type, network_type) cell with at least 500 streams, carrying stream_count, rebuffer_ratio >= 0 (NULL where no row had played_seconds > 0) and start_failure_rate in [0, 1].

Verify it yourself

• The ungrouped ratio equals the played_seconds-weighted mean of the per-cell ratios, not their unweighted mean.
• Every cell whose start_failure_rate is exactly 1.0 has a NULL rebuffer_ratio.
• COUNT(*) summed across all cells before the HAVING equals the unfiltered row count for the 24-hour window.

Follow-up

• Two points of presence show the same rebuffer ratio but one has twice the start-failure rate. Which do you escalate, and what does the second number tell you that the first cannot?
• How would you decide whether a cell's ratio moved beyond what day-to-day variation explains, given that cell sizes differ by orders of magnitude?

Paid retention triangle with monthly and annual curves kept apart

Medium · cohorts · retention · subscriptions

fct_subscription_period has subscription_period_id, account_id, period_index, billing_interval, period_start_ts, period_end_ts, payment_status, cancel_requested_ts and renewal_outcome. dim_account has account_id, first_paid_ts and is_test_account. Build a paid-retention triangle: label non-test accounts with the calendar month of first_paid_ts, and for month offsets 0 through 6 report the share of the cohort still holding a period with payment_status in ('paid','retried_paid') that contains the instant first_paid_ts + offset months. The anchor is each account's own first payment, not the first day of the offset calendar month. Report monthly and annual billing_interval as separate curves. Exclude cohorts not yet mature at offset 6.

Approach

1. Take the cohort key from dim_account.first_paid_ts, not from MIN(period_start_ts). The period table includes trial periods as zero-amount rows with payment_status = 'trial_no_charge', so the minimum start shifts trialling accounts a cohort early and inflates the youngest cohort's offset-0 denominator.
2. Anchor every offset on that same first_paid_ts; the calendar month is only the row label on the triangle. Anchoring survival on the first day of the offset month instead breaks offset 0 for everyone who did not first pay on the 1st — an account that first paid on the 20th holds no paid period covering that month's 1st, so its offset-0 cell reads as churned and the curve rises from offset 0 to offset 1. A retention chart whose first step goes up is almost always this. Two conventions to write down while you are here: PostgreSQL clamps timestamp + INTERVAL '1 month' at month end, so an account first paid on the 31st is tested on the 28th or 30th at some offsets; and the whole construction assumes first_paid_ts falls inside its own first paid period, which fails if the platform charges shortly before period_start_ts, in which case anchor on that period's period_start_ts instead.
3. Build the offsets 0..6 as an explicit ordered set and cross-join it to the cohort list. A grid built by aggregating survivors alone loses any (cohort, offset) cell with no survivors, so a collapsing cohort reads as a missing row instead of a zero.
4. Test survival as interval containment, not date equality: EXISTS a period for that account with payment_status in ('paid','retried_paid') where first_paid_ts + offset months falls inside [period_start_ts, period_end_ts). One annual row contains seven consecutive anniversaries on its own; an equality join against period_start_ts finds it at offset 0 and nowhere else.
5. Partition everything by billing_interval and never pool. An annual account has had no opportunity to churn before day 365, so pooling makes an annual-heavy cohort read as retentive when what it actually is, is un-renewed. billing_interval lives on the period and not on the account, so take the cohort's interval from the period containing first_paid_ts and hold it fixed for all seven offsets; an account that switches monthly to annual at offset 4 otherwise appears in both curves and is counted twice in the denominators.
6. Apply maturity last, and against the anchor rather than the label: keep a cohort only when the last day of that cohort month plus six months is at or before the last fully closed day in the data, so every account in it has actually reached its offset-6 anniversary. Otherwise the newest cohorts print 0% at the far offsets for a reason that is entirely calendar.

Follow-up

• An account migrates plan mid-cohort with renewal_outcome = 'migrated_plan'. Retained or churned, and what does your choice do to the revenue story told beside this chart?
• A period fails and a retry succeeds as a new row. Does your survival test find the account at that offset, and should it?
• Can you produce a curve for annual accounts that is comparable to the monthly one before twelve months have elapsed, and what do you give up?

How would you approach churn prediction for a subscription-based audio service?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How would you measure the success of a new recommendation algorithm for our podcast platform?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Fix the population and the time window before naming any metric.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Describe your experience with A/B testing and how you ensure statistical significance in your experiments.

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

Triage a sample ratio mismatch before reading segment cuts

Easy · sample ratio mismatch · multiple comparisons · experiment hygiene

A 50/50 account-randomised test of a new detail-page layout has run ten days. The scorecard reports 2,014,880 accounts in control and 1,997,340 in treatment. It also reports 22 pre-declared segment cuts across signup_platform, device_type and signup_country, of which three are significant at p < 0.05, including a strongly negative one on game_console. The PM wants to ship with a console carve-out. State what you check first and what it returns, what the three segments are worth, and what would have to be true before any part of this scorecard can be read.

Approach

1. Run the sample ratio check before looking at any effect: a chi-square goodness-of-fit against the intended 50/50 on raw assignment counts, one degree of freedom. Treat p < 0.001 as a hard stop rather than a warning banner.
2. If it fires, stop reading effects entirely. Unequal counts mean the arms are not exchangeable, so every estimate on the scorecard, overall and per segment, carries a bias of unknown size and unknown direction.
3. Hunt the cause in the order that explains most mismatches: assignment happening downstream of a redirect or a feature-flag fetch that fails at different rates per arm, exposure logging dropped on one arm's code path, bot and is_test_account filtering applied after assignment rather than before, and deduplication of accounts that were reassigned.
4. Treat the segments as a multiplicity problem even in a clean test. Twenty-two cuts at alpha 0.05 give 1.1 false positives in expectation, and the chance of at least one is 1 - 0.95^22 = 68 percent, so three hits is barely news.
5. Apply Benjamini-Hochberg at FDR 0.05 across the 22 p-values, and apply it as the step-up procedure it is: order the p-values ascending, compare the k-th against k * 0.05 / 22, take the largest k that passes, and reject everything at or below it. Require any survivor to come with a pre-registered mechanism before it becomes a product carve-out instead of a post-hoc story.

Worked solution (15 min)

1. Expected count per arm = (2,014,880 + 1,997,340) / 2 = 2,006,110, so the deviation is 8,770 accounts.
2. chi-square = 2 * 8,770^2 / 2,006,110 = 2 * 76,912,900 / 2,006,110 = 76.7 on one degree of freedom.
3. Convert: z = sqrt(76.7) = 8.76, so p is on the order of 1e-18. The realised split is 50.22 / 49.78.
4. Set the segments aside, and write down the Benjamini-Hochberg rule correctly for when there is a valid scorecard to apply it to. Order the 22 p-values ascending and find the largest k with p_(k) <= k * 0.05 / 22. The ladder is 0.00227 at k = 1, 0.00455 at k = 2, 0.00682 at k = 3, rising linearly to 0.05 at k = 22. Because it steps up, that largest k also rejects every smaller p-value beneath it: p-values of 0.001, 0.004 and 0.006 would all be rejected, even though two of them sit above 0.00227. The single threshold 0.05 / 22 = 0.00227 is Bonferroni, a different and more conservative rule.

Expected result: chi-square approximately 76.7 on 1 df, p on the order of 1e-18: an unambiguous sample ratio mismatch, not a rounding artefact. The scorecard is unreadable, nothing ships, and the work is to find the assignment or exposure-logging asymmetry and re-run. Separately, under the null 22 cuts yield three or more hits about 9.5 percent of the time, so the segment pattern is unremarkable even in a valid test, and whether any of the three would survive Benjamini-Hochberg depends on where they sit on the k * 0.05 / 22 ladder, not on a single fixed threshold.

Verify it yourself

• Under a true 50/50 split at this size the standard deviation of one arm's count is 0.5 * sqrt(4,012,220) = 1,002 accounts, so 8,770 is 8.8 standard deviations out; a gap of a few hundred would be ordinary.
• Recomputing the ratio on raw assignment events rather than on the deduplicated scorecard population should reproduce the same imbalance. If it does not, the loss is in the analysis pipeline rather than in assignment.
• Under the null the expected number of significant cuts among 22 at alpha 0.05 is 1.1, and the probability of three or more is 9.5 percent.
• Benjamini-Hochberg must reject at least as many hypotheses as Bonferroni on the same p-values, never fewer. An implementation that rejects fewer, or that compares every p-value against 0.05 / 22, is running Bonferroni under the wrong name and will discard real segment effects.

Follow-up

• An engineer finds that assignment happens after a client-side flag fetch that times out more often on smart_tv. Is that a mismatch cause, and can the test be salvaged by excluding those accounts?
• Suppose the counts had been balanced. What evidence would make the game_console result worth acting on?
• Under Benjamini-Hochberg, a p-value of 0.006 can be rejected among these 22 cuts while a p-value of 0.004 elsewhere in the same family is not. Explain how both can be true at once.

Slate click rate jumped overnight with no ranker change

Medium · pipeline-failure · rate-denominator · logging

Home-row click rate — was_clicked over rendered rows in fct_impression (impression_id, slate_id, surface, slate_position, rendered_at, viewport_visible_ms, ranker_version, was_clicked) — rose from 4.1% to 6.8% on a single day and held there. ranker_version is unchanged across the step. Streams attributed to home_row in fct_stream (impression_id, start_source, started_at) did not rise. Using impression and stream data only, find the cause and state what you tell the ranking team, who have already claimed the improvement in a review deck. Deliverable: root cause plus the correctly restated series.

Approach

1. Ask which side of the ratio moved. A rate that steps while the downstream count is flat is almost always a denominator event, so plot clicks and impressions as separate absolute series before touching the ratio again. Do the arithmetic first as well: with the numerator fixed, a rate going from 4.1% to 6.8% pins the denominator to 4.1/6.8 of its former size, so you know the size of the impression loss to look for before you open the first query.
2. If impressions fell, locate the fall: group by surface, slate_position band and viewport_visible_ms = 0. Loss concentrated in high positions and in never-visible rows is the signature of a client or collector dropping below-the-fold renders, which removes mostly non-clicked rows and lifts the rate arithmetically with no behaviour change.
3. Prove rows are missing rather than never rendered. Left-join fct_stream rows with start_source = 'algorithmic_slate' and non-null impression_id back to fct_impression and count the ones that no longer resolve. Orphaned attributions date the breakage to the hour and are hard to argue with.
4. Look at impressions per slate_id, and require it to reconcile with the rate step. A twenty-item row now logging twelve is a truncation bug in the renderer or collector, and it accounts for the loss exactly if the distinct slate_id count is flat; a stable per-slate count with fewer slates is a sampling or partition drop upstream. The two have different owners and different backfill stories, and any per-slate figure that implies a larger loss than the rate step requires means a second thing is also wrong.
5. Restate the series on a basis the defect does not touch — clicks per slate_id, or the rate restricted to slate_position <= 8 where logging is intact — and take the dated evidence to the ranking team before the number reaches the review, not after.

Follow-up

• Impressions in the truncated tail are missing for six days. Can an experiment that ran in that window still be read, and under exactly what assumption?
• What monitor catches this within an hour, and what does it cost in false alarms on a normal release day?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

How do you handle missing or noisy data in large-scale user interaction logs?

Medium · behavioural and stakeholder questions

Approach

1. Quantify the outcome, including what you would not claim credit for.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

• How did you know the outcome was caused by your change?
• What did you decide not to do, and why?

Announce a stream-definition change that shifts payouts

Hard · cross-functional communication · metric definition · payouts

You find that the 60-second idle gap used to sessionise heartbeats into fct_stream rows splits one continuous listen into two streams whenever a phone backgrounds briefly on cellular. Correcting the gap lowers qualified stream counts on phones by an estimated four percent; total played_seconds is unchanged. Per-stream counts drive rights-holder payout shares. Deliverable: what you verify before telling anyone, the order in which you take it to the engineering owner, finance and content partnerships, and your recommendation on restating history.

Approach

1. The probe is whether you can tell a technical correction from a commercial decision and keep them apart in the room. Verify the split streams are genuinely one listen before anything else: same profile_id, same content_version_id, contiguous max_position_seconds across the boundary, and a gap distribution with a spike at the background-timeout duration rather than a smooth tail.
2. Compute the distributional effect, not the average. The four percent aggregate is not what anyone will argue about; recompute under the corrected gap and report which rights_holder_id groups gain and lose share, because short-form catalogue on mobile is where the splits concentrate and that is not spread evenly across counterparties.
3. Defend the new rule on its own terms rather than on the direction of the number. The idle gap is a choice, so the argument is evidence that the two rows describe one continuous listen — never that the corrected count is lower and therefore more conservative, which invites the symmetric accusation next time the fix goes the other way.
4. Name the ownership boundary out loud: engineering owns the sessionisation rule, finance and partnerships own whether payouts are restated. Conflating them is how a correct fix gets blocked by a commercial objection it should never have been exposed to.
5. Sequence the conversations so the number stops moving before it leaves the building: engineering owner first to confirm the rule and land the fix, finance second to size the restatement, partnerships last. Recommend restating history for internal metrics so trends stay comparable, and recommend against retroactive payout adjustment unless the contracts require it — naming who must answer that contractual question rather than answering it yourself.

Follow-up

• Partnerships asks you to hold the fix until after the quarter closes. What do you do, and who else needs to know you were asked?
• How do you present a change that raises some counterparties' shares and lowers others', in the same meeting, to people who will compare notes afterwards?
• The four percent estimate itself has an interval spanning roughly two to seven percent. Does that change the recommendation or only the sequencing?

State the impact of your last year without inflation

Hard · self-assessment · attribution · counterfactual reasoning

You are asked what your work was worth over the last year. Two candidates are on the table. First: a dunning-schedule change you analysed, shipped in March with no holdout, after which the involuntary share of gross churn fell three points. Second: a metric-definition change you drove, which nobody can attribute revenue to. Deliverable: the impact claim you make for each, the counterfactual attached to each, and which one you lead with in a performance review — with the reason.

Approach

1. The probe is whether you apply the same causal standard to your own work as to a product experiment. Attach the counterfactual before the claim: involuntary share of gross churn responds to card-updater coverage, billing_provider mix, retry schedules and the dunning horizon you read it at, any of which can move three points with the schedule change contributing nothing.
2. Say which evidence would separate those, then say plainly which you actually have. A staggered rollout by billing_provider, a holdout, or at minimum a flat pre-period series would each support a different strength of claim; with none of them, the honest claim is a contribution to a favourable movement, stated as such.
3. Check the classification itself before claiming the movement is real: the split between cancelled_voluntary and failed_involuntary must be computed only after the dunning horizon has closed for every period in the month, or retries still in flight are counted as churn and the share reads too involuntary in one direction and corrects in the other.
4. Price your contribution rather than the outcome. The value of analysis that changed a decision is the decision's delta multiplied by the probability the decision would not have been taken without it, and being explicit about that second factor is most of what honesty means here.
5. Make the definition-change claim concrete rather than apologetic: name the decisions that would have been taken on the wrong number, the reports it reconciled, the recurring argument it closed, with dates. Then lead with whichever claim survives questioning, not whichever carries the larger number, because an inflated first claim makes the second unbelievable.

Follow-up

• Your manager writes up the churn improvement and credits it to you. Do you correct it, and to whom?
• How would you have designed the March rollout so that attribution was possible, at what cost in delay?
• Name something you worked on last year that had no impact, and say what you learned from that rather than from the wins.