Block Usa Data Scientist Interview Questions & Guide 2026

The 60-second brief

The Data Scientist role at Block Usa is critical in harnessing data to drive strategic decision-making and enhance product offerings. You will play a pivotal role in analyzing complex datasets to uncover insights that shape business strategies, optimize operational efficiency, and improve user experiences. Your findings will directly influence product development, marketing initiatives, and customer engagement, making your work vital to the company's success.

SQL is seldom the hardest round and is often the one that eliminates people. The working bar is usually window functions, correct deduplication, and joins that do not silently fan out rows, rather than obscure syntax.

Focus areas

• Define a qualified stream and defend it
• Measure catalogue breadth beyond head consumption
• Split voluntary churn from involuntary payment failure

Overview

In this position, you will collaborate with cross-functional teams, including engineering, product management, and marketing, to address key challenges and leverage data for innovative solutions. You will engage with large-scale data systems and be involved in projects that span various domains, including time series analysis and predictive modeling. This role not only demands technical expertise but also a keen understanding of the business context, making it a uniquely impactful and rewarding opportunity within Block Usa.

The Block Usa interview process

Block Usa candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Screening Interview

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Panel Interview

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

• Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
• Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
• Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

• Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
• For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
• Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Reading recommendation quality from click-through or play-through on logged slates.

Slate position drives interaction independently of relevance, and the ranker that produced the log also decided what got exposure, so logged data is confounded by the policy you are trying to evaluate. Inverse-propensity estimators fix this only when the logging propensities are known and every item had a non-zero chance of being shown; without exploration slots that positivity condition fails silently and the estimate is confidently wrong. The visible symptom is a feedback loop: offline metrics improve, consumption concentrates further into the head of the catalogue, and the online test does not reproduce the offline gain.

Treating the account as the person, or the profile as the person.

A household account carries several people, profiles frequently are not switched, and shared-screen, car and speaker playback often lands on a default profile with no user behind it. Personalisation trained on a profile therefore learns a mixture, retention regressions attribute one member's behaviour to another, and a per-account taste statistic describes a household composite. The practical consequence is that apparent personalisation wins can be device or context effects, so any identity-level claim needs a stated unit and an acknowledgement of what that unit actually aggregates.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Questions & practice

Describe a project where you used machine learning. What algorithms did you consider?

Medium · machine learning and modelling

Approach

1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
2. Check what information would not exist at prediction time, and exclude it.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

What statistical methods would you use to evaluate the performance of a model?

Medium · machine learning and modelling

Approach

1. Pick an evaluation metric that matches the cost of each error type, not a default.
2. Set a baseline first, so any model has something honest to beat.
3. Check what information would not exist at prediction time, and exclude it.

Follow-up

• How would you choose the decision threshold, and who owns that choice?
• Where could label leakage enter this setup?

Permutation test for hours per account across two ranker arms

Medium · permutation-test · heavy-tails · experiment-analysis

arm_hours holds one row per account: account_id, arm in {control, treatment}, qualified_hours over a seven-day window. Roughly 40 thousand accounts per arm, about 38 percent of them at zero hours, and the non-zero tail is long. Without calling a library test function, write a permutation test on the difference in mean hours with 10,000 relabellings. Then run it as two parts: the difference in the share of accounts with any hours, and the difference in mean hours among accounts with hours. Report all three and say which belongs in the readout.

Approach

1. Shuffle the labels, not the data. Draw a permutation of the arm indicator over accounts, which is the unit that was randomised, and hold the hours vector fixed.
2. Make each replication O(n): precompute the grand sum and the arm sizes, so a shuffled difference is the treated subset sum over n_t minus (grand sum minus that subset sum) over n_c. Ten thousand replications then take seconds instead of a minute.
3. Use the two-sided p-value (1 + count of permuted absolute differences at or above the observed) divided by (B + 1). The plus one is not cosmetic: it makes the p-value valid rather than optimistic, and it means the smallest reportable value here is 1/10001, not zero.
4. For the two-part version, run the same machinery on the 0/1 indicator for the rate, then on the non-zero subset for the conditional mean, and say plainly that conditioning on a post-treatment outcome breaks the randomisation, so the conditional arm is descriptive rather than causal.
5. Report the rate test and the overall mean test as the result, with the conditional mean as colour, and give the effect size in hours beside each p-value, because at 80 thousand accounts almost anything is detectable.

Worked solution (30 min)

1. Pull hours into a float array and arm into a boolean array; record n_t, n_c, the grand sum and the observed difference of means.
2. Loop B = 10,000 times, taking np.random.permutation(n)[:n_t] as the treated index set and accumulating hours[idx].sum().
3. Convert each shuffled treated sum into a difference of means, compare its absolute value against the observed, count, and apply the plus-one form.
4. Repeat with hours replaced by (hours > 0).astype(float) for the rate test.
5. Subset to hours above zero, re-run for the conditional mean, and assemble a three-row result carrying effect, p-value and the unit each is defined on.

Expected result: Three rows: the mean-hours difference with a permutation p-value bounded below by 1/10001; the streaming-rate difference in percentage points with its own p-value; and the conditional mean difference among non-zero accounts, labelled descriptive because it conditions on a post-treatment outcome.

Verify it yourself

• The permutation distribution of the mean difference is centred on zero within its own Monte Carlo error. A centre away from zero means the labels were not actually being reshuffled between replications.
• The permutation p-value on the rate agrees to two decimal places with a two-proportion z-test, which is a fair cross-check because a 0/1 indicator is not heavy-tailed.
• Doubling B leaves the p-value stable to within roughly 1 over sqrt(B). A p-value that moves a lot between runs is not yet resolved.
• Multiplying every hours value in both arms by 60 leaves all three p-values identical under the same seed and scales only the effect sizes, since a permutation test is invariant to a common positive rescaling.

Follow-up

• The permutation p-value on the mean is 0.03 and the rate test is flat. What is the most likely explanation, and does it change the decision?
• How would CUPED on pre-period hours change your power here, and what would disqualify a covariate?
• Accounts are households. Does that affect the validity of this test, or only its interpretation?

Sessionise playback into listening sessions with an idle gap

Medium · window functions · gaps and islands · sessionisation

fct_stream has profile_id, stream_id, content_version_id, started_at, ended_at (nullable when the client stopped reporting) and played_seconds. Collapse consecutive streams by the same profile into listening sessions: a new session begins when a stream starts more than 30 minutes after the latest end reached by any of that profile's earlier streams. One profile can play on two devices at once, so streams overlap and 'the previous stream' is not well defined — the idle gap is measured against the running maximum end, not against the row immediately before. Treat a NULL ended_at as started_at plus played_seconds. Return one row per session with profile_id, session_start, session_end, stream_count, distinct content_version_id count and total played_seconds. Use window functions; a self-join on fct_stream is not an acceptable answer.

Approach

1. In a first CTE compute effective_end = COALESCE(ended_at, started_at + played_seconds * INTERVAL '1 second'). This both fills the abandoned-client case and bounds it, since played_seconds is advancing playback only and cannot run past the true end.
2. Take prev_max_end = MAX(effective_end) OVER (PARTITION BY profile_id ORDER BY started_at, stream_id ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING), and flag is_new_session = (prev_max_end IS NULL OR started_at > prev_max_end + INTERVAL '30 minutes'). LAG(effective_end) is the reflex here and it is wrong under concurrent devices: with A 10:00-12:00, B 10:05-10:10 and C starting 10:45, LAG hands C the end of B at 10:10, so C opens a new session 35 minutes into a session still running until 12:00. The NULL branch is not defensive noise either: the first row of every profile has no preceding frame and would otherwise fall outside any session.
3. Number the sessions with SUM(is_new_session::int) OVER (PARTITION BY profile_id ORDER BY started_at, stream_id ROWS UNBOUNDED PRECEDING). A running count of boundaries is the gaps-and-islands device. Specify ROWS rather than letting the default RANGE frame apply, because RANGE lumps peer rows that share a started_at into one frame position — the same reason the ORDER BY carries stream_id as a tie-break.
4. Group by (profile_id, session_number) and aggregate: MIN(started_at), MAX(effective_end), COUNT(*), COUNT(DISTINCT content_version_id), SUM(played_seconds).
5. The running maximum makes the sessions disjoint in wall-clock time, but it does not make SUM(played_seconds) an elapsed-time figure: two devices playing at once contribute overlapping seconds, so the total can exceed session_end minus session_start. Report it as consumption and say which of the two the requester asked for.

Worked solution (30 min)

1. Materialise the effective_end CTE and check for rows where effective_end < started_at, which means negative played_seconds or a clock-skewed ended_at and must be quarantined before anything else.
2. Add the running MAX and the boundary flag; verify that exactly one row per profile has prev_max_end IS NULL and that it is flagged as a session start.
3. Add the running SUM to number sessions; confirm the numbers are dense and begin at 1 within each profile.
4. Aggregate and reconcile: SUM(stream_count) across all sessions must equal the input row count.
5. Run the overlap fixture — one profile with A 10:00-12:00, B 10:05-10:10 and C at 10:45 — and confirm it collapses to one session. With LAG(effective_end) in place of the running MAX it splits into two, and the second session's start, 10:45, sits inside the first session's span.

Expected result: One row per (profile_id, session_number). Session numbers are dense from 1 within each profile, SUM(stream_count) equals the input row count exactly, and a profile's sessions are disjoint in wall-clock time: each session_start exceeds the previous session's session_end by more than 30 minutes.

Verify it yourself

• Input row count equals SUM(stream_count); a shortfall means rows fell out of the window partition or the NULL branch of the boundary flag.
• For any profile, each session's session_start exceeds the previous session's session_end by more than 30 minutes. This holds only because the boundary is measured against the running maximum end — under LAG(effective_end) the overlap fixture above produces a session whose session_start, 10:45, precedes the prior session_end of 12:00.
• Changing the gap from 30 to 60 minutes reduces the session count or leaves it unchanged, and never raises it.

Follow-up

• Two devices on one profile play simultaneously. Is that one session or two, and what changes in the query for each answer?
• Re-run at a 10-minute and a 60-minute idle gap. How far does session count move, and which number do you publish and defend?
• Can SUM(played_seconds) for a session ever exceed session_end minus session_start, and what does it mean when it does?

Pick one canonical version per work without losing remaster plays

Medium · deduplication · window functions · dimension grain

dim_content_version has content_version_id, content_id (stable across versions of the same work), title, content_type, duration_seconds, catalogue_added_ts, is_active_version and updated_at. A remaster, re-edit, dub or re-encode is a new row sharing content_id, and more than one row per content_id can carry is_active_version = true. Produce exactly one canonical row per content_id, preferring is_active_version = true, then the latest catalogue_added_ts, then the highest content_version_id as a deterministic tie-break. Then join fct_stream (content_version_id, played_seconds, is_qualified, started_at) and report qualified hours per content_id for last month.

Approach

1. ROW_NUMBER() OVER (PARTITION BY content_id ORDER BY is_active_version DESC, catalogue_added_ts DESC NULLS LAST, content_version_id DESC), keeping rn = 1. The third sort key exists only so two runs return the same row; without it the winner among ties is whatever the engine happened to produce, and the result diffs against yesterday for no reason.
2. Spell NULLS LAST out. In PostgreSQL, DESC defaults to NULLS FIRST, so a version with no catalogue_added_ts would win the canonical slot ahead of every dated one. Other engines order NULLs differently, which is exactly why the clause belongs in the query rather than in your head.
3. Aggregate the streams first, at content_id grain, by joining fct_stream to dim_content_version on content_version_id and grouping on content_id. Every version's plays belong to the work.
4. Only then attach the canonical attributes, with a join from the aggregate to the rn = 1 CTE on content_id. Inner-joining fct_stream to the canonical row instead discards every play of every non-canonical version, which is most of the catalogue's remasters and all of its dubs.
5. Report the count of content_ids carrying more than one is_active_version row as a data-quality line beside the result, since that condition is what made the dedup necessary and its size decides whether this is a query fix or a pipeline fix.

Follow-up

• Which grain should hours be reported at, content_id or content_version_id, and who is the consumer of each?
• Rights expire per version, so one version's window can be closed while another is live. What does your canonical pick then mean for an availability report built on the same CTE?

How would you evaluate the potential impact of a new business initiative using data?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How do you ensure the reliability of your data analysis results?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How do you prioritize your work when managing multiple projects?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Describe your thought process when tackling a data-related problem you had not encountered before.

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Fix the population and the time window before naming any metric.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would you design an experiment to test a new feature in a product?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. State the primary metric and the minimum effect worth shipping, then size the test.
3. Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

Can you walk us through your data cleaning process?

Medium

Approach

1. Work from the decision backwards to the evidence you would need.
2. Clarify what is being asked and what a complete answer would contain.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Read a test that won early and faded late

Medium · peeking · sequential testing · novelty effect

A ranker test was declared for 21 days at 80 percent power for a 1.0 percent lift in qualified hours per account-week, alpha 0.05 two-sided, with every eligible account ramped on day one. The team watched a daily dashboard. It crossed p < 0.05 on day 3 reading plus 2.9 percent, read plus 1.7 percent at p = 0.002 on day 9, and at day 21 reads plus 0.6 percent at p = 0.09. The same dashboard reports weekly slices of plus 2.1, plus 0.4 and minus 0.7 percent. Reconcile the cumulative readings against the slices and against the declared power before interpreting either, say what the day-21 p-value is worth given the daily looks, and say what you would run next.

Approach

1. Reconcile the dashboard against itself before interpreting anything on it. With a day-one ramp, exposure is roughly flat across the window, so each cumulative estimate must be the day-weighted average of the slices beneath it, and the declared MDE pins the standard error at the horizon. Both the point estimates and the p-values are checkable arithmetic; a cumulative that does not match its own slices is computed on a different population and neither number is usable.
2. Separate the arithmetic from the decision. Because the test ran to its pre-declared 21-day horizon and nobody stopped, the day-21 p-value is a valid fixed-horizon p-value needing no correction. What the daily looks damaged is the decision rule, not this number.
3. Quantify what stopping would have cost. Under the null, repeated significance testing on accumulating data at alpha 0.05 gives a true error rate near 14 percent at five looks, 19 percent at ten and roughly 25 percent at twenty, so shipping on day 3 was a bet at those odds.
4. Recognise the day-3 estimate as conditioned on having crossed. An effect read at the moment it becomes significant is selected for being large, so it is biased upward before any decay argument is made.
5. Re-index the effect on days since each account's first exposure rather than on calendar date. Novelty appears as a per-account effect decaying with exposure age; a calendar decay that vanishes under re-indexing is a release or a mix shift.
6. Separate novelty from primacy by tenure. Accounts created after the change never saw the old surface, so a genuine novelty effect is absent or much smaller in them, while primacy has the opposite sign and grows with exposure.
7. Distinguish the cumulative from the steady state. A cumulative estimate over a decaying profile is an average of a launch effect the product delivers once and a post-novelty effect it delivers forever. Neither is the other, and the second is the one a ship decision needs.
8. Conclude at the declared horizon: not significant at 0.05, a cumulative effect around plus 0.6 percent whose post-novelty component is not separable from zero, and any further looking requires an alpha-spending boundary fixed in advance.

Worked solution (25 min)

1. Reconcile the cumulative readings against the slices. Day 9 = (7 * 2.1 + 2 * 0.4) / 9 = 15.5 / 9 = 1.72 percent, matching the reported plus 1.7. Day 21 = (2.1 + 0.4 - 0.7) / 3 = 0.6 percent, matching the reported plus 0.6. Back out the within-week-one profile too: days 1 to 3 at plus 2.9 forces days 4 to 7 to average (7 * 2.1 - 3 * 2.9) / 4 = 1.5 percent, so the per-day path is 2.9, 1.5, 0.4, minus 0.7, monotone decay.
2. Reconcile the p-values against the declared power. 80 percent power for 1.0 percent at alpha 0.05 two-sided fixes the day-21 standard error at 1.0 / 2.802 = 0.357 percent, so z = 0.6 / 0.357 = 1.68 and p = 0.09. Scaling the standard error as 1 / sqrt(t) gives 0.545 percent at day 9 (z = 3.12, p = 0.002) and 0.944 percent at day 3 (z = 3.07, p = 0.002). Every reported number falls out of the design; none of them has to be taken on trust.
3. Recompute the day-21 estimate outside the dashboard with one row per account and the pre-registered cap, and confirm plus 0.6 percent at p = 0.09 reproduces.
4. Rebuild the series indexed on days since first exposure per account rather than calendar day, and plot the treatment effect against exposure age 1 to 21.
5. Split the day-21 estimate by tenure at assignment: accounts created after the change versus accounts with 90 or more days of history.
6. Check the release and rights calendar inside the window, and confirm the week-one lift is not one arm's share of a release landing.
7. Estimate the steady state from the week-three slice alone: minus 0.7 percent with a one-week standard error of 0.357 * sqrt(3) = 0.618 percent, so z = minus 1.13 and p = 0.26. It is not separable from zero, and it is the quantity the follow-up must be powered on.

Expected result: Every reported figure reconciles: the cumulative readings are the day-weighted averages of the slices, and the p-values fall out of the declared 0.357 percent standard error at the horizon. The day-21 p = 0.09 therefore stands as a valid fixed-horizon test because the declared horizon was honoured, so the ship decision is no, and roughly 25 percent is the false-positive rate the team was exposed to across 21 daily looks. The 2.9 / 1.5 / 0.4 / minus 0.7 per-day path re-indexed on exposure age is the novelty signature. Powering a re-run for plus 0.6 percent needs (1.0 / 0.6)^2 = 2.78 times the sample, about 58 days, and is the wrong design: plus 0.6 percent is an average over a decaying profile, not an effect the product will keep delivering. Power the post-novelty slice instead. Reaching a 0.7 percent MDE on a one-week slice whose standard error is currently 0.618 percent needs (0.618 / 0.250)^2 = 6.1 times that slice's exposure, about 43 days of measurement after novelty has decayed, on top of a burn-in at least as long as the decay observed here.

Verify it yourself

• Each cumulative reading must equal the exposure-weighted average of the slices beneath it, to within rounding. A gap means the cumulative and the slices are computed on different denominators or different populations, and the dashboard is reporting two incompatible experiments.
• 1 / sqrt(t) is a floor on the standard error, because within-account correlation across days makes precision accumulate more slowly than independent sampling would. The reconstructed early z-scores are therefore upper bounds. If the dashboard's own error bars are tighter than 1 / sqrt(t) implies, it is treating account-days as independent observations.
• Re-indexing on exposure age flattens a release-driven decay and preserves a novelty-driven one, but only if the ramp was staggered. With everyone exposed on day one, exposure age is collinear with calendar day and this cut cannot separate them; the tenure split is then the only available separation.
• Accounts with no exposure to the old surface should show a smaller early lift than tenured accounts if novelty is real. Equal lifts in both argue against it.
• An A/A run over the same 21 days should cross p < 0.05 at least once on roughly a quarter of replicates, which is the direct demonstration of the peeking rate rather than an argument about it.

Follow-up

• Specify a boundary that lets the team look daily and still ship at 5 percent. What does an O'Brien-Fleming boundary cost in final-look MDE compared with a Pocock boundary?
• The team proposes re-running the test powered for plus 0.6 percent. What is wrong with that target, and which quantity should the follow-up be powered on instead?
• What would primacy look like on this dashboard, and which metric would register it first?

Slate click rate jumped overnight with no ranker change

Medium · pipeline-failure · rate-denominator · logging

Home-row click rate — was_clicked over rendered rows in fct_impression (impression_id, slate_id, surface, slate_position, rendered_at, viewport_visible_ms, ranker_version, was_clicked) — rose from 4.1% to 6.8% on a single day and held there. ranker_version is unchanged across the step. Streams attributed to home_row in fct_stream (impression_id, start_source, started_at) did not rise. Using impression and stream data only, find the cause and state what you tell the ranking team, who have already claimed the improvement in a review deck. Deliverable: root cause plus the correctly restated series.

Approach

1. Ask which side of the ratio moved. A rate that steps while the downstream count is flat is almost always a denominator event, so plot clicks and impressions as separate absolute series before touching the ratio again. Do the arithmetic first as well: with the numerator fixed, a rate going from 4.1% to 6.8% pins the denominator to 4.1/6.8 of its former size, so you know the size of the impression loss to look for before you open the first query.
2. If impressions fell, locate the fall: group by surface, slate_position band and viewport_visible_ms = 0. Loss concentrated in high positions and in never-visible rows is the signature of a client or collector dropping below-the-fold renders, which removes mostly non-clicked rows and lifts the rate arithmetically with no behaviour change.
3. Prove rows are missing rather than never rendered. Left-join fct_stream rows with start_source = 'algorithmic_slate' and non-null impression_id back to fct_impression and count the ones that no longer resolve. Orphaned attributions date the breakage to the hour and are hard to argue with.
4. Look at impressions per slate_id, and require it to reconcile with the rate step. A twenty-item row now logging twelve is a truncation bug in the renderer or collector, and it accounts for the loss exactly if the distinct slate_id count is flat; a stable per-slate count with fewer slates is a sampling or partition drop upstream. The two have different owners and different backfill stories, and any per-slate figure that implies a larger loss than the rate step requires means a second thing is also wrong.
5. Restate the series on a basis the defect does not touch — clicks per slate_id, or the rate restricted to slate_position <= 8 where logging is intact — and take the dated evidence to the ranking team before the number reaches the review, not after.

Follow-up

• Impressions in the truncated tail are missing for six days. Can an experiment that ran in that window still be read, and under exactly what assumption?
• What monitor catches this within an hour, and what does it cost in false alarms on a normal release day?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

• Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
• For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
• Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

• Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
• Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
• Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

• Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
• Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
• Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

• Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
• For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
• Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

• Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
• Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
• Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

• Repeat the depth protocol on the second-ranked area with the same six-problem structure.
• Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
• Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

• Re-run the six prompts from day one under the same clock and compare both correctness and time.
• Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
• Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Give an example of how you influenced team decisions using data.

Medium · behavioural and stakeholder questions

Approach

1. Pick a story where you drove the decision, not one where you observed it.
2. Close with what you would do differently, concretely.
3. State the situation in two sentences and spend the rest on your reasoning.

Follow-up

• How did you know the outcome was caused by your change?
• What did you decide not to do, and why?

Defend a flat result on a flagship discovery launch

Medium · defending findings · statistical power · experiment readout

A new ranker ran to 50 percent of accounts for four weeks. Your read: qualified hours per active account-week is +0.4 percent with a 95 percent interval of [-0.9 percent, +1.7 percent]. Meanwhile the share of qualified streams with start_source = 'algorithmic_slate' rose six points and the share with start_source = 'search' fell five. The team reads the slate shift as the win and wants to ship. The launch review is Friday. Deliverable: a five-minute verbal position and the one table you put on the screen.

Approach

1. The probe is whether you can hold a position under social pressure without overclaiming in the other direction. Separate the two claims explicitly: the start-source shift is well measured and real; the hours effect the launch was justified on is not distinguishable from zero at this sample size. Those are different statements and only one is contested.
2. Do the power arithmetic before the meeting, using the observed per-account variance on a right-skewed hours metric. Arrive able to say "at this n we could not have detected less than X percent", which is a fact, instead of "it didn't work", which is an opinion the room can simply disagree with.
3. Show the substitution as arithmetic, not interpretation: decompose treatment hours by start_source and show slate hours rising by approximately what search hours fell. A start-source mix shift with no change in total is exactly what displacement looks like, and the decomposition sums to the total so there is nothing to argue about.
4. Refuse the symmetric overclaim. The interval's upper bound is commercially meaningful, so "the ranker does nothing" is not supportable either; say that out loud, because it is what makes the rest of your position credible.
5. Offer the falsifiable next step with a price: a longer read powered for the hours effect, or CUPED on a pre-period consumption covariate measured before assignment, with the required account-weeks stated. Put the disagreement on the pre-registered metric, not on the team's judgement.

Follow-up

• The PM argues start-source mix is a leading indicator of retention. How would you test that claim, and how long would it take?
• Suppose the interval had been [+0.1 percent, +1.9 percent]. Does your position change, and by how much?
• The decision goes against your read and the ranker ships. What do you do on Monday?

Walk through an analysis you shipped that was wrong

Medium · error ownership · data quality · postmortem

Describe a case where you delivered a result that was later shown to be wrong, and it had already been acted on. Cover how the error surfaced, whether you or someone else found it, what the wrong number caused, and what you changed afterwards. Prepare an example whose root cause was a definition, a denominator or a join — not a transcription slip. The interviewer will push on the mechanism, not the apology. Deliverable: a four-minute account that ends with a specific control now running in a pipeline.

Approach

1. The probe is whether your account has a mechanism in it. Choose an error that generalises — a denominator that silently changed population, a join that fanned rows, a metric partitioned on event_date while offline playback arrived days late and landed in the wrong partition — rather than one that only teaches you to check your typing.
2. State the blast radius factually and early: which decision was taken, how long the number stood, what it cost. A candidate who softens this is answering a different and easier question, and the interviewer can hear the substitution.
3. Explain how it surfaced without adjusting who found it. The generalisable detail is why your own checks did not catch it, which is a statement about your checks rather than about your luck.
4. Name the control you added and where it now lives: a row-count assertion after the fan-out join, a reconciliation that recomputes a closed day after late-arriving offline playback and alerts above a threshold, a denominator assertion inside the query. A fix that lives in a pipeline is different in kind from a resolution to be more careful.
5. Close with whether the control has fired since, or how you tested that it would. That single sentence is what separates a fix from an intention, and interviewers ask for it when candidates do not offer it.

Follow-up

• Why didn't your own review catch it? Be specific about what you did check.
• What class of error would that control still not catch, and what would you add next?
• Have you found an error in someone else's published analysis since? How did you raise it?