Interview guides · Handshake - stryder

Handshake - stryder Data Scientist Interview Questions & Guide 2026

Handshake - stryder data scientist prep: probability, sampling distributions, and stating the assumption behind every test you run.

The 60-second brief

A Data Scientist at Handshake - stryder plays a pivotal role in shaping the future of early-career recruiting. Unlike traditional job boards, Handshake - stryder operates as a complex, two-sided marketplace connecting college students, university career centers, and employers. As a data scientist, you are responsible for building the algorithmic matching engines, search systems, and recommendation pipelines that ensure students find relevant opportunities and employers discover qualified, diverse talent.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Diagnose whether a market is supply- or demand-constrained
Design switchback tests that survive supply-side interference
Price incentives against contribution margin, not gross bookings

Overview

Your work directly impacts user engagement, application conversion rates, and the overall efficiency of the marketplace. This is not a standard content-consumption environment; it is a reciprocal matching ecosystem. The models you build must account for mutual interest—similar to a dating platform—rather than one-way recommendation systems. This requires a deep understanding of marketplace dynamics, network effects, and cold-start problems for students with limited professional histories.

By joining this team, you will tackle high-dimensional data, design sophisticated machine learning models, and translate complex behavioral patterns into actionable product features. It is a highly collaborative role where you will work alongside product managers, software engineers, and business stakeholders to drive strategic decision-making and product evolution.

The Handshake - stryder interview process

Handshake - stryder candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Screen

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Virtual Onsite Panel

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Pooling across markets with a shifting mix

Aggregate take rate, order frequency and fill rate all move when the market mix moves, because a fast-growing newly launched market has lower prices, lower frequency and different supply density. A company-level metric can decline while every individual market improves, which is Simpson's paradox with a launch calendar attached. Report within-market deltas, or fix the mix by weighting to a reference period, and say which one you did.

Treating abandoned requests as missing rather than censored

Consumers who give up before matching are censored observations, so the mean time-to-match computed over matched requests understates true waiting and improves mechanically whenever abandonment rises. A change that makes people quit sooner will look like a latency win. Use survival methods with abandonment as the censoring event, or never report time-to-match without reporting abandonment next to it.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Questions & practice

Solve a medium-level Python algorithm challenge involving array manipulation or string processing.

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Trailing 30-day prior-order counts without rolling or asof

Medium · vectorisation · searchsorted · window semantics

You are given orders: order_id, consumer_id, completed_at_utc (tz-aware UTC), about two million rows, one row per completed order. For every order, compute how many completed orders the same consumer had in the 30 days before that order, counting the window as [t - 30 days, t) so the order itself and any exact-timestamp twin are excluded. You may not use groupby().rolling, merge_asof, or apply over groups. Return the input frame, in its original row order and index, with one added integer column prior_30d.

Approach

Sort once by (consumer_id, completed_at_utc) while keeping the original index, and move to NumPy int64 nanoseconds; the whole problem is two searchsorted calls per group, and a Python loop over two million rows is what makes this fail on time rather than on logic.
Find group boundaries with np.flatnonzero on a consumer_id change mask instead of iterating a groupby object, then slice the timestamp array per block.
Within a block, prior_30d[i] = searchsorted(ts_block, t_i, 'left') - searchsorted(ts_block, t_i - 30 days, 'left'), which is exactly the half-open window and needs no special case for the first order.
State the tie rule out loud: side='left' on the upper bound means simultaneous orders do not count each other, which is the defensible choice when the timestamp has second resolution.
Scatter the result back through the sort permutation so the added column aligns with the caller's frame, and assert the index is unchanged before returning.

Worked solution (30 min)

order_idx = np.lexsort((ts_ns, consumer_ids)); ts = ts_ns[order_idx]; cid = consumer_ids[order_idx].
starts = np.concatenate(([0], np.flatnonzero(cid[1:] != cid[:-1]) + 1, [len(cid)])).
For each block, lo = np.searchsorted(block, block - 308640010**9, 'left'); hi = np.searchsorted(block, block, 'left'); out_block = hi - lo.
Concatenate block results, then invert: result = np.empty(n, int); result[order_idx] = out_sorted.
Attach as orders['prior_30d'] and assert the frame's index and row order are identical to the input.

Expected result: The original frame plus an integer prior_30d column with no NaN, zero for every consumer's first order, and a per-consumer maximum of at most (that consumer's order count minus 1).

Verify it yourself

Brute-force O(n^2) on a 2,000-row sample and assert exact equality with the vectorised result.
assert frame.index.equals(original_index) and that all other columns are untouched.
A consumer with two orders exactly 30 days apart lands on the boundary the way your stated half-open rule says; write that case as a two-row fixture.
Runtime is a few seconds on two million rows; if it is minutes, a Python-level loop survived.

Follow-up

How does the implementation change if the count must be restricted to the same market?
This column will feed a model scored at request time. What leakage would you check for, and which timestamp defines the cut-off?

Permutation test and block bootstrap for a switchback

Medium · switchback · permutation test · clustered variance · bootstrap

You have mh: market_id, hour_start_utc, block_id, arm ('treat' or 'control'), eligible_requests, matched_in_sla. One row per market-hour, fourteen days in a single market, randomised in two-hour blocks. Without scipy.stats or statsmodels, produce three things: the effect estimate as the difference in ratio-of-sums fill rate between arms; a two-sided permutation p-value that reassigns arm at the block level holding the observed number of treated blocks fixed; and a 95% confidence interval from a bootstrap that resamples whole blocks with replacement. Use 10,000 iterations for each.

Approach

Collapse to the randomisation unit first: sum eligible_requests and matched_in_sla per block_id and keep the arm label. Every later operation runs over 168 blocks, which is what makes the variance estimate honest.
Estimate the effect as ratio of sums within each arm, not as the mean of per-block rates, because blocks carry very different volume and the mean of rates answers a different question.
Permutation: shuffle the arm vector across blocks keeping the treated count fixed, recompute the same statistic, and report p = (1 + count of |stat*| >= |stat_obs|) / (B + 1). The plus-one is the correct finite-sample form and keeps p strictly positive.
Bootstrap: resample block indices with replacement within each arm, recompute the ratio difference, and take the 2.5th and 97.5th percentiles; resampling blocks rather than hours propagates both the rate and the volume variation.
State the effective sample size as 168 blocks rather than the request count, and convert that into the minimum detectable effect the design actually supports.

Follow-up

Blocks are two hours and the median order lasts 25 minutes. Where does carryover leak across the boundary, and what would you do with the first minutes of each block?
How would you add covariate adjustment on pre-period market-hour fill rate without breaking the permutation argument?

Given a dataset of job applications, write a Python script to identify the top three industries each university's students apply to.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Explain how you would optimize a slow-running SQL join involving a massive student activity log table and a smaller employer metadata table.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Write a SQL query to calculate the month-over-month retention rate of active student profiles on the platform.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Rolling 28-day completed orders per active consumer, computed daily

Hard · running aggregates · window functions · distinct counts

fct_request has request_id, consumer_id, market_id, requested_at_utc. fct_order has order_id, consumer_id, market_id, order_status, completed_at_utc. For each market and each of the last 90 report days, return completed orders whose completed_at_utc falls in the trailing 28 days including the report day, divided by the count of distinct consumers with at least one request in that same 28 days. Emit a row for every report day, including days with no activity. COUNT(DISTINCT ...) as a window aggregate is not available to you.

Approach

Split the problem on the property that matters: the numerator is additive across days and the denominator is not. Compute them separately and join on (market_id, report_day) at the end.
Build one per-market calendar that runs from 27 days before the first report day to the last report day, 117 rows per market, and report only the last 90. Both trailing windows read 27 days of history, so a calendar that starts on the first report day under-counts the first 27 rows of the output with no error anywhere.
Numerator: aggregate completed orders to (market_id, order_day), LEFT JOIN the calendar so empty days exist as rows, then SUM(...) OVER (PARTITION BY market_id ORDER BY day RANGE BETWEEN INTERVAL '27 days' PRECEDING AND CURRENT ROW). The RANGE frame is only safe because the calendar join made every day present.
Denominator: reduce requests to distinct (consumer_id, market_id, activity_day). Each activity day makes that consumer active on report days d through d+27, and those windows overlap, so naively summing per-activity-day contributions counts a frequent consumer many times over.
Make the coverage disjoint with LEAD: for activity day d with next activity day d_next, exclusive coverage is d through LEAST(d + 27, d_next - 1), and d through d + 27 when d_next is NULL. Emit +1 at each interval start and -1 the day after its end, then take a running SUM over the calendar to get the distinct active count per day in one pass. Feed the emission from the warm-up days too, or a consumer whose last request predates the window drops out of denominators they still belong in.
Join numerator to denominator on (market_id, report_day) and divide with NULLIF. Keep both sides on UTC days or convert both to market-local days; using local days on one side and UTC on the other shifts the ratio by up to a day of volume.
Report the denominator next to the ratio. The ratio falls whenever acquisition succeeds, because new consumers transact less than tenured ones, so the two numbers are only interpretable together.

Worked solution (45 min)

Build the per-market calendar first and confirm it has 117 rows per market, 90 report days plus the 27-day warm-up, zero-activity days included.
Validate the denominator for one market and one report day against a direct COUNT(DISTINCT consumer_id) with a plain BETWEEN filter; the running-sum answer must match exactly, not approximately.
Validate the numerator the same way on the same day with a plain range filter.
Join, divide with NULLIF, and read the 90-day series for step changes that line up with calendar effects rather than product changes.

Expected result: One row per (market_id, report_day) for 90 days, with an integer numerator, an integer denominator equal to the true distinct consumer count over the trailing 28 days, and a ratio. A report day with no requests of its own still carries a large positive denominator, because the denominator counts every consumer who requested anywhere in the preceding 28 days; it moves by at most the consumers entering and leaving the window that day. The denominator is 0, and the ratio NULL, only where the market saw no request at all for 28 consecutive days, which in a live market reads as an ingestion outage rather than a quiet day.

Verify it yourself

On three randomly chosen (market, day) pairs the running-sum denominator equals a direct COUNT(DISTINCT consumer_id) with a BETWEEN filter.
Take a report day whose own request count is zero: its denominator sits between its neighbours' and does not collapse. A 0 there means the denominator was computed from the report day's requests instead of the trailing 28 days.
The denominator on any day never exceeds the number of distinct consumers who requested in that market across the whole 90 days.
Day-over-day the denominator changes by at most the number of consumers whose coverage interval starts or ends that day; a jump larger than that means intervals are overlapping and being double counted.
Summing per-market denominators exceeds the company-level distinct count whenever consumers transact across markets; exact equality means either nobody crosses markets or the denominator is being computed wrong.

Follow-up

The ratio has fallen for six straight weeks while completed orders grew. What is your first hypothesis, and which query separates denominator drift from a real engagement decline?
A consumer requests in two markets inside the same window. How does your denominator treat them, and do the market denominators sum to a company-level denominator?

Design a switchback for a batched matching algorithm

Hard · switchback · carryover · delta method · power

A matching change batches requests for up to 15 seconds before assigning, instead of assigning greedily. Median engaged time per order is 18 minutes. You have six markets, 14 days, and 12 active hours per day. The primary metric is SLA fill rate from fct_request. Design the switchback: block length, burn-in, assignment scheme, the estimand, and the variance estimator. Then compute the minimum detectable effect at 80% power and two-sided alpha = 0.05, given a block-level fill-rate standard deviation of 6 percentage points.

Approach

Set block length from the carryover horizon, not from convenience. An accepted order occupies a provider for roughly the engaged duration, so assignments made under one arm constrain the pool available in the next block. With an 18-minute median, use 1-hour blocks and discard the first 20 minutes of each block as burn-in, analysing the trailing 40 minutes. Shorter blocks buy more units but leak more of the previous arm into every block, pulling both arms toward the middle and biasing the effect toward zero.
Randomise arm independently per market-hour, stratified so every market gets a balanced count within each hour-of-week. Do not alternate deterministically: a fixed ABAB pattern aliases with the daily demand cycle and with any weekly operational rhythm, and no variance estimator will warn you.
Count the real sample. Six markets times 12 hours times 14 days is 1,008 blocks, about 504 per arm. That is the sample size. The millions of requests inside those blocks share a treatment assignment and a supply pool, so they are not independent units and cannot be counted as such.
Fix the estimand before the variance estimator, because there are two defensible choices and mixing them is the common error. Equal weight per block, the mean of block-level fill rates, matches the randomisation unit and yields a plain two-sample comparison. Demand-weighted, the ratio of summed numerator to summed denominator, matches the reported business metric and needs the delta method: Var(R) = (1/n) * (1/mu_D^2) * [Var(N) - 2RCov(N, D) + R^2*Var(D)] with R = mu_N / mu_D. State which one you are powering and reporting.
Compute the MDE on the equal-weight estimand: (z_0.975 + z_0.80) * sigma * sqrt(2 / n_per_arm) = 2.80 * 0.06 * sqrt(2/504) = 2.80 * 0.06 * 0.0630 = 0.0106, about 1.1 percentage points absolute. If the change is expected to move fill rate by less than that, the test as scoped cannot decide, and the honest options are more markets, more days, or variance reduction rather than a longer squint at the result.
Check serial correlation before trusting the interval. Adjacent blocks inside one market are correlated, so run an A/A over historical blocks; if residuals are autocorrelated, cluster at market-day rather than block, or use randomisation inference over the actual assignment mechanism, which is exact and assumes nothing about block independence, and recompute power with the resulting design effect.

Worked solution (40 min)

Fix 1-hour blocks against the 18-minute carryover horizon and define a 20-minute burn-in excluded from analysis.
Count blocks: 6 markets * 12 hours * 14 days = 1,008 blocks, 504 per arm.
Compute the standard error of the difference on the equal-weight estimand: 0.06 * sqrt(2/504) = 0.06 * 0.0630 = 0.00378.
Multiply by 2.80, the sum of z at two-sided 5% and z at 80% power, giving an MDE of 0.0106.
Compare 1.1 percentage points against the shipping threshold and decide whether to extend the window, add markets, or apply pre-period covariate adjustment (rho = 0.7 gives sigma 0.0428 and an MDE near 0.76 percentage points).
Run an A/A over historical blocks to confirm the realised standard error and to test for residual autocorrelation before launching.

Expected result: One-hour market-hour blocks with a 20-minute burn-in, stratified by hour-of-week, 504 blocks per arm, an equal-weight-per-block estimand with block-clustered or randomisation-based inference, and a minimum detectable effect of about 1.1 percentage points absolute on fill rate.

Verify it yourself

An A/A switchback replayed on historical blocks should return a non-significant effect and a standard error near 0.0038.
Dropping the burn-in changes the requests inside each block but not the block count; if fill rate in the burn-in window differs from the analysed window within the same arm, carryover is real and the block is too short.
The equal-weight and demand-weighted estimates will differ whenever block sizes vary, so the reported number, the powering calculation and the variance estimator must all refer to the same one.

Follow-up

Median engaged time rises to 45 minutes in one vertical. What changes in the design, and what does it cost in power?
What MDE would you get with covariate adjustment at rho = 0.7 on pre-period market-hour fill rate?
The team wants provider earnings per online hour from the same test. Is the market-hour block the right unit for that metric?

Estimate a single-market fee change without a control group

Medium · synthetic control · parallel trends · permutation inference

Regulation forces one market to cut its service fee on a known date. Randomisation is impossible and only that market is affected. You have 52 weeks of pre-period weekly series for every market: completed orders, active consumers, SLA fill rate, provider online hours and net take rate. Estimate the effect on completed orders per active consumer over the 12 weeks after the change. Specify the estimator, the donor pool and its exclusions, the pre-period fit criterion you would accept, and how you would produce a p-value with exactly one treated unit.

Approach

Choose synthetic control as the estimator. Fit non-negative donor weights that sum to one, matching the treated market pre-period outcome path plus a small set of predictors such as vertical, launch age, price level and supply density. The convexity constraint is the point: it forbids extrapolation beyond the donor support, which an unrestricted regression fit would happily do.
Build the donor pool by exclusion, not by convenience. Drop markets with their own policy or pricing change inside the window, and drop markets adjacent enough that providers or consumers can cross into the treated market within a session. A spilled-over donor is contaminated toward the treated path and shrinks the estimated effect toward zero.
Set the fit criterion before looking at the post period. Require a small pre-period RMSPE relative to the effect size that would change the decision, and validate it out of sample by fitting weights on weeks 1 to 40 and checking fit on weeks 41 to 52, so the weights are not tuned to pre-period noise.
Produce inference by permutation, because there is one treated unit and no cluster to compute a standard error over. Run the identical procedure treating each donor as if it were treated, compute the ratio of post-period to pre-period RMSPE for each, and rank the true treated market. With 30 donors the smallest attainable p-value is 1/31 = 0.032, so state that floor rather than reporting a conventional significance claim.
Add a placebo in time: move the treatment date 12 weeks earlier and confirm the method produces a gap near zero. A method that manufactures an effect on a date when nothing happened cannot be trusted on the date when something did.
Report the ratio decomposed into its parts. A fee cut moves completed orders and active consumers together, so completed orders per active consumer can stay flat while both components move substantially, and the decomposition is what the decision actually needs.

Follow-up

Three markets get the same regulation on three different dates. What changes in the estimator, and what goes wrong with a naive two-way fixed-effects specification?
Pre-period fit is excellent but the treated market is the largest in the pool. What should you suspect about the weights?
How would you separate the effect of the fee change from the effect of the consumer-facing price change it caused?

Contribution margin looks best in the least mature month

Hard · fan-out · attribution lag · development factors

Contribution margin per completed order has risen for three consecutive months and is highest in the month that just closed. It is net take minus processing fees, refunds and chargebacks drawn from fct_money_movement, over completed fct_order rows, attributed to the order's completion month. Ledger rows carry order_id, entry_type, amount_cents (signed, positive into the platform), currency_code, fx_rate_to_usd, posted_at_utc, settled_at_utc and settlement_status. Explain the trend, correct it, and state what the corrected series shows.

Approach

Check the join shape before reading any number. fct_order to fct_money_movement is one-to-many, so summing gross_booking_cents across the joined rows multiplies gross bookings by the number of ledger entries per order. Aggregate the ledger to order grain in a CTE first, then join one to one.
Respect the ledger's conventions. Amounts are signed, so add them rather than subtracting absolute values; convert each row at its own fx_rate_to_usd before summing across currencies; and fix one explicit rule for 'pending', 'failed' and 'reversed' entries, applied identically in every month, since a failed charge is not collected revenue and a reversed one must not be counted as still collected.
Measure the deduction lag instead of assuming it. Build a development table whose rows are completion months and whose columns are days since completion, holding cumulative refund plus chargeback dollars. Chargebacks in particular post on a long, right-skewed delay, so the newest month has absorbed only a fraction of its eventual deductions, and the exact fraction must be read from mature months rather than guessed.
Restate the series on an ultimate basis. Either report only months that have reached a stated maturity, for example 95% developed, or scale each immature month's observed deductions by the inverse development factor and label those points as estimates on the chart.
Re-read the trend after correction and say whether it survives. If the rise was development lag, the corrected series is flat or falling, and the correct recommendation is the reverse of the uncorrected one.
Break the corrected margin into its drivers per completed order, net take, incentive intensity, refund rate and processing fee, so that whatever the corrected trend is can be explained rather than merely asserted.

Follow-up

Your month-one development factor is averaged over nine prior months. What breaks it if the dispute policy changed six months ago?
Finance closes on posted date and you are attributing to completion month. How do you present both without implying one is wrong?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Choose between three teams' requests with one analyst week

Medium · prioritisation · stakeholder management · unit economics

Three requests arrive the same morning and you have one analyst-week. Pricing wants a fee elasticity refresh for a change scheduled in ten days. Supply wants a churn model for approved providers who never opened a session. Finance wants the monthly contribution-margin restatement that attributes refunds and chargebacks in fct_money_movement to the order's completion month rather than the posting month. Deliverable: your ranking, the criterion behind it, the smallest useful version of each, and what you say to the two teams you rank below first.

Approach

Rank on decision coupling rather than requester seniority or intrinsic interest: what decision hangs on this, on what date does it become useless, and how expensive is it to reverse if the answer is wrong.
Notice the dependency before the ranking: the margin restatement changes the denominator of every unit-economics answer, including the elasticity work, so doing it second means redoing part of the first task.
Unbundle each request into its smallest decision-bearing piece. The restatement is a change to the attribution date in one query, about half a day. The elasticity refresh needs a range and its preconditions, about two days. The churn model is the only item with no date and the longest build.
Replace the churn model with the descriptive cut that may make it unnecessary: approved-provider time-to-first-session conversion by approval cohort and acquisition_channel, from dim_user provider_approved_at_utc against the first fct_supply_session.online_at_utc, one day. If conversion collapses in one channel or one market, the fix is operational and no model is needed.
Communicate the ranking in writing where all three can see it, with the reason and the trigger that would reorder it, and give each deprioritised team a smaller concrete deliverable rather than a place in a queue.

Follow-up

The supply lead escalates to your manager - what do you say, and what do you not say?
What if the fee change's elasticity cannot be estimated observationally over the range they need?
Which of the three would you drop entirely if you lost two days to an incident?

Explain a switchback confidence interval to a non-technical executive

Easy · communicating uncertainty · switchback · decision framing

A switchback test of a dispatch-radius change ran 1,152 market-hour blocks across six markets. SLA fill rate moved +1.8 percentage points, 95% interval [-0.4, +4.0], variance clustered at the block. Those markets serve about 250,000 eligible requests a week at 88% fill and 93% completion. An executive with no statistics background wants a ship-or-wait answer inside a five-minute update. Deliverable: the two-minute spoken explanation, your recommendation, and the single condition that would change it. You may not use the words significant, p-value, or confidence interval.

Approach

Open with the decision and the recommendation, then justify; an executive who hears the caveat first stops listening before the ask arrives.
Translate both interval bounds into the unit the executive already manages: eligible requests times percentage points times completion rate gives weekly completed orders, so the range becomes 'between about 1,000 fewer and about 9,300 more completed orders a week, best single guess about 4,200 more'.
Say plainly what the range does and does not rule out: it does not rule out a small loss, and it is wide because the test has 1,152 effective units, not 250,000 consumers. Block-level randomisation is the reason the sample is small, and it is the reason the number is trustworthy at market level.
Price the two errors against each other: a reversible dispatch parameter with a bounded downside is cheap to ship and cheap to revert, so the decision rule is not 'is the effect proven' but 'is the worst case affordable and detectable'.
End with the one condition that flips you: name the monitoring metric (provider utilisation and idle time, since a wider radius can raise fill by burning provider hours) and the threshold at which you revert.

Follow-up

How many more weeks of blocks would it take to halve the width of that range, and is that worth the delay?
The executive asks 'so is it real or not' - what do you say without reaching for statistical vocabulary?
What would you monitor post-ship that the experiment itself could not measure?

Handle a request to re-cut a test after the readout

Hard · pre-registration · selection · guardrails

A four-week consumer-credit test reads flat on completed orders per active consumer and negative on contribution margin per completed order, which was the pre-registered guardrail. The sponsor asks for three re-cuts: on gross bookings instead of margin, on a seven-day window instead of four weeks, and excluding one market that 'had an outage'. One of the three is defensible under conditions. Deliverable: which you run, which you decline, the words you use to decline, and what appears in the written readout about all three requests.

Approach

Sort the three requests by one test: could this have been specified before anyone saw the result, and is it symmetric across arms. That test, not the sponsor's seniority, decides what you run.
Decline the gross-bookings switch on mechanism rather than on process: the credit operates by spending incentive dollars, and gross bookings excludes incentive spend by construction, so it cannot see the cost the guardrail exists to catch.
Decline the seven-day window because the credit's payback horizon is longer than the window, so a short read measures the redemption spike rather than the behaviour change, and because the window was chosen after the four-week result was known.
Run the outage exclusion only under stated conditions: the outage is visible in a metric nobody selected, such as requests per market-hour in fct_request, it hit both arms in the same proportion, and it is timestamped independently of this test. Report it as a sensitivity beside the primary, never as a replacement.
Put all three requests in the readout with their status and reasoning, which makes the selection visible and removes the incentive to ask again quietly; then give the sponsor a real path forward: the incentive level at which the credit would break even on contribution margin, and a powered follow-up if that level is reachable.

Follow-up

The sponsor says the guardrail was the wrong metric all along - how do you respond?
What if the outage is real but hit only the treatment arm?
How would you have pre-registered exclusions so that this conversation never happened?