LegalZoom Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at LegalZoom, you will play a pivotal role in harnessing data to drive business decisions and enhance user experiences. This position is crucial for analyzing complex datasets, developing predictive models, and generating actionable insights that impact product development and customer satisfaction. You will collaborate with cross-functional teams, including product management and engineering, to ensure that data-driven decisions align with the company's strategic goals.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

• Define numerator, denominator and window precisely
• Turn a vague request into a measurable question
• Decompose a metric move by segment and mix

Overview

The impact of your work will be felt across various products and services offered by LegalZoom, from improving user interfaces to optimizing service delivery. The role encompasses a variety of challenges, such as understanding customer behavior, conducting A/B testing, and implementing machine learning algorithms. The complexity and scale of the data you will work with make this an exciting opportunity to influence significant outcomes in a fast-paced legal technology environment.

The LegalZoom interview process

LegalZoom candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

• Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
• Whether your reason for wanting the role points at the work itself rather than the company's reputation
• Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

• Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
• Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
• Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Interview

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Where candidates lose points

Treating last-touch attribution as the causal value of a channel

The attribution label on dim_user is the output of a rule that assigns full credit to whichever touch happened to be recorded last inside a lookback window, and that rule systematically rewards channels that sit close to the conversion, especially branded search and retargeting, which largely intercept demand that already existed. Reallocating spend on those labels moves budget toward the channels that are best at being last, which is why attributed return on ad spend often improves while total signups do not. Nothing in the touchpoint data can settle this, because the counterfactual of not running the channel was never observed. The credible reads are a geo holdout or a scheduled pause, sized in advance on the total-signups metric rather than on the attributed one, and the honest framing in the meantime is that the label describes correlation with conversion and not incremental contribution.

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Generalising beyond the population the sample actually supports

State the frame the sample was drawn from and where it diverges from the population you want to talk about: time window, platform, geography, opt-in. If a group is excluded from the frame, either weight to known margins or narrow the claim rather than quietly extending it.

Questions & practice

What are the assumptions of a linear regression model?

Medium · machine learning and modelling

Approach

1. Frame the prediction: the label, the moment of prediction, and the action it triggers.
2. Say how the offline result would be validated online before it is trusted.
3. Check what information would not exist at prediction time, and exclude it.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Sessionise an event stream with gap and midnight rules

Hard · sessionisation · vectorisation · event streams

Sessionise a raw event stream. Input: a DataFrame with visitor_id, user_id (often NULL), occurred_at_utc and event_name, unsorted, up to 5 million rows. A session breaks when the gap from that visitor's previous event exceeds 30 minutes, and is force-closed at UTC midnight so no session spans two calendar dates. A gap of exactly 30 minutes does not break. Emit one row per session with session_id, visitor_id, the user_id as of the last event in the session, started_at_utc, ended_at_utc, session_date, duration_seconds and event_count. Vectorise; do not loop per visitor.

Approach

1. Sort by ['visitor_id', 'occurred_at_utc', 'event_id'] once, then express the whole problem as one boolean vector: a row starts a new session when the visitor changed, or the gap exceeds 30 minutes, or the UTC date differs from the previous row's UTC date. Cumsum that vector and you have the session key.
2. Get the comparison direction right on the gap: the rule is strictly greater than 1800 seconds, so an event at exactly 1800 seconds continues the session. Write it as gap > pd.Timedelta(minutes=30), and make the tie a test case rather than an assumption.
3. Derive the midnight break from the date change, not from inserting synthetic boundary rows. A date change implies a break even when the gap is two seconds, which is precisely the force-close rule and is why the two conditions are ORed rather than one subsuming the other.
4. Aggregate with a single groupby on the session key: min and max of occurred_at_utc, size for event_count, and last for user_id, which is correct because the frame is already sorted so 'last' is the final event in the session. That is the identity-as-of-session-end rule.
5. Compute duration_seconds as (max - min).dt.total_seconds(), which makes a single-event session 0 seconds. Say so explicitly, because a downstream mean session duration is sensitive to whether single-event sessions are 0 or excluded.

Worked solution (35 min)

1. df = df.sort_values(['visitor_id','occurred_at_utc','event_id']).reset_index(drop=True).
2. new_visitor = df.visitor_id.ne(df.visitor_id.shift()); gap = df.occurred_at_utc.diff(); new_day = df.occurred_at_utc.dt.date.ne(df.occurred_at_utc.dt.date.shift()).
3. is_start = new_visitor | (gap > Timedelta(minutes=30)) | new_day; df['session_key'] = is_start.cumsum().
4. g = df.groupby('session_key'); out = g.agg(visitor_id=('visitor_id','first'), user_id=('user_id','last'), started_at_utc=('occurred_at_utc','min'), ended_at_utc=('occurred_at_utc','max'), event_count=('event_id','size')).
5. out['session_date'] = out.started_at_utc.dt.date; out['duration_seconds'] = (out.ended_at_utc - out.started_at_utc).dt.total_seconds(); assign session_id from the sorted index.

Expected result: One row per session where session_count equals is_start.sum(), event_count sums exactly to len(input), every session's started_at and ended_at share one UTC date, and duration_seconds is at least 0 and strictly less than 86400.

Verify it yourself

• out['event_count'].sum() equals len(df); a mismatch means rows were lost to a join or a dropna.
• started_at_utc.dt.date equals ended_at_utc.dt.date on every row.
• For a planted visitor with events at 12:00:00 and 12:30:00 exactly, one session is emitted; move the second to 12:30:01 and two are emitted.
• For a planted visitor with events at 23:59:50 and 00:00:10, two sessions are emitted on consecutive dates despite a 20-second gap.

Follow-up

• Sessions are used as the denominator of a conversion rate. How does moving the inactivity gap from 30 to 45 minutes move that rate, and in which direction?
• A visitor's clock is 40 minutes ahead, so their events arrive with future occurred_at values. What does your sessioniser do, and what would you rather it did?
• The same person signs up mid-session on mobile and continues on desktop. How many sessions and how many users does your output show, and is that the right answer?

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

1. State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
2. Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
3. The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
4. Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
5. Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
6. Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Follow-up

• The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
• Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
• Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Can you discuss a specific project where you utilized SQL to extract insights?

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

1. Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
2. Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
3. Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
4. Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
5. Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

1. Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
2. Write the active-account spine and the 28-day activity CTE.
3. Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
4. Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
5. Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

• Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
• The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
• No account_id appears twice in the output.
• Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

• How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
• Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
• Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

How would you approach analyzing user engagement data for a new product feature?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

If you were tasked with improving conversion rates, what data would you analyze first?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

What metrics would you consider when evaluating the success of a product?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

Explain how you would prioritize multiple data science projects.

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Describe how you would set up an A/B test for a new onboarding process.

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. State the primary metric and the minimum effect worth shipping, then size the test.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

How would you handle missing data in a dataset?

Medium

Approach

1. Work from the decision backwards to the evidence you would need.
2. Clarify what is being asked and what a complete answer would contain.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Measure a paid channel without a user-level experiment

Hard · difference-in-differences · geo holdout · incrementality · clustered inference

Finance needs the incremental contribution of paid social before next year's budget. It carries 18% of last-touch-attributed signups in dim_user.last_touch_channel, but the ad platform will not randomise users and unexposed users are unobservable. The ad platform targets at market level, your warehouse resolves a market label onto fct_session, and 40 markets are eligible with 12 months of weekly signups derivable from dim_user. Design the read, size it on total signups rather than attributed ones, and state the identifying assumption and the check that would falsify it.

Approach

1. Say why the attributed number cannot answer the question. Last-touch assigns full credit to whichever touch was recorded nearest the conversion, which systematically rewards channels that intercept demand that already existed, and the counterfactual of not running the channel is nowhere in the touchpoint data.
2. Choose a geo holdout: randomise markets into a paused arm and a spend-as-usual arm, and measure total signups per market. The metric has to be the total, because the question is whether demand is created or merely relabelled, and attributed signups fall to zero in a paused market by construction.
3. Size on markets, since the market is the randomisation unit and therefore the variance unit. Forty markets give at most twenty per arm, so form matched pairs on twelve months of pre-period weekly signups and randomise within pair, which removes the between-market level differences that would otherwise dominate the residual variance.
4. Fix the identifying assumption in plain words: absent the pause, treated and control markets' signup trends would have moved in parallel. Falsify it by running the same estimator on placebo cut dates inside the pre-period; a non-zero placebo estimate or visibly diverging pre-trends kills the design as specified.
5. Estimate with market and week fixed effects and cluster standard errors on market. Below about forty clusters the cluster-robust error is biased downward, so use a wild cluster bootstrap or randomisation inference over the pair assignment. If only one or two markets can be paused, switch to synthetic control: weight untreated markets to match the treated market's pre-period path and use in-space placebos for inference.

Worked solution (40 min)

1. Aggregate weekly signups per market for twelve months from dim_user with is_internal = FALSE, and drop markets whose history is too short or too volatile to pair.
2. Rank markets on pre-period mean and trend, form twenty matched pairs, and randomise one of each pair into the paused arm.
3. Derive the MDE from the between-pair residual standard deviation of weekly signups, not the within-market one, because power is set by the number of pairs; express it as a percentage of total signups and compare it with the effect finance cares about.
4. Run placebo estimates at three pre-period cut dates before committing; re-pair or abandon if any is significant.
5. After the pause, estimate difference-in-differences with market and week fixed effects, report a wild cluster bootstrap interval, and convert the point estimate into incremental cost per signup against the spend withheld.

Expected result: A matched-pair geo holdout across twenty pairs, read on total signups with market and week fixed effects and market-clustered inference, yielding an incrementality ratio of incremental signups to attribution-credited signups. The identifying assumption is parallel counterfactual trends, checked with pre-period placebo cuts. The MDE is set by twenty pairs rather than by millions of sessions, and is usually far coarser than the requester expects.

Verify it yourself

• Pre-period placebo estimates should centre on zero with intervals covering it. A systematically non-zero placebo means the pairing failed and the main estimate inherits that bias.
• Treated plus control market signups in the pre-period must reconcile with the total the finance dashboard reports, otherwise the market label is silently dropping sessions.
• An incrementality ratio above 1 implies the channel drives conversions it never touched. That is possible through view-through effects the model never sees, but it demands a stated mechanism rather than a shrug.

Follow-up

• Campaign commitments mean the pause starts in different weeks in different markets. What does staggered timing do to a two-way fixed effects estimate when effects vary across cohorts?
• How long do you pause, given that paid social has a delayed conversion tail, and how do you keep that tail out of the pre-period?
• The estimate says the channel is 40% incremental. What happens to the last-touch number on the dashboard after that?

Decide whether a one-day core-action drop is real

Easy · seasonality · day-of-week · late-arriving-data

A daily dashboard counts distinct fct_event.user_id with is_core_action = TRUE, filtered on occurred_at_utc, and is read at 09:00 UTC. This morning it shows yesterday down 22% against the day before. fct_event is partitioned on received_at_utc. You have fct_event, fct_session and dim_user with thirteen months of history. Deliver a one-paragraph verdict, escalate or do not escalate, with the evidence that settles it, before anyone proposes a product hypothesis.

Approach

1. Identify which two weekdays the comparison actually spans, then pull the same weekday-pair transition for the last 52 weeks and place the observed 22% inside that distribution. A day-over-day comparison in a product with a weekday pattern is a comparison of two different populations, so the reference class is the same transition historically, not the prior day.
2. Measure partition completeness rather than assuming it. For each of the last 30 days compute the share of that day's occurred_at_utc rows that had landed by 09:00 UTC the following morning, split by surface; mobile clients buffer events offline, so the freshest partition is systematically short and the shortfall is not uniform across surfaces.
3. Recompute the same series keyed on received_at_utc. If the drop survives on both keys it is not a lateness artefact; if it exists only on occurred_at_utc it is the partition filling in.
4. Check the two exclusion flags before segmenting anything: a change in is_bot_flagged coverage or a batch of is_internal accounts entering or leaving moves a distinct-user count with no user behaviour behind it.
5. Only if the movement survives all of the above, begin the segment decomposition. Say explicitly in the verdict which of these four checks the movement passed, so the next reader does not repeat them.

Follow-up

• What publication lag would you set for this dashboard, and how would you derive the number rather than pick it?
• If you switch the metric to received_at_utc, what does that break for anyone comparing to historical figures?
• How would you detect the same problem automatically, so a human does not have to notice it each morning?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

• Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
• For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
• Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

• Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
• Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
• Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

• Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
• Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
• Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

• Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
• For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
• Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

• Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
• Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
• Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

• Repeat the depth protocol on the second-ranked area with the same six-problem structure.
• Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
• Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

• Re-run the six prompts from day one under the same clock and compare both correctness and time.
• Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
• Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

How do you handle tight deadlines and pressure?

Medium · behavioural and stakeholder questions

Approach

1. Close with what you would do differently, concretely.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• How did you know the outcome was caused by your change?
• What would you do differently if you ran that project again?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

1. Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
2. Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
3. Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
4. Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
5. Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

• The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
• Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
• Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?

Choose between three teams' requests with one analyst-week

Medium · prioritisation · stakeholder · judgement

You have one analyst-week. Three requests land the same morning. A growth team wants a paid-channel readout before a Friday spend decision. A billing team wants gross monthly revenue churn rebuilt, because the current figure recognises cancellation at canceled_at_utc rather than period_end_utc and is therefore wrong. A product team wants a dashboard for a feature launching in six weeks. All three sponsors are peers of your manager. Deliver your ranking, the explicit rule that produced it, and the message you send to the two teams you defer.

Approach

1. Recognise what is being probed: whether you prioritise on decision value and reversibility or on who asked most recently and most loudly. The generic answer sorts by importance; the strong one states a rule, applies it, and accepts the ranking it produces even where that is uncomfortable.
2. Score each request on three statable things: the decision it unblocks and the date that decision is made, the cost of being wrong in the meantime, and whether the work is one-off or compounding. A wrong published churn figure compounds, because it is quoted downstream and enters forecasts; the channel readout has a fixed date that cannot move; the dashboard has six weeks of slack.
3. Notice the tension between value and urgency rather than resolving it by feel. The churn defect is the most valuable item and the least urgent one, which is exactly the shape of work that never gets done.
4. Break the churn item in two. A one-hour severity check, sizing the gap between the two recognition points in MRR, is cheap enough to do before ranking anything and may promote the item outright. Do that first, then rank.
5. Make the deferrals concrete. Each deferred team gets a date, a reason expressed as another team's decision deadline rather than as relative importance, and the smallest thing you can hand them immediately.

Follow-up

• The dashboard team escalates to your manager. What do you say in that conversation?
• Your severity check shows churn is overstated by 15%. Does the ranking change, and does anybody need to be told today regardless of the ranking?
• A fourth request arrives Wednesday with a Thursday deadline. What comes off the list, and who do you tell first?