GitHub Data Scientist Interview Questions & Guide 2026

The 60-second brief

At GitHub, the home for over 100 million developers, data is at the core of every strategic decision. As a Data Scientist, you will operate at the intersection of product development, engineering, and business strategy. Your primary mission is to translate massive volumes of developer telemetry, repository interactions, and platform workflows into actionable insights that shape the future of software development.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

• Read NRR on a fixed account cohort
• Measure churn only on renewal-eligible accounts
• Strip CI, retry and synthetic traffic first

Overview

The impact of this role is immense. Whether you are optimizing the collaboration loops of pull requests, analyzing the adoption of generative AI tools like GitHub Copilot, or ensuring the reliability and security of the platform's infrastructure, your work directly influences how software is built globally. You will not just report on what happened; you will design experiments, build predictive models, and define the key performance indicators that guide product roadmaps.

This role requires a unique blend of technical rigor and product intuition. You will operate in a highly collaborative, remote-first environment, partner closely with product managers and engineers, and present complex analytical findings to senior leadership. To succeed, you must be comfortable navigating ambiguity and processing data at an extraordinary scale.

The GitHub interview process

GitHub candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

• Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
• Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
• The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

• Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
• Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
• Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Hiring Manager Interview

Underneath the questions about your past work sits a resourcing question. Given four things worth doing and one of you, which gets done and what happens to the rest? Managers ask because that is the daily texture of the job, and because the answer shows whether you rank work by effort or by what it changes. The weak version sorts by personal interest or by whoever asked most insistently. The strong version ties each candidate piece of work to a decision somebody downstream is waiting on, and then names the one you would drop and who you would tell.

What to demonstrate

• Whether you rank work by the decision it unblocks or by how interesting the method is
• How you describe a request you declined, and whether you can say who you said it to
• Whether your sense of how long something takes survives one follow-up question about the messy part
• How you decide something is good enough to hand over unfinished

How to prepare

• Write out your current queue and, next to each item, the decision that stays stalled until it lands. Anything with no waiting decision becomes your example of work you would cut
• Rehearse turning down a plausible stakeholder request out loud, including the smaller alternative you offered instead
• Have one case where you shipped a rough answer early and one where you refused to, with the reason that separated them

Technical Evaluation

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Final Round

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

• Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
• Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
• Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
• Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

• Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
• Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
• Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Randomising an experiment at the user level when users share an account

Two problems fire at once. Colleagues in one workspace see each other's work and talk to each other, so a treated user changes the behaviour of a control user in the same account, which violates the no-interference assumption and biases the estimate toward zero. Separately, outcomes within an account are strongly correlated, so the effective sample size is roughly n / (1 + (m - 1) * rho) for m users per account and intra-class correlation rho, not n. With rho around 0.3 and twenty users per account that is a design effect near 6.7, meaning a user-level confidence interval is about two and a half times narrower than it should be and results cross significance thresholds on noise alone. Randomise the account and cluster the standard errors.

Reporting a mean over accounts when account revenue is heavy-tailed

When a small number of accounts hold most of the revenue, the sample mean is dominated by whichever of them happens to be in the sample, and the sample variance keeps growing as more data arrives instead of stabilising. In that regime the usual central-limit-based confidence interval understates uncertainty, and a single renewal or a single large account's batch job can flip the sign of a measured effect. The fixes are to pre-register a winsorisation or capping rule before looking at the outcome, to report account counts crossing a threshold alongside the revenue figure, or to define the estimand on a bounded transform. Choosing the cap after seeing the result is a separate and worse problem, because the cap then encodes the answer.

Building features from data that postdates the prediction time

Check every feature against the timestamp at which the model would actually score, and drop anything computed from a window that includes or follows the label event. For a forecasting use case, split train and test by time rather than at random, and split by entity when the same entity recurs.

Naming a model class before naming the deployment constraints

Set out the latency budget, the label delay, the retraining cadence, the interpretability requirement and the number of labelled examples, then pick the model that fits them. A boosted-tree answer to a problem where each decision must be explained to the affected user is a well-executed answer to the wrong question.

Questions & practice

Walk through how you would build a predictive model to identify users at risk of churning from a paid GitHub Enterprise subscription.

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Check what information would not exist at prediction time, and exclude it.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Quantify billable volume created by retries after server errors

Hard · telemetry-hygiene · pandas · unit-economics

From fct_api_request (request_id, account_id, endpoint, idempotency_key, is_retry, http_status, request_at, billable_units), measure how much billable volume in a 28-day window is retry traffic that followed a 5xx. Group requests into attempt chains by (account_id, endpoint, idempotency_key) ordered by request_at; a request is error-driven if any earlier attempt in its chain returned 5xx. Requests with a null idempotency_key cannot be chained, so report them as their own class rather than assuming each is unique. Return billable_units split into first-attempt, error-driven retry, other retry and unchainable, per account.

Approach

1. Split the population before measuring anything. A null idempotency_key is not a chain of one, it is an unknown; report its share of billable_units first, because if it is 40% of volume then the headline estimate is a lower bound and the deliverable has to say so.
2. Within chainable rows, sort by (account_id, endpoint, idempotency_key, request_at) and derive 'any earlier attempt failed' with arithmetic rather than a per-group lambda: with is5 = (http_status >= 500), the per-chain cumsum minus the row's own value is positive exactly when an earlier attempt in that chain returned 5xx. A groupby-apply gives the same answer and is unusable at five million rows.
3. Do not take is_retry as the definition. It is set by the client whenever an idempotency_key is resent, which covers retries after client-side timeouts and after 4xx as well; compute the flag yourself and then cross-tabulate it against is_retry, because the disagreement is a finding in its own right.
4. Aggregate billable_units by (account_id, class) and assert the classes sum to each account's total. The spine sets billable_units to zero on 5xx responses, so the failed attempt contributes nothing and the whole inflation sits in the successful retry that follows it.
5. Report the per-account share and look at its distribution, not the fleet total. One account in a retry storm dominates any blended figure, which is the same failure that makes a fleet-wide error rate useless.

Worked solution (40 min)

1. unchainable = df.idempotency_key.isna(); report df.loc[unchainable].groupby('account_id').billable_units.sum() before proceeding.
2. keys = ['account_id','endpoint','idempotency_key']; c = df[~unchainable].sort_values(keys + ['request_at'], kind='mergesort'); c['is5'] = (c.http_status >= 500).astype(int)
3. c['attempt_no'] = c.groupby(keys, sort=False).cumcount(); c['prior_5xx'] = (c.groupby(keys, sort=False).is5.cumsum() - c.is5) > 0
4. c['cls'] = np.where(c.attempt_no == 0, 'first_attempt', np.where(c.prior_5xx, 'error_driven_retry', 'other_retry')); out = pd.concat([c, df[unchainable].assign(cls='unchainable')]).groupby(['account_id','cls']).billable_units.sum().unstack(fill_value=0)

Expected result: A per-account table of four columns summing exactly to that account's total billable_units in the window. error_driven_retry is zero for any account with no 5xx; the fleet-level error-driven share is typically low single digits while a small number of accounts sit far above it.

Verify it yourself

• out.sum(axis=1) equals df.groupby('account_id').billable_units.sum() for every account.
• Every row classed error_driven_retry has attempt_no >= 1.
• An account with no http_status >= 500 in the window has error_driven_retry == 0.
• Cross-tabulating cls against is_retry: no first_attempt row should carry is_retry True, and any that do mark chains truncated by the window boundary.

Follow-up

• An account's error-driven share is 22%. Is that the platform's fault or the client's, and what do you look at next?
• How would you define a consumption-based north-star metric that an outage cannot inflate?
• Chains straddle the 28-day boundary. How large is that bias and in which direction?

Permutation-test a consumption experiment randomised at account level

Hard · permutation-test · heavy-tails · cuped · experiment-readout

An experiment randomised 900 accounts into two arms. You have one row per account: account_id, arm, consumption_28d (billable units after launch) and consumption_pre (the 28 days before). Consumption is heavy-tailed and the largest account is several percent of the total. Write a permutation test from scratch: winsorise at the pooled 99th percentile as a pre-registered rule, use the difference in arm means of the winsorised outcome as the statistic, and obtain a two-sided p-value from 20,000 relabellings of the account-level arm vector. Report the observed effect, the p-value, and the same test on a CUPED-adjusted outcome.

Approach

1. Be precise about what the permutation test needs. Under the sharp null of no effect for any account, the outcomes are exchangeable across arm labels, and the test is valid for ANY statistic T(outcomes, labels) provided the identical function is applied to the observed labels and to all 20,000 relabellings. The pooled 99th percentile is a function of the outcome vector alone, so recomputing it inside the loop returns the same number 20,000 times: that is wasted CPU, not a bias, and hoisting it out is an optimisation rather than a correctness fix. Say plainly that capping at all changes the estimand from mean consumption to mean capped consumption; it is not a neutral cleaning step.
2. The mistake that does invalidate the test is an asymmetry between the observed statistic and the permuted ones, and the easiest way to create it is to derive the cleaning rule from the observed arm labels and then freeze it — winsorise each arm at its own observed 99th percentile, hold those two caps fixed, and permute. The observed value is then computed with caps matched to its own partition while every relabelling is scored with caps belonging to a different one, so the null distribution no longer answers the question the p-value claims to answer. A per-arm cap recomputed consistently inside every permutation is a valid test, but it estimates a contrast whose two sides are capped at different thresholds, so prefer the pooled cap on estimand grounds and pre-register it.
3. Permute the account-level arm vector, because the account is the randomisation unit. Relabelling anything finer — users, workspaces, requests — generates a null distribution narrower than the design actually supports and returns p-values that are anti-conservative.
4. Vectorise the null: tile the treatment indicator into a (B, n) matrix and permute along axis 1 with rng.permuted(..., out=...). The statistic is a difference of means, so the treated sum alone determines it and the whole null is one matrix-vector product. Use the two-sided p-value (1 + count(|stat_perm| >= |stat_obs|)) / (B + 1); the plus-one on each side is not cosmetic, it keeps the p-value away from exactly zero and keeps the test valid at finite B.
5. For CUPED, fit theta = cov(y, x) / var(x) on the pooled data and use that same theta for the observed statistic and every relabelling. Pooled theta, like the pooled cap, carries no label information, so where in the loop you compute it is again only a performance question; fitting theta within arms is what goes wrong, because the adjusted outcome then depends on the labels and an observed-label fit frozen across all 20,000 relabellings breaks the match between observed and permuted statistics. x must be measured entirely before launch, which consumption_pre is. Expected variance reduction is about 1 - corr(y, x)^2; measure the achieved reduction from the two null distributions rather than asserting it.

Follow-up

• The p-value is 0.04 with the cap and 0.31 without it. What do you report, and what did you pre-register?
• Colleagues in a shared workspace can see the treated behaviour. How does that change the design and the estimate?
• How many accounts would you need to detect a 5% lift given this outcome's distribution?

How would you optimize a query that joins a massive commits table with a smaller user metadata table to prevent memory errors?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Write a query to calculate the month-over-month retention rate of active developers based on their first repository commit.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How does the query change if the join becomes one-to-many?
• What breaks if events arrive late or out of order?

Explain the difference between RANK, DENSE_RANK, and ROW_NUMBER, and describe a scenario where you would use each to analyze user activity.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Sessionise interactive API traffic with a thirty-minute inactivity gap

Medium · window functions · sessionisation · gaps and islands

fct_api_request carries account_id, user_id (null for service accounts), request_at, traffic_class and http_status. Using only traffic_class = 'interactive' rows with a non-null user_id, group each seat's requests into sessions on a 30-minute inactivity threshold: a request more than 30 minutes after the previous request from the same (account_id, user_id) opens a new session. For one ISO week return, per account, the session count, the median session duration in minutes and the median requests per session. A single-request session has a duration of zero.

Approach

1. Filter first: traffic_class = 'interactive' and user_id IS NOT NULL. Machine traffic has no sessions in any useful sense, and leaving CI or batch rows in produces sessions that are really cron schedules.
2. Get the previous timestamp with LAG(request_at) OVER (PARTITION BY account_id, user_id ORDER BY request_at). Partitioning by user_id alone stitches one person's work across two different accounts into one fabricated session, because a human holds memberships in several accounts.
3. Flag a boundary where the lag is NULL or request_at - lag > interval '30 minutes'. Decide and state whether exactly 30 minutes continues the session; strictly greater is the conventional choice and needs to be written down either way.
4. Assign session ids with SUM(boundary::int) OVER (PARTITION BY account_id, user_id ORDER BY request_at ROWS UNBOUNDED PRECEDING), the standard running-count construction for islands.
5. Roll up to sessions with min(request_at), max(request_at) and count(*), then to accounts with percentile_cont(0.5) WITHIN GROUP (ORDER BY ...). Use medians, not means: session length is strongly right-skewed and one long-running client dominates the average.

Worked solution (30 min)

1. Build the filtered week: interactive rows with user_id IS NOT NULL inside [week_start, week_start + 7 days).
2. Add prev_at via LAG(request_at) OVER (PARTITION BY account_id, user_id ORDER BY request_at) and derive is_new_session = (prev_at IS NULL OR request_at - prev_at > interval '30 minutes').
3. Add session_seq = SUM(is_new_session::int) OVER (PARTITION BY account_id, user_id ORDER BY request_at ROWS UNBOUNDED PRECEDING).
4. Aggregate to sessions by (account_id, user_id, session_seq) taking min, max and count, with duration_minutes = EXTRACT(EPOCH FROM max - min) / 60.
5. Aggregate to accounts with count(*) AS sessions, percentile_cont(0.5) WITHIN GROUP (ORDER BY duration_minutes) and percentile_cont(0.5) WITHIN GROUP (ORDER BY request_count).

Expected result: One row per account with session_count, median_session_minutes and median_requests_per_session, computed over interactive human traffic only, where a lone request yields a session of duration zero and request count one.

Verify it yourself

• Summing request_count across all sessions equals the filtered row count for the week exactly.
• Every session duration is non-negative, and no two consecutive requests inside one session are more than 30 minutes apart when you re-derive the LAG within sessions.
• Recomputing with the partition on user_id alone yields strictly fewer sessions and longer medians for any account sharing users with another account, which confirms the partition key matters.

Follow-up

• Sessions belonging to colleagues in one account are correlated. What does that do to a t-test on session length across an experiment arm?
• A long-poll or streaming endpoint keeps a connection open for hours. How do you stop it reading as one twelve-hour session?

What metrics would you track to determine the health and engagement of an open-source community repository?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How would you measure the success and adoption of a new collaboration feature within GitHub Discussions?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

If pull request merge times suddenly increased by 15% globally, how would you investigate the root cause?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Fix the population and the time window before naming any metric.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Describe how you would design an A/B test to evaluate a new onboarding flow for first-time open-source contributors.

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. Say whether units interfere with each other, and switch design if they do.
3. State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

• How would you handle interference between treated and control units?
• What would you do if you could not randomise at all?

Separate novelty from durable lift in a console redesign

Medium · novelty-effect · time-heterogeneity · holdout

A redesigned request console was tested for four weeks, randomised at the account level. The lift in weekly successful interactive requests per account was +9.2%, +5.1%, +1.8% and +0.4% in experiment weeks one through four, and the week-four 95% interval spans plus or minus 3.1 percentage points. Accounts entered on a rolling basis as they next signed in, so week one is a different calendar week for different accounts. Say what the decaying series can and cannot establish, specify the readout that would settle it, and state what you would ship on Monday.

Approach

1. Name the distinct explanations the same curve is consistent with, because the shipping decision differs across them: a novelty effect decaying to zero; a novelty effect decaying to a small positive plateau the test has no power to see; and a genuine effect whose measured size shrinks as control accounts learn about the change from colleagues or release notes.
2. Fix the time axis before interpreting anything. With rolling entry, calendar week and weeks-since-first-exposure are different variables, and pooling them mixes the decay curve with a change in cohort composition, since the accounts that signed in on day one are systematically the most engaged. Recut on weeks-since-first-exposure and verify that plan_tier and pre-period usage mix are stable across entry cohorts.
3. Read the interval width honestly. Plus or minus 3.1 percentage points at week four cannot separate a durable +1% from zero. The defensible statement is that no durable effect larger than roughly 3.5% was detected, not that there is no durable effect, and those two sentences lead to different decisions.
4. Separate novelty from primacy using account tenure. Accounts created after launch have no prior console to be surprised by, so they cannot show novelty; if the redesign is genuinely better their curve should be flat or rising, while established accounts show the spike and decay. If both cohorts decay to zero, the effect really was novelty.
5. Specify the settling readout rather than arguing about the four weeks you have. Hold back 5% of accounts as a long-run holdout for 90 days, and compare whole ISO weeks only: usage in this domain follows a hard five-to-two weekday cycle, and a window containing four business days instead of five moves this metric by several percent with no product change.
6. Give the Monday answer. Ship if guardrails are clean and maintenance cost is low, because a decayed-to-zero effect with no harm is a neutral trade; but book none of the +9.2% in any forecast, and do not run a follow-up test on the same accounts inside the novelty window, because their baseline has not returned to steady state.

Worked solution (30 min)

1. Recut the four weekly estimates on weeks-since-first-exposure and confirm entry-week cohorts are comparable on plan_tier and pre-period usage.
2. Split each week's estimate by account tenure, created before versus after launch, and compare the shapes of the two curves.
3. Compute the horizon needed to halve the week-four interval: precision scales with the square root of exposure, so about four times the account-weeks are required.
4. Write the ship note with the 5% holdout design, the 90-day re-read date, and an explicit statement of the effect sizes that remain unexcluded.

Expected result: The series is consistent with pure novelty, with a durable plateau anywhere below about 3.5% left unexcluded. Separating the explanations requires the tenure split plus a 5% holdout re-read at 90 days, and halving the week-four interval to plus or minus 1.55 points needs roughly four times the account-weeks.

Verify it yourself

• Interval width scales with one over the square root of the sample, so quadrupling exposure should roughly halve 3.1 points. A plan claiming a larger gain is double-counting repeated weeks on the same accounts, which are correlated rather than independent.
• Accounts created after launch cannot exhibit novelty against an experience they never had, so a week-one spike in that cohort points at an assignment or instrumentation problem rather than at novelty.
• Every weekly comparison must be a whole ISO week. A window containing a different number of business days than its comparator is not a like-for-like contrast in this domain.

Follow-up

• What sample or horizon would you need to rule out a durable +1.5% at 80% power, given the week-four interval you have?
• How would you distinguish a novelty effect from a control arm that gradually learned about the change?
• The metric that decides renewal is up to eleven months away on annual contracts. What proxy do you use in the meantime, and how do you validate it once renewals land?

Activation drops six points starting at a deploy hour

Medium · instrumentation · release analysis · metric inputs

Seven-day activation, defined as an account's first request with http_status < 400, api_key_id not null and traffic_class <> 'synthetic_monitor' within 168 hours of created_at, fell six points for sign-up cohorts after a Tuesday. A client SDK major version shipped that morning. From fct_api_request (account_id, api_key_id, sdk_name, sdk_version, http_status, traffic_class, request_at) and dim_account (account_id, created_at, is_internal), decide whether activation actually fell or the metric's inputs changed, and state in advance what evidence would convince you of each.

Approach

1. Decompose the definition and recompute activation under each relaxation: status only, status plus traffic_class, then the full definition. If the entire drop lives in the api_key_id clause, this is an instrumentation question rather than a behavioural one.
2. Measure the null rate of api_key_id by sdk_version and by hour. A stamping change appears as a step at the deploy boundary confined to the new version; a behavioural change appears as a ramp that grows with adoption and leaves old-version traffic untouched.
3. Hold the cohort's SDK mix fixed before comparing. New sign-ups adopt the newest version first, so a cohort-level drop can be pure composition even when no individual version changed at all.
4. Corroborate with a source the release did not touch: whether the same cohorts appear in fct_usage_daily with non-zero billable_quantity, and whether their fct_support_ticket rows with category = 'onboarding' rose.
5. Write the decision rule down before looking at the answer. An instrumentation artefact predicts unchanged downstream usage and a version-confined null step; a real regression predicts falling downstream usage and more onboarding tickets in the same cohorts.

Follow-up

• Old-version and new-version populations are not exchangeable, because new accounts adopt the new version first. How would you build a comparison that is not confounded by cohort age?
• What backfill or metric-versioning policy keeps the historical series interpretable once you fix the stamping?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

• Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
• For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
• Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

• Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
• Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
• Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

• Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
• Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
• Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

• Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
• For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
• Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

• Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
• Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
• Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

• Repeat the depth protocol on the second-ranked area with the same six-problem structure.
• Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
• Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

• Re-run the six prompts from day one under the same clock and compare both correctness and time.
• Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
• Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Describe a time when you had to convince a product manager to change direction based on data they initially disagreed with.

Medium · behavioural and stakeholder questions

Approach

1. Name the disagreement or constraint, and how you resolved it with evidence.
2. Pick a story where you drove the decision, not one where you observed it.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• What would you do differently if you ran that project again?
• What did you decide not to do, and why?

Announce a metric fix that cuts the headline number

Medium · metric definitions · stakeholders · communication

Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.

Approach

1. The interviewer is probing whether you can land a correction as an operational change with a plan attached, rather than as an announcement other people then have to clean up after.
2. Quantify each exclusion separately before telling anyone: internal accounts, synthetic monitors, load tests. Three known quantities are a discussion; one alarming total is an argument.
3. Be precise about which side of the metric each exclusion touches, because one team's goal is on a count and the other's is on a ratio. The traffic-class filters remove requests, so they shrink the numerator only. Dropping internal accounts removes them from the ratio's denominator as well, since internal accounts carry ordinary account_status values and therefore sit in that denominator. Internal accounts are active in almost every week while the real base is not, so the numerator loses a larger share than the denominator and the ratio falls by less than the count does. Compute both and say which one the 11 percent is before anybody assumes.
4. Check whether the trend changes, not only the level. A constant 11 percent shift is a rebasing and nothing more. A shift that widens over time means the reported growth was partly internal or synthetic, which makes the existing goals unachievable as written and changes what you are asking teams to do.
5. Sequence the disclosure: the metric owner and the two teams whose goals move first and privately, then the board channel with a written bridge, then the dashboard. The dashboard is last because a number that changes without explanation is read as instability rather than as a fix.
6. Run both series for one reporting period with the bridge visible, restate history rather than letting the series break at a date, and set the date the old series is removed.
7. Propose the goal treatment yourself: rebase each target by the shift measured on the metric that target is written against, rather than leaving each team to negotiate individually, which is where corrections of this kind usually die.

Follow-up

• One team's quarterly goal is now unreachable. Rebase the target or let it miss, and what does each choice teach the organisation?
• How would this have been caught when the metric was first defined?
• What else on that dashboard shares this failure mode, and how would you find out this week?

State the measured impact of your own work honestly

Hard · impact · causal inference · self-assessment

You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.

Approach

1. The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
2. Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
3. Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
4. Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
5. State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
6. Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.

Follow-up

• Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
• What would have had to be true for the naive covered versus uncovered comparison to be valid?
• If the holdout costs the team real renewals, how do you justify asking for it, and to whom?