Bristlecone Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at Bristlecone, you serve as the technical engine behind complex supply chain and business analytics solutions. You are not just building models; you are solving mission-critical problems for global industry leaders like Nestle, Exxon Mobil, and Unilever. Your work directly influences how these organizations optimize their supply chains, demand forecasting, and root cause analysis platforms.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

• Define numerator, denominator and window precisely
• Decompose a metric move by segment and mix
• Size an experiment before anyone launches it

Overview

This role requires a balance of high-level statistical rigor and practical, production-grade engineering. You will operate in a dynamic environment where you must translate raw, messy tabular data into actionable business insights. Because Bristlecone prides itself on being a top-tier system integrator, you will often find yourself working at the intersection of business strategy and technical implementation, making your ability to communicate complex findings to non-technical stakeholders just as vital as your Python proficiency.

The Bristlecone interview process

Bristlecone candidates report 7 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Group Discussions

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Technical Interviews

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Case Studies

A case has a fixed clock, and a good deal of what is being scored is how you spend it. Thirty to forty-five minutes buys one pass across the whole problem or a deep read of one part of it, and choosing between those is the work rather than a compromise forced on you. Announce the shape early: the structure you are using, the branch you think carries the decision, and what you are setting aside. An answer that is thorough for the first third and silent on the recommendation reads worse than one that is rougher throughout and lands.

What to demonstrate

• Whether a visible structure appears in the opening minutes and survives the rest of the case
• Whether the depth goes to the branch that carries the decision, rather than the branch you find most comfortable
• Whether you say what you are leaving out and why, instead of quietly omitting it and hoping nobody asks

How to prepare

• After each practice case, write down the branches you chose not to open and the reason for each, then check whether you said any of them out loud while the case was running. A branch you only cut privately reads to the interviewer as one you missed.
• Redo a case you have already worked in half the time, deciding in advance which single branch you keep, then compare which version a listener would find more useful.
• Write a two-sentence opening you can reuse, holding the restated question and your plan for the available time, and deliver it within the first ninety seconds of every practice run.

Coding Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Discussions with Leads

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

• Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
• Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
• How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

• Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
• Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
• Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Where candidates lose points

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Watching an experiment daily and stopping when it crosses significance

A fixed-sample test controls type I error at one pre-declared look. Checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate to roughly 0.15 to 0.20 for ten looks, and it rises further with more frequent checks, because the p-value takes a random walk that will eventually dip below the threshold under the null. The usual defences are a fixed horizon declared before launch, group-sequential boundaries such as O'Brien-Fleming that spend alpha across a planned number of looks, or always-valid confidence sequences that are correct under continuous monitoring. Compounding it, the effect size reported conditional on having crossed the threshold is biased away from zero, and the bias is larger the lower the power was, so an underpowered test that 'won' typically overstates the lift it found.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Questions & practice

What statistical tests do you use for outlier detection and correlation analysis?

Medium · statistics and probability

Approach

1. Write down the assumption the method needs before you use the method.
2. Say what the estimate is of, and over what population it generalises.
3. Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

• Which assumption here is most likely to be violated in practice?
• How would you explain this result to someone who does not know statistics?

Explain how you would detect and handle quantile crossing violations in a probabilistic forecasting model.

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Set a baseline first, so any model has something honest to beat.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

1. Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
2. Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
3. For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
4. Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
5. Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
6. Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

1. Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
2. Set ref = df['received_at_utc'].max() and build the five masks against it.
3. Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
4. Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
5. Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

• Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
• Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
• Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
• Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

• The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
• Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
• Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

How would you optimize a complex SQL query for faster data extraction from a large supply chain database?

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Can you explain the difference between various join types in SQL and when to use each in a production environment?

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. State the window function and its partition and ordering out loud before writing it.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Why do you want to join Bristlecone specifically, and how do you see your skills contributing to our supply chain analytics focus?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

1. Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
2. Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
3. Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
4. Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
5. Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

1. Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
2. Write the active-account spine and the 28-day activity CTE.
3. Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
4. Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
5. Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

• Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
• The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
• No account_id appears twice in the output.
• Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

• How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
• Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
• Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

These questions assess your problem-solving process and your ability to align with company goals.

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Fix the population and the time window before naming any metric.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

Expect to demonstrate your ability to write optimized, clean, and production-ready code.

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Fix the population and the time window before naming any metric.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

Walk me through a challenging project where you had to work in a cross-functional team.

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Fix the population and the time window before naming any metric.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

How do you ensure experiment reproducibility in a team setting?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• What would you do if you could not randomise at all?

A completion rate the owning team can move without fixing anything

Medium · gaming · instrumentation · funnel

A team's target is core-flow completion rate: distinct fct_event.flow_instance_id with a 'flow_completed' event within 30 minutes of its 'flow_started' and no 'error_shown' carrying the same flow_instance_id in between, over distinct flow_instance_id with a 'flow_started' in the window, split by surface and app_version. The same team owns the client that emits those events and the tracking plan that defines them. List the ways this rate rises without any user completing more flows, then redefine the metric and its guardrails so those routes are closed. Deliverable: the hardened definition.

Approach

1. Work the emission side first, because that is what the team controls: delay minting flow_started until after the first screen so the highest-dropping attempts leave the denominator; stop emitting or rename error_shown; mint a fresh flow_instance_id on each retry so one failed attempt becomes several attempts whose last one completes; move flow_completed earlier in the flow.
2. Sort those moves by where they are visible. None of them shows in the rate itself; three of them show only in volume, which is why the denominator has to be published on the same chart as the rate.
3. Re-anchor the numerator on something outside the flow's own instrumentation: require a downstream is_core_action = TRUE event for the same user_id within 30 minutes of flow_completed, so a completion only counts when it produced the thing the flow exists to produce.
4. Add the guardrail that catches the retry route specifically: mean and p90 flow_instance_id per user per day, with the rule written down that a rising completion rate alongside rising attempts per user is a regression and not a win.
5. Make definition changes visible instead of forbidden: stamp a tracking-plan version on the series and re-base the history when event semantics or the 30-minute window change, rather than splicing two definitions into one line.

Worked solution (25 min)

1. Enumerate at least four emission-side moves and mark for each whether it shows in the rate or only in volume.
2. Pin the denominator: publish flow_started per 1,000 sessions and per distinct user beside the rate.
3. Rewrite the numerator to require a downstream core action for the same user within 30 minutes.
4. Add the attempts-per-user guardrail with its interpretation rule stated as a sentence.
5. Version the definition and write the re-base policy for the historical series.

Expected result: A hardened definition in which each enumerated gaming route is either blocked by the numerator, exposed by a published volume series, or forced to surface as a definition version change.

Verify it yourself

• Replay one untouched historical week under the old and hardened definitions. A large gap means the hardening changed the meaning of the metric, not only the incentives around it.
• flow_started per 1,000 sessions should be roughly continuous across an app_version boundary unless the flow's entry point genuinely moved; a step change there is an instrumentation change.
• Attempts per user and completion rate should not rise together within one build.

Follow-up

• How do you distinguish a genuine instrumentation fix from gaming, given both look like a step change confined to one release?
• The jump appears in exactly one app_version. Does that exonerate the team or implicate it?
• What do you do with eighteen months of history once the definition is hardened?

A conversion rate that fell in one regulatory region

Medium · consent · identity-keys · denominator · geography

Visit-to-signup conversion fell 1.3 points over six weeks. Signups cut by dim_user.country_code put the fall in one regulatory region where a consent banner shipped in week one, but absolute signups from that region are flat. fct_session carries consent_state, visitor_id, is_bot_flagged and session_date and no country column, so the denominator cannot be cut the same way. Using fct_session and fct_event, decide whether behaviour changed or the denominator did, state what these tables cannot settle, and name the one column that would settle it.

Approach

1. Name the asymmetry before computing anything. The numerator is user-keyed and therefore cuttable by country; the denominator is visitor-keyed and is not. Dividing a region-filtered numerator by an unfiltered denominator produces a quantity that is not a rate, and presenting it as a regional conversion rate is the first mistake available here.
2. Attack the denominator on the dimension you do have. Compute distinct visitor_id per week and sessions per distinct visitor_id per week: a consent banner that blocks or shortens the identity cookie raises the distinct-visitor count and lowers sessions per visitor, which depresses any visitor-keyed rate with no behaviour behind it.
3. Split on consent_state. Sessions with consent_state = 'denied' can enter the denominator but can never be joined forward to a signup, so a rising denied share mechanically drives the pooled rate down by roughly its own share. Report the granted-only rate and the denied share as two separate numbers rather than one blended figure.
4. Cross-check with measures that do not depend on the visitor key at all: absolute weekly signups, which are given as flat, and signups per session rather than per visitor.
5. State the limit honestly. Without country on the session or on its entry event, the regional attribution rests on the numerator alone, and the correct request is that one column, not a more elaborate model on top of the data you have.

Follow-up

• If granted-only conversion is the metric going forward, what selection bias have you accepted, and in which direction does it point?
• How would you handle the six weeks of already-published history once the new definition is adopted?
• What is the smallest instrumentation change that restores a cuttable denominator without collecting more personal data than before?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Describe a time you had to explain a complex ML model to a non-technical stakeholder.

Medium · behavioural and stakeholder questions

Approach

1. Quantify the outcome, including what you would not claim credit for.
2. Pick a story where you drove the decision, not one where you observed it.
3. State the situation in two sentences and spend the rest on your reasoning.

Follow-up

• How did you know the outcome was caused by your change?
• What would you do differently if you ran that project again?

Quantify your own impact without claiming the topline you touched

Hard · self-assessment · attribution · communication

You are writing the impact section of your own review. Over the year you ran four experiments, one of which shipped and three of which were flat; you corrected the definition of gross monthly revenue churn so that cancellation is recognised at period_end_utc; and you built a self-serve funnel dashboard. Weekly active accounts rose 14% over the same period. Your reviewer knows the data well. Write the three impact claims you would defend, stating for each what you contributed, what evidence supports it, and what portion of the outcome you are not claiming.

Approach

1. Recognise what is being probed: whether you apply to your own work the causal standard you would apply to somebody else's roadmap claim. Nearly everyone who would reject 'accounts that do Y retain better' will write 'I drove a 14% increase' without noticing it is the same error with a friendlier subject.
2. Sort the work by the kind of evidence it can carry. The shipped experiment is the only item with a randomised estimate, so it is the only one where an effect size is defensible, and you claim the interval rather than the point estimate.
3. Claim the three flat experiments as decisions prevented and price them. Features not built, or built differently, on evidence, with the engineering weeks reallocated as the number somebody else can verify. A defensible null is a delivered decision and should be written as one.
4. Claim the definition fix as correctness, not as improvement. The old figure was overstated by a specific percentage and appeared in a specific set of recurring documents; the impact is the change it produced in the forecast built on top of it, not a change in churn itself.
5. Claim the dashboard on usage and displacement: distinct weekly users of it, and the ad-hoc request count for six months before against six months after. If the request log does not exist, record the claim as unverified rather than estimating it upward.
6. Disclaim the 14% explicitly and once. State that it cannot be separated from seasonality, other teams' launches and a pricing change, and bound your own contribution from above using the shipped experiment's interval converted into headline units.

Follow-up

• Your shipped experiment's interval was +0.2pp to +1.4pp on activation. How much of the 14% can that account for, and how do you say so without undercutting yourself?
• A peer in the same cycle claims the full 14%. What, if anything, do you do about it?
• If you could only keep two of your three claims, which do you drop, and why that one?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

1. Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
2. Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
3. Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
4. Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
5. Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

• The executive says to give the midpoint and they will manage the risk. What do you do?
• How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
• Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?