Gallup Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at Gallup plays a critical role in bridging the gap between human behavioral science and modern product analytics. Known globally for its public polling, workplace consulting, and the CliftonStrengths assessment, Gallup relies heavily on data to help organizations solve their most complex people problems. As a member of the data science team, you will not just build predictive models; you will design the quantitative frameworks that power Gallup Access, the company’s flagship SaaS platform used by millions of managers and employees worldwide.

Ask early whether the loop includes an asynchronous take-home or a timed live case, because the two are graded on different things. A take-home is read as an artifact: the question you decided to answer, what you did about missing or malformed records, and a conclusion stated plainly enough for someone to act on. A reviewer who cannot rerun your notebook discounts the result whatever score is printed in it. Hold to the stated time box and write down what you would have done with more of it, since the follow-up round is usually a live defence of the same work.

Focus areas

• Separate novelty effects from durable behaviour change
• Decompose a metric move by segment and mix
• Pick a randomisation unit that respects interference

Overview

Your work will directly influence how organizations measure employee engagement, predict retention, and optimize team performance. This requires a unique blend of traditional product analytics, psychometric evaluation, and survey methodology. You will collaborate closely with product managers, software engineers, and behavioral psychologists to transform vast amounts of behavioral and attitudinal data into scalable, actionable product features.

This role is highly strategic, requiring you to think deeply about how "success" is defined and measured. Whether you are optimizing user journeys on Gallup's digital platforms or developing algorithms to analyze global opinion data, your contributions will have a direct, visible impact on how global leaders make decisions. It is an inspiring space for data scientists who are passionate about human behavior, organizational psychology, and large-scale digital products.

The Gallup interview process

Gallup candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Application

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Assessment

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

• Whether you can state the other side's argument accurately before you explain why you disagreed
• What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
• Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

• Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
• For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
• Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Structured Phone Interview

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

• Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
• The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
• Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

• Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
• Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
• Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Technical Interviews

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Where candidates lose points

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Watching an experiment daily and stopping when it crosses significance

A fixed-sample test controls type I error at one pre-declared look. Checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate to roughly 0.15 to 0.20 for ten looks, and it rises further with more frequent checks, because the p-value takes a random walk that will eventually dip below the threshold under the null. The usual defences are a fixed horizon declared before launch, group-sequential boundaries such as O'Brien-Fleming that spend alpha across a planned number of looks, or always-valid confidence sequences that are correct under continuous monitoring. Compounding it, the effect size reported conditional on having crossed the threshold is biased away from zero, and the bias is larger the lower the power was, so an underpowered test that 'won' typically overstates the lift it found.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

How do you identify and mitigate non-response bias in large-scale organizational surveys?

Medium · statistics and probability

Approach

1. Translate the result into the decision it informs, in one plain sentence.
2. Write down the assumption the method needs before you use the method.
3. Say what the estimate is of, and over what population it generalises.

Follow-up

• What sample size would you need to detect an effect half this size?
• How would you explain this result to someone who does not know statistics?

How would you handle missing or incomplete data in a dataset containing longitudinal employee feedback?

Medium · statistics and probability

Approach

1. Translate the result into the decision it informs, in one plain sentence.
2. Sanity-check the answer against a simple bound or a simulated case.
3. Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

• Which assumption here is most likely to be violated in practice?
• How would you explain this result to someone who does not know statistics?

Describe your approach to validating a predictive model when the underlying data is highly subjective.

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Say how the offline result would be validated online before it is trusted.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• What would you monitor after launch to know the model is still valid?
• How would you choose the decision threshold, and who owns that choice?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

1. Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
2. Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
3. For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
4. Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
5. Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
6. Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

1. Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
2. Set ref = df['received_at_utc'].max() and build the five masks against it.
3. Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
4. Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
5. Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

• Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
• Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
• Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
• Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

• The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
• Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
• Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Pair flow starts to completions within a thirty-minute bound

Medium · event-streams · conditional-aggregation · flow-funnel

fct_event holds event_id, flow_id, flow_instance_id, event_name, occurred_at_utc, surface, app_version. Using only rows where event_name is 'flow_started', 'flow_completed' or 'error_shown', compute the daily core-flow completion rate by surface and app_version. Numerator: distinct flow_instance_id whose flow_completed occurs within 30 minutes of its flow_started with no error_shown for the same instance in between. Denominator: distinct flow_instance_id with a flow_started that day. Report rows with a NULL flow_instance_id as a separate coverage count, not inside either side.

Approach

1. Collapse the stream to one row per flow_instance_id using conditional aggregation: MIN(occurred_at_utc) FILTER (WHERE event_name = 'flow_started') AS started, MIN(...) FILTER (WHERE event_name = 'flow_completed') AS completed, MIN(...) FILTER (WHERE event_name = 'error_shown') AS first_error. This makes the 'first' semantics explicit and avoids a self-join entirely.
2. Apply the predicates on that single row: completed IS NOT NULL AND completed <= started + interval '30 minutes' AND (first_error IS NULL OR first_error > completed). Note the last clause encodes 'in between' literally, so an error surfaced after a successful completion does not disqualify the attempt.
3. Bucket by started::date, surface and app_version. Splitting by app_version is not decoration: a completion-rate regression is nearly always confined to one client build, and a pooled daily rate hides it behind the installed base.
4. Count NULL flow_instance_id rows as a separate coverage figure. They cannot be attributed to an attempt at all, so they belong in neither numerator nor denominator, and their share tells you how much of the rate is unmeasurable.
5. For any roll-up to week or to all surfaces, re-sum the numerator and denominator rather than averaging the daily rates.

Follow-up

• occurred_at_utc is client-supplied. What do you do with an instance whose completion timestamp precedes its start?
• flow_instance_id is missing on 4 percent of starts for one app_version. What can and cannot you conclude about that build?
• An instance starts at 23:55 and completes at 00:04. Which day owns it, and what does the alternative do to the daily series?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

1. Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
2. Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
3. Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
4. Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
5. Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

1. Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
2. Write the active-account spine and the 28-day activity CTE.
3. Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
4. Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
5. Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

• Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
• The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
• No account_id appears twice in the output.
• Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

• How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
• Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
• Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

How would you design a metric to measure long-term user engagement on a platform like Gallup Access?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

If user retention suddenly dropped by 15% month-over-month, how would you investigate the root cause?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How does your work ethic compare to that of other professionals in your field?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Fix the population and the time window before naming any metric.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

What statistical techniques would you use to segment users based on their interaction patterns with our tools?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Explain how you would set up and analyze an A/B test for a new feature designed to increase survey completion rates.

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• How would you handle interference between treated and control units?
• What would you do if you could not randomise at all?

Explain the difference between statistical significance and practical significance in the context of behavioral research.

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• How would you handle interference between treated and control units?
• What would you do if you could not randomise at all?

Define success for a rebuilt first-run onboarding checklist

Easy · activation · metric tree · cohorts

A rebuilt first-run checklist ships to all new signups next month. You have dim_user (user_id, account_created_at_utc, signup_surface, is_internal) and fct_event (user_id, occurred_at_utc, is_core_action, event_name, surface). Propose a primary metric with an explicit numerator, denominator, window and publication lag, plus two guardrails and one diagnostic you would refuse to treat as success. The team wants a number it can read weekly, and the constraint is that everything must be computable from those two tables alone. Deliver the metric tree from the north star down to the metric you chose.

Approach

1. Fix the grain and the cohort key first: the checklist is seen by users, so count on user_id with account_created_at_utc as the cohort key, and say out loud that the north star is account-grained so the tree crosses grains here deliberately rather than by accident.
2. Take seven-day activation as the primary — core action on at least 2 distinct UTC dates inside [account_created_at_utc, +7 days) — because the two-distinct-days predicate cannot be satisfied by the single checklist-completion click the feature itself produces.
3. Demote checklist completion rate to a diagnostic and give the reason: it is an output of the feature, so it is near-perfectly correlated with having shipped the feature and cannot fall when the feature is bad.
4. Pick guardrails by the failure each one catches, not by what is easy to query: week-4 signup-cohort retention catches an activation gain that does not persist, and p50 minutes to first core action catches a checklist that adds steps to a path users already completed.
5. State the operational rules explicitly: is_internal = FALSE, an 8-day publication lag, and signup_surface values 'invite' and 'sso_provisioned' split out because those users arrive through an administrator rather than a self-serve signup and may not be shown the checklist at all.

Worked solution (20 min)

1. Write the primary in full: numerator = cohort users with is_core_action = TRUE events on at least 2 distinct UTC dates in [account_created_at_utc, +7 days); denominator = all dim_user rows in the cohort week with is_internal = FALSE.
2. Draw the tree downward: weekly active accounts completing a core action, then seven-day activation rate multiplied by weekly signups, then checklist step completion and time-to-first-core-action labelled as diagnostics.
3. Attach the timing rules: 8-day lag, and a cohort with fewer than seven full elapsed days published blank rather than partial.
4. Attach each guardrail to its named failure mode in one line each, so the guardrail list reads as a list of specific risks rather than a list of metrics.
5. Write the exclusion rule and the surface split, and state what you would do if 'invite' users turn out to be a third of the cohort.

Expected result: A primary metric stated as numerator, denominator, window, lag and exclusions; two guardrails each tied to a named failure mode; and checklist completion explicitly demoted to a diagnostic with the reason given.

Verify it yourself

• The denominator includes cohort users who never returned. If it is built from users with any event, the metric is conditioned on survival and will read several points too high.
• Recomputing a cohort's activation rate three days after first publication must return the same number. If it moves, the right-hand window bound or the late-arrival rule is missing.
• The per-surface cohort counts must sum to the unsplit cohort count, so confirm signup_surface IN ('invite','sso_provisioned') is broken out rather than dropped. Note the limit while you are there: neither dim_user nor fct_event carries a pre-signup identifier, so whether those users were ever shown the checklist is not answerable from these two tables — the surface split is the only handling available, and the metric definition has to say that rather than imply exposure was verified.

Follow-up

• Activation rises three points but week-4 retention is flat. What do you tell the team, and what would you need to distinguish a real gain from pulled-forward activity?
• The checklist ships on web only. What changes in the denominator, and what breaks if you leave every surface in?
• Signup mix shifted toward 'invite' the same week. How would you show whether the activation move was mix or behaviour?

Gross revenue churn doubled with no cancellations behind it

Hard · billing-periods · fx · revenue-churn · decomposition

Gross monthly revenue churn computed from fct_subscription_period doubled from 1.8% to 3.6% in one month. The support queue shows no rise in cancellations and renewals look normal. You have fct_subscription_period with subscription_id, account_id, period_start_utc, period_end_utc, mrr_cents, mrr_cents_constant_fx, seats_billed, period_status, change_reason and canceled_at_utc, plus dim_account. Decompose the 1.8-point rise into named mechanisms, size each in points of the headline, and state the remainder you cannot explain.

Approach

1. Reconstruct the numerator row by row and group it by change_reason before arguing about causes. A mid-period plan or seat change closes the current period row and opens a new one, so any implementation that treats a closed row as lost revenue books upgrades, downgrades and seat changes as churn; the change_reason breakdown of the numerator makes that visible in one query.
2. Check the recognition timestamp. Churn belongs at period_end_utc, because revenue continues until the period ends, not at canceled_at_utc when the button was pressed. Then count period_end_utc rows per month across thirteen months: annual cohorts concentrate their period ends in the month twelve months after they were signed, so a spike that repeats in the same month last year is seasonality in the book, not an event this month.
3. Recompute the whole numerator and denominator on mrr_cents_constant_fx. If the constant-currency figure is materially flatter, the move is an exchange-rate translation and belongs nowhere in a churn narrative.
4. Check period_status handling. Rows with period_status = 'past_due' are dunning, not cancellation; a status-based rule counts them as loss while a period-end rule does not, and a dunning backlog can move the number by itself.
5. Express every mechanism in points of the headline, sum them, and print the residual explicitly next to the month-to-month standard deviation of the prior twelve months. A decomposition without a stated remainder is a story, not an accounting.

Follow-up

• Which of these mechanisms should be fixed in the metric definition and which should be reported as a genuine business fact?
• How would you present a month whose churn is dominated by an annual cohort anniversary without the audience concluding the business is deteriorating?
• What would you change so that an upgrade can never enter the churn numerator again, and how would you test that it worked?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

How do you manage situations where you disagree with a manager's decision or methodology?

Medium · behavioural and stakeholder questions

Approach

1. Pick a story where you drove the decision, not one where you observed it.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Close with what you would do differently, concretely.

Follow-up

• How did you know the outcome was caused by your change?
• What would you do differently if you ran that project again?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

1. Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
2. Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
3. Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
4. Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
5. Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

• The executive says to give the midpoint and they will manage the risk. What do you do?
• How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
• Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?

Quantify your own impact without claiming the topline you touched

Hard · self-assessment · attribution · communication

You are writing the impact section of your own review. Over the year you ran four experiments, one of which shipped and three of which were flat; you corrected the definition of gross monthly revenue churn so that cancellation is recognised at period_end_utc; and you built a self-serve funnel dashboard. Weekly active accounts rose 14% over the same period. Your reviewer knows the data well. Write the three impact claims you would defend, stating for each what you contributed, what evidence supports it, and what portion of the outcome you are not claiming.

Approach

1. Recognise what is being probed: whether you apply to your own work the causal standard you would apply to somebody else's roadmap claim. Nearly everyone who would reject 'accounts that do Y retain better' will write 'I drove a 14% increase' without noticing it is the same error with a friendlier subject.
2. Sort the work by the kind of evidence it can carry. The shipped experiment is the only item with a randomised estimate, so it is the only one where an effect size is defensible, and you claim the interval rather than the point estimate.
3. Claim the three flat experiments as decisions prevented and price them. Features not built, or built differently, on evidence, with the engineering weeks reallocated as the number somebody else can verify. A defensible null is a delivered decision and should be written as one.
4. Claim the definition fix as correctness, not as improvement. The old figure was overstated by a specific percentage and appeared in a specific set of recurring documents; the impact is the change it produced in the forecast built on top of it, not a change in churn itself.
5. Claim the dashboard on usage and displacement: distinct weekly users of it, and the ad-hoc request count for six months before against six months after. If the request log does not exist, record the claim as unverified rather than estimating it upward.
6. Disclaim the 14% explicitly and once. State that it cannot be separated from seasonality, other teams' launches and a pricing change, and bound your own contribution from above using the shipped experiment's interval converted into headline units.

Follow-up

• Your shipped experiment's interval was +0.2pp to +1.4pp on activation. How much of the 14% can that account for, and how do you say so without undercutting yourself?
• A peer in the same cycle claims the full 14%. What, if anything, do you do about it?
• If you could only keep two of your three claims, which do you drop, and why that one?