Brandwatch Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at Brandwatch plays a pivotal role in transforming massive streams of social intelligence and consumer data into actionable insights for global brands. As a leader in digital consumer intelligence, Brandwatch relies on its data science team to build the models, metrics, and analytical frameworks that power its core products. You will not just be running queries; you will be helping the business understand consumer trends, sentiment, and market shifts at an unprecedented scale.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

• Turn a vague request into a measurable question
• Pick a randomisation unit that respects interference
• Size an experiment before anyone launches it

Overview

The work is inherently product-focused and cross-functional. You will collaborate closely with product managers, engineers, and UX designers to define how data is surfaced to users and how internal product performance is measured. Whether you are improving the accuracy of sentiment analysis or diagnosing why a specific user metric has shifted, your contributions directly influence the strategic direction of the Brandwatch platform.

Success in this role requires a blend of rigorous technical skill and a pragmatic "product-sense." You must be able to translate complex statistical concepts into clear, business-driven recommendations. Because the company values curiosity and collaboration, you will find yourself in an environment where your ability to communicate your findings to non-technical stakeholders is just as important as your ability to write complex code.

The Brandwatch interview process

Brandwatch candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

• Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
• Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
• The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

• Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
• Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
• Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Discussions

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Final Panel Interviews

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

• Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
• Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
• Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
• Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

• Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
• Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
• Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Interpreting a change before checking data quality and logging

Spend the first pass on row volume by day, null rates, duplicate keys, and whether the step change lands on a release or tracking-migration date. A discontinuity that coincides with a deploy is an instrumentation hypothesis before it is a behavioural one.

Questions & practice

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

1. Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
2. Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
3. For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
4. Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
5. Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
6. Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Follow-up

• The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
• Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
• Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Simulate the false positive cost of repeated peeking

Medium · simulation · peeking · type i error

Quantify the cost of peeking. Simulate a two-arm experiment with no true effect: each arm accumulates Bernoulli conversions at a base rate of 0.10 up to 40,000 units per arm. Run a two-sided two-proportion z-test at alpha 0.05 at ten equally spaced interim points, and record whether the test ever crossed. Report the false positive rate over at least 10,000 replications, alongside the rate for a single look at the final sample only. Use a fixed seed and report a Monte Carlo standard error on both figures.

Approach

1. Generate each replication as two cumulative sums of Bernoulli draws, then read the interim points off the cumulative arrays. Regenerating data at each look would make the looks independent, which destroys exactly the dependence the exercise is about: later looks share data with earlier ones.
2. Use the pooled-variance two-proportion z: p_pool = (x1+x2)/(n1+n2), z = (p1-p2) / sqrt(p_pool*(1-p_pool)*(1/n1 + 1/n2)), reject when |z| > 1.96. State that the normal approximation is fine here because the smallest look has roughly 400 expected conversions per arm.
3. Vectorise across replications rather than looping: draw a (reps, n) array of uniforms, threshold at 0.10, cumsum along axis 1 and slice the ten look indices. A per-replication loop at 10,000 by 40,000 is unnecessarily slow.
4. Record the any-cross indicator per replication, take the mean, and compute the Monte Carlo standard error as sqrt(p*(1-p)/reps) so the reported figure comes with its own precision.
5. Report the single-look rate in the same run as a control. If it does not land near 0.05, the bug is in the test statistic and not in the peeking argument.

Follow-up

• Re-run with 40 looks instead of 10. Why does the curve flatten rather than continue rising linearly?
• Among the replications that crossed, what is the mean observed lift, and why is it not zero?
• What does an O'Brien-Fleming boundary or an always-valid confidence sequence change about this simulation, and what does each cost in power?

Implement seven-day activation from its written definition

Medium · metric definition · cohorts · pandas · joins

Implement the seven-day activation rate. Inputs: dim_user with user_id, account_created_at_utc and is_internal; fct_event with user_id, occurred_at_utc and is_core_action. A user activates when core-action events carrying a non-NULL user_id fall on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). The denominator is every non-internal user whose account_created_at_utc lands in the cohort week, including users with no events at all. Return one row per cohort week with numerator, denominator and rate, publishing only weeks whose last signup is at least eight days old.

Approach

1. Build the denominator first, from dim_user alone, filtered on is_internal = False. Deriving it from the join is the standard way to lose every user who never fired an event, which is exactly the population the metric is about.
2. Join events to users on user_id with a left join from the user side, then apply the window as a half-open interval: occurred_at >= created AND occurred_at < created + 7 days. The right bound is exclusive, so an event at exactly created + 7 days does not count.
3. Count distinct UTC dates per user, not distinct events. Floor occurred_at_utc to date before the nunique, and do it in UTC rather than local time so the threshold does not move with the user's country.
4. Apply the >= 2 threshold, aggregate to cohort week, and compute the rate by re-summing numerator and denominator per week rather than averaging any per-user or per-day rate. Fix the week anchor explicitly: cohort_week is the Monday of the signup week in UTC, which is what Postgres DATE_TRUNC('week') returns and what any SQL version of this metric will produce. In pandas, subtract dt.weekday days from the floored timestamp. If you reach for periods instead, the anchor that matches is to_period('W') (equivalently 'W-SUN'), whose weeks end Sunday and therefore start Monday; to_period('W-MON') labels weeks that end on Monday, so it runs Tuesday through Monday and its start_time is a Tuesday. Mixing the two shifts every cohort label by one day and silently moves Mondays into the previous week.
5. Suppress immature weeks: drop any cohort week whose maximum account_created_at_utc is within 8 days of the data cut, and return them as absent rather than as a partial number.

Worked solution (30 min)

1. users = dim_user[~dim_user.is_internal].copy(); created = users['account_created_at_utc']; users['cohort_week'] = created.dt.floor('D') - pd.to_timedelta(created.dt.weekday, unit='D'), which is the Monday-start week. The period spelling that agrees with it is created.dt.to_period('W').dt.start_time; 'W-MON' does not agree and is off by a day.
2. ev = events[events.is_core_action & events.user_id.notna()]; merge onto users on user_id with how='inner' for the numerator side only.
3. Filter to the half-open window, add ev_date = occurred_at_utc.dt.date, group by user_id and count distinct dates, keep users with >= 2.
4. numer = users.merge(activated_user_ids, how='left', indicator=True) then group by cohort_week and sum the indicator; denom = users.groupby('cohort_week').size().
5. rate = numer / denom; drop weeks where users.groupby('cohort_week')['account_created_at_utc'].max() > data_max - 8 days.

Expected result: One row per mature cohort week with numerator, denominator and rate, where denominator equals the count of non-internal signups in that week regardless of activity, numerator is less than or equal to denominator, and the most recent 1 to 2 weeks are absent rather than reported low.

Verify it yourself

• For a synthetic user with two core actions 90 minutes apart on the same UTC date, activation is False.
• For a synthetic user with one core action on day 0 and one at exactly created + 7 days, activation is False; move the second to created + 6 days 23 hours and it becomes True.
• Pin the week anchor with two users created at Sunday 23:59 UTC and the following Monday 00:01 UTC: they must land in different cohort weeks, and the Sunday user's cohort_week must be the Monday six days before them. to_period('W-MON') puts both in one period, which is the off-by-one this check exists to catch.
• The denominator summed across all published weeks equals the count of non-internal users created in that date range.
• Re-summing numerator and denominator to a monthly figure gives a different answer from averaging the weekly rates; the re-summed one is the metric.

Follow-up

• The threshold is 2 distinct days. What changes in the reported history if someone moves it to 3, and how would you publish that change?
• Invited seats and SSO-provisioned users have no pre-signup session. Should they be in this denominator at all, and what does including them do to the rate for sales-assisted accounts?
• How would you produce the same metric at account grain, and which of the two would you put on the dashboard?

How do you use SQL window functions to calculate rolling averages or identify trends over time?

Medium · sql and data manipulation

Approach

1. Say which table is the grain you start from, and join outward from it.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Describe a scenario where you had to clean a messy dataset before performing analysis.

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Say which table is the grain you start from, and join outward from it.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

How would you optimize a slow-running SQL query that joins multiple large tables?

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. State the window function and its partition and ordering out loud before writing it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

Rebuild sessions from raw events with a thirty-minute gap

Medium · window-functions · sessionisation · gaps-and-islands

From fct_event (event_id, visitor_id, occurred_at_utc) alone, rebuild sessions: a new session begins when the gap from that visitor's previous event exceeds 30 minutes, and every session is force-closed at UTC midnight so none spans two calendar dates. Return one row per session with visitor_id, session_start, session_end, event_count and session_date. Do not read fct_session; the point is to reproduce it. Assume duplicate occurred_at_utc values exist for the same visitor.

Approach

1. Get the previous timestamp per visitor with LAG(occurred_at_utc) OVER (PARTITION BY visitor_id ORDER BY occurred_at_utc, event_id). The event_id tiebreaker is required, not stylistic: with duplicate timestamps an unstable ordering makes the boundary flags non-deterministic between runs.
2. Set a boundary flag when prev IS NULL, or occurred_at_utc - prev > interval '30 minutes', or occurred_at_utc::date <> prev::date. The third disjunct is the midnight rule, expressed as a date change rather than a clock comparison so it holds across any gap length.
3. Number the islands with SUM(flag::int) OVER (PARTITION BY visitor_id ORDER BY occurred_at_utc, event_id ROWS UNBOUNDED PRECEDING). Because event_id is unique the ordering is total, so no two rows are peers and RANGE UNBOUNDED PRECEDING would compute exactly the same numbers here. Write ROWS anyway: it is the half of the guard that survives someone later simplifying the ORDER BY back to occurred_at_utc alone, at which point the default RANGE frame gives every row sharing a timestamp one shared running total.
4. GROUP BY visitor_id and the island number, then MIN(occurred_at_utc) AS session_start, MAX(...) AS session_end, COUNT(*) AS event_count, session_start::date AS session_date.
5. Reconcile against fct_session on one sample day. The counts should agree except for server-emitted events carrying no client session, so a systematic difference beyond those is a bug in the gap rule or in the ordering.

Worked solution (30 min)

1. Pick one high-volume visitor and dump their ordered event timestamps for a day to trace by hand.
2. Add LAG with the tiebreaker and eyeball the computed gaps.
3. Add the three-part boundary flag and confirm the first event of each date is flagged.
4. Add the running SUM with an explicit ROWS frame, then group and aggregate.
5. Compare total event_count against the raw input count, and compare session counts to fct_session for the sample day.

Expected result: One row per (visitor_id, island). SUM(event_count) equals the input row count exactly, with no events lost or duplicated. No session_end falls on a different date from its session_start.

Verify it yourself

• Total events in equals total events out.
• MAX(session_end::date - session_start::date) is 0 across all rows.
• For a visitor ordered 09:00, 09:10, 09:55, 23:58, 00:02, the result is four sessions, not three.
• No two sessions for the same visitor have overlapping [session_start, session_end] ranges.
• Swapping ROWS for RANGE while event_id stays in the ORDER BY changes nothing; dropping event_id from both ORDER BY clauses does change the boundaries for a visitor with duplicate timestamps, which locates the guard in the tiebreaker rather than in the frame keyword.

Follow-up

• Why 30 minutes? What does a 5-minute rule do to sessions-per-visitor and to any per-session conversion rate?
• The same person uses phone then laptop. Two visitor_ids, two sessions. What breaks if you sessionise on user_id instead?
• The midnight rule splits an overnight session. Which metrics does that bias, and in which direction?

If you notice a sudden drop in a core engagement metric, what steps would you take to diagnose the root cause?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

How would you design a metric to measure the success of a new dashboard feature?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

How would you explain the trade-offs between two competing product metrics to a product manager?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How do you determine the sample size required for an A/B test?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the randomisation unit first; it decides the variance and what the test can detect.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

What are some common pitfalls you encounter when running experiments in a product environment?

Medium · a/b testing and experimentation

Approach

1. Decide the analysis before seeing data, including how long it runs and when you look.
2. Say whether units interfere with each other, and switch design if they do.
3. State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

Size an activation test before anyone writes code

Easy · power · minimum detectable effect · proportions

A team wants to test a new onboarding checklist against the seven-day activation rate: users with is_core_action = TRUE events on at least two distinct UTC dates inside the first seven days, counted from dim_user.account_created_at_utc with is_internal = FALSE. Baseline is 22%. Weekly non-internal signups are 9,000, split evenly between two arms. State the minimum detectable effect after two weeks of enrolment, the enrolment needed to detect a 1.5 percentage point absolute lift, and the earliest date a readout would be honest.

Approach

1. Use the two-proportion sizing shortcut n = 16 p(1-p) / d^2 per arm, where d is the absolute lift. The 16 is 2(1.96 + 0.84)^2 = 15.7 rounded, i.e. two-sided alpha 0.05, 80% power, equal arms.
2. Invert it for the MDE at fixed n: d = sqrt(16 p(1-p) / n). Two weeks gives 9,000 per arm, so d = sqrt(2.746 / 9000) = 0.0175, which is 1.75 percentage points absolute or 7.9% relative on a 22% base.
3. Run it forward for d = 0.015: n = 2.746 / 0.000225 = 12,202 per arm, 24,404 total, about 2.7 weeks of enrolment. Round up to three whole weeks so the enrolment window contains no partial week, since signups are not uniform across weekdays.
4. Add the metric's own lag rather than reporting the enrolment end date. Activation is unobservable until seven days after the last enrolled signup, and the definition publishes on an eight-day lag, so the readout lands 29 days after launch.
5. Close by testing whether the ask is realistic. If the team's honest prior is a 0.5pp lift, the requirement is about 110,000 per arm and roughly 24 weeks of enrolment, and saying so before launch is the deliverable.

Worked solution (20 min)

1. Compute p(1-p) = 0.22 x 0.78 = 0.1716 and 16 x 0.1716 = 2.746.
2. MDE at two weeks: n = 4,500 per arm per week x 2 = 9,000; d = sqrt(2.746 / 9000) = 0.0175.
3. Sample for d = 0.015: n = 2.746 / 0.000225 = 12,202 per arm, which is 2.71 weeks of enrolment, so enrol three whole weeks.
4. Readout date = 21 days of enrolment + 8 days of follow-up and publication lag = 29 days from launch.
5. State the boundary of what this quarter can answer: below roughly 0.7pp the requirement passes 56,000 per arm and more than a full quarter of enrolment.

Expected result: MDE after two weeks is 1.75 percentage points absolute, 7.9% relative. Detecting 1.5pp needs 12,202 users per arm, so three weeks of enrolment and a readout 29 days after launch. Detecting 0.5pp needs about 110,000 per arm, roughly 24 weeks, which is the answer the team needs to hear before the code is written.

Verify it yourself

• MDE must fall as 1 / sqrt(n). Doubling enrolment from two weeks to four should move 1.75pp to 1.24pp, not to 0.88pp.
• Substituting d = 0.0175 back into 16 p(1-p) / d^2 must return about 9,000 per arm.
• An exact two-proportion calculator at alpha 0.05 two-sided, power 0.80, p1 = 0.22, p2 = 0.235 returns about 12,260 per arm, within half a percent of the shortcut.

Follow-up

• The team also wants trial-to-paid read from the same test. What does a second primary metric do to your alpha, and what would you actually do about it?
• Signups are heavier on weekdays. What goes wrong if enrolment runs 17 days instead of 14?
• The activation definition uses a two-distinct-days threshold. If that were loosened to one day, what happens to the baseline rate and to the sample you need?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

1. Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
2. Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
3. Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
4. Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
5. Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

• How do you choose the fixed cohort age, and what do you lose by choosing it too long?
• If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
• The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

What was the biggest challenge you have faced in your data science career, and how did you manage it?

Medium · behavioural and stakeholder questions

Approach

1. Pick a story where you drove the decision, not one where you observed it.
2. Close with what you would do differently, concretely.
3. Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

• What did you decide not to do, and why?
• How did you know the outcome was caused by your change?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

1. Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
2. Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
3. Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
4. Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
5. Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

• The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
• Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
• Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?

Handle a request for numbers supporting a decision already made

Hard · integrity · framing · stakeholder

A senior leader has already decided to sunset a plan tier and asks you for the analysis showing it is the right call. Accounts on that tier carry 6% of MRR at constant FX and have the highest licensed-seat utilisation in the book. The leader's support matters to your next review cycle, and the decision is being presented in four days. Deliver what you produce, what you decline to produce, and the exact sentence you will say in the meeting where the number appears on a slide.

Approach

1. Recognise what is being probed: whether you can find the legitimate request inside an illegitimate framing instead of either complying or refusing on principle. The generic answer promises to push back; the strong one produces something genuinely useful and states its limits in the room, without ambushing anybody.
2. Separate the decision from the justification. Sunsetting the tier may be correct for reasons the data does not hold, such as support cost, roadmap surface area or sales motion. What you decline is a one-sided document. What you produce is the case read both ways, which also happens to be more useful to the leader.
3. Build the symmetric analysis: MRR at risk at constant FX, the share of affected accounts with a plausible migration path given seats_licensed and billing_term, the recovery rate assumed for that migration and where it came from, and the downside case in which high-utilisation accounts treat the sunset as a reason to re-evaluate the vendor entirely.
4. Surface the inconvenient fact privately and early. The highest seat utilisation in the book is a retention signal, and the leader should hold it before the room does, so they can incorporate it rather than be caught by it.
5. Agree the meeting sentence in advance with the leader, so that nobody is surprised. Something to the effect that the tier is 6% of MRR and its accounts are the most heavily used in the book, and that the case for sunsetting rests on cost and focus rather than on revenue. That is true, it supports the decision on its real grounds, and it stops the deck claiming the numbers endorse it.
6. Decide your own line before you need it: what you will not put your name to, and that the route if asked anyway is your own manager rather than a confrontation in the meeting.

Follow-up

• The deck circulates with your analysis included and the downside case removed. What do you do, and by when?
• What changes if the honest analysis says the sunset is clearly the wrong call?
• How do you write the same memo when the leader is your skip-level and the meeting is tomorrow?