IntegriChain Data Scientist Interview Questions & Guide 2026

The 60-second brief

The role of a Data Scientist at IntegriChain is pivotal in transforming vast amounts of healthcare data into actionable insights that drive strategic decisions. As a Data Scientist, you will leverage advanced analytical techniques to uncover patterns and trends that inform product development, optimize supply chain operations, and enhance customer engagement. The insights generated from your work will directly impact the effectiveness of IntegriChain’s solutions, ultimately improving patient outcomes and operational efficiency for healthcare providers.

When randomisation is off the table, the skill being checked is naming an identification strategy together with the assumption it rests on: parallel trends for difference-in-differences, relevance and exclusion for an instrument, overlap and conditional ignorability for matching. Say the assumption out loud and say how you would try to break it.

Focus areas

• Define numerator, denominator and window precisely
• Pick a randomisation unit that respects interference
• Separate novelty effects from durable behaviour change

Overview

This role is not just about crunching numbers; it encompasses a rich blend of statistical analysis, machine learning, and domain expertise in the healthcare sector. You will collaborate with cross-functional teams to tackle complex problems such as demand forecasting and resource optimization. Your contributions will be critical in ensuring that IntegriChain remains at the forefront of innovation in healthcare analytics, making this position both challenging and rewarding.

The IntegriChain interview process

IntegriChain candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

• Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
• Whether your reason for wanting the role points at the work itself rather than the company's reputation
• Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

• Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
• Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
• Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Hiring Manager Call

Expect a live problem with pieces of it missing, closer to a conversation than an exam. A metric moved, or somebody wants to know whether a change worked, and you are asked how you would find out. The manager is watching the first ninety seconds, specifically whether you establish what decision hangs on the answer before you start proposing methods. Candidates who open with a technique get steered back. Once the decision is clear, describe what the data would look like if the story were true, and say what you would accept as evidence that it is not.

What to demonstrate

• Whether you fix the decision the analysis serves before choosing an approach
• How you continue when you are told the data you just asked for does not exist
• Whether you state what would change your mind, not only what would confirm the hypothesis you started with
• How you size an effect before you have measured it

How to prepare

• Take a metric you know well and practise explaining in under two minutes the four things that could have moved it and how you would separate them
• Pick a recent launch or experiment and write the single number you would ask for first, plus what you would conclude if it came back flat
• Practise being interrupted: have someone remove a data source halfway through your answer and carry on without restarting

Onsite Interview

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

• Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
• Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
• Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

• Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
• Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
• Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
• Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Problem-Solving Assignment

A take-home is graded as an argument, not as a notebook. Somebody reads the submission without you in the room, so every choice has to survive on the page: why the question was framed this way, and what was deliberately left out. The gap between a strong and a weak submission is almost never model quality. It is whether the writeup names the specific question it answers and commits to a recommendation, including what evidence would overturn it. A high-accuracy model attached to no conclusion reads as effort that stopped before the decision.

What to demonstrate

• Whether the question you answered is stated outright, and whether it is the question the prompt posed rather than an easier neighbour of it
• Whether the recommendation is specific enough to act on, with the uncertainty attached to it instead of parked in a caveats section at the end
• Whether analytical choices such as the metric definition, the population filter and the time window are justified in the prose, not merely visible in code

How to prepare

• Take a dataset you have already worked with, write the one-paragraph conclusion first, then check whether the analysis you were planning actually supports it and cut whatever does not
• Practise stating a metric in one sentence that fixes the population, the time window and the denominator, then confirm your query computes exactly that sentence and nothing adjacent to it
• Hand a draft to someone outside the problem and ask them to tell you back what you recommended and why; anything they cannot recover is not on the page yet

Final Onsite Interview

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

• Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
• Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
• Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
• Whether you ask what decision is actually on the table before explaining anything

How to prepare

• Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
• For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
• Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Reading a pooled rate that moved because the mix moved, not because any behaviour changed

A pooled conversion rate is a weighted average, and a shift in the weights can move it in the opposite direction to every one of its parts. A paid campaign that brings low-converting traffic drops overall signup conversion even if desktop, mobile web and app conversion each rose that week, which is Simpson's paradox and it is the single most common cause of an inexplicable dashboard move. The discipline is to decompose before explaining: recompute the rate holding last period's segment weights fixed, and compare that counterfactual to the actual, so the mix effect and the rate effect are separated numerically rather than argued about. Segment on the dimensions that actually reweight, which in this domain are almost always device_type, referrer_channel, country and new versus returning.

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Solving silently instead of narrating the reasoning

Say which branch you are taking and why you chose it over the alternative, for example checking the denominator first because it changes what the comparison means. A correct answer that arrives with no visible path scores below a rigorous one that needed a hint.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Questions & practice

Describe a machine learning project you have worked on. What were the challenges and outcomes?

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Set a baseline first, so any model has something honest to beat.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Sessionise an event stream with gap and midnight rules

Hard · sessionisation · vectorisation · event streams

Sessionise a raw event stream. Input: a DataFrame with visitor_id, user_id (often NULL), occurred_at_utc and event_name, unsorted, up to 5 million rows. A session breaks when the gap from that visitor's previous event exceeds 30 minutes, and is force-closed at UTC midnight so no session spans two calendar dates. A gap of exactly 30 minutes does not break. Emit one row per session with session_id, visitor_id, the user_id as of the last event in the session, started_at_utc, ended_at_utc, session_date, duration_seconds and event_count. Vectorise; do not loop per visitor.

Approach

1. Sort by ['visitor_id', 'occurred_at_utc', 'event_id'] once, then express the whole problem as one boolean vector: a row starts a new session when the visitor changed, or the gap exceeds 30 minutes, or the UTC date differs from the previous row's UTC date. Cumsum that vector and you have the session key.
2. Get the comparison direction right on the gap: the rule is strictly greater than 1800 seconds, so an event at exactly 1800 seconds continues the session. Write it as gap > pd.Timedelta(minutes=30), and make the tie a test case rather than an assumption.
3. Derive the midnight break from the date change, not from inserting synthetic boundary rows. A date change implies a break even when the gap is two seconds, which is precisely the force-close rule and is why the two conditions are ORed rather than one subsuming the other.
4. Aggregate with a single groupby on the session key: min and max of occurred_at_utc, size for event_count, and last for user_id, which is correct because the frame is already sorted so 'last' is the final event in the session. That is the identity-as-of-session-end rule.
5. Compute duration_seconds as (max - min).dt.total_seconds(), which makes a single-event session 0 seconds. Say so explicitly, because a downstream mean session duration is sensitive to whether single-event sessions are 0 or excluded.

Follow-up

• Sessions are used as the denominator of a conversion rate. How does moving the inactivity gap from 30 to 45 minutes move that rate, and in which direction?
• A visitor's clock is 40 minutes ahead, so their events arrive with future occurred_at values. What does your sessioniser do, and what would you rather it did?
• The same person signs up mid-session on mobile and continues on desktop. How many sessions and how many users does your output show, and is that the right answer?

Cluster bootstrap for a per-session rate randomised on users

Hard · cluster bootstrap · randomisation unit · variance

An experiment randomised on user_id reports a per-session conversion rate, so sessions inside a user are correlated. Input: one row per session with user_id, variant in {control, treatment} and converted in {0,1}. Write a cluster bootstrap from scratch: resample users with replacement within each arm, keep every session of a drawn user, recompute each arm's ratio of converted sessions to sessions, and take the difference. Return the point estimate, a 95 percent percentile interval from at least 2,000 resamples, the naive session-level interval that ignores clustering, and the ratio of their widths.

Approach

1. Name the estimand precisely: it is a ratio of sums, sum(converted) over sum(sessions) within an arm, not the mean of per-user rates. Those differ whenever session counts vary across users, and the ratio is what the reported metric is.
2. Resample the cluster, not the row. Draw n_users user ids with replacement inside each arm and take every session belonging to each draw, including duplicate draws of the same user. Keeping the user count fixed per arm rather than the session count is what preserves the sampling design.
3. Precompute per-user (converted_sum, session_count) once, so each resample is two vector lookups and a division rather than a repeated filter over the session frame. That turns 2,000 resamples from minutes into under a second.
4. Take the 2.5th and 97.5th percentiles of the 2,000 differences for the interval, and report the point estimate from the full data rather than from the bootstrap mean, since the bootstrap mean carries the resampling bias.
5. Compute the naive interval from the session-level binomial standard error and compare widths. The expected inflation is roughly sqrt(1 + (m-1)*rho), with m the mean sessions per user and rho the intraclass correlation of converted within users, so a computed ratio far from that value points at a bug in one of the two intervals.

Worked solution (35 min)

1. per_user = df.groupby(['variant','user_id'])['converted'].agg(['sum','size']); split into two arrays per arm.
2. point = (t_sum.sum() / t_n.sum()) - (c_sum.sum() / c_n.sum()).
3. For b in range(B): idx = rng.integers(0, len(t_sum), len(t_sum)); ratio_t = t_sum[idx].sum() / t_n[idx].sum(); same for control; store the difference. Vectorise by drawing a (B, n) index matrix if memory allows.
4. ci = np.percentile(diffs, [2.5, 97.5]); naive_se = sqrt(p_t*(1-p_t)/n_sessions_t + p_c*(1-p_c)/n_sessions_c); naive_ci = point +/- 1.96*naive_se.
5. width_ratio = (ci[1]-ci[0]) / (naive_ci[1]-naive_ci[0]).

Expected result: A point estimate identical to the full-data ratio difference, a percentile interval containing that point estimate, and a width ratio greater than 1 whose value approximates sqrt(1 + (m-1)*rho) for the data's mean cluster size m and intraclass correlation rho. A ratio of approximately 1.0 means sessions were resampled instead of users.

Verify it yourself

• Construct data where every user has exactly one session: the cluster and naive intervals then agree to within Monte Carlo error, since m = 1 makes the design effect 1.
• Construct data where converted is constant within each user: rho is 1, and the width ratio should approach sqrt(m).
• The bootstrap distribution's mean sits close to the point estimate; a large offset means the resampled arms have drifted in size.
• Doubling B narrows the Monte Carlo noise in the percentile endpoints without shifting the interval's centre.

Follow-up

• Users average 3.4 sessions and the intraclass correlation is 0.12. What width ratio do you predict before running it, and does your bootstrap land there?
• Give the delta-method standard error for this ratio and say when you would prefer it to the bootstrap.
• Half the users in the treatment arm have exactly one session. What does that do to the cluster bootstrap's coverage, and how would you check it?

Given a dataset, how would you visualize the relationships between different variables?

Medium · sql and data manipulation

Approach

1. Say which table is the grain you start from, and join outward from it.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Explain how you would optimize a slow-running SQL query.

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Write a Python function to perform data cleaning on a given DataFrame.

Medium · sql and data manipulation

Approach

1. Say which table is the grain you start from, and join outward from it.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Read an experiment from first exposure, not assignment

Hard · experiments · window-functions · srm

fct_experiment_exposure holds experiment_id, unit_type, unit_id, variant, user_id, assigned_at_utc, first_exposed_at_utc, is_in_analysis_population and planned_end_utc. fct_event holds user_id, occurred_at_utc, is_core_action, and carries events up to a known data cut, :data_cut_utc. For one experiment randomised on unit_type = 'user', return per variant: exposed units, units with at least one core action in the seven days after that unit's own first exposure, the rate, and the variant share of exposed units. Only units whose seven-day window has fully elapsed as of the data cut belong in the readout. Units appearing under more than one variant are excluded from both arms and counted separately.

Approach

1. Run the contamination pass as an aggregate, not a window: SELECT unit_id FROM fct_experiment_exposure WHERE experiment_id = :exp GROUP BY unit_id HAVING COUNT(DISTINCT variant) > 1, then anti-join it away. PostgreSQL rejects COUNT(DISTINCT variant) OVER (PARTITION BY unit_id) outright, since DISTINCT is not implemented for window functions; if you want the test inline, MIN(variant) OVER (PARTITION BY unit_id) <> MAX(variant) OVER (PARTITION BY unit_id) is the equivalent that does run.
2. Do not resolve contamination by keeping the earliest variant. A unit that saw both arms carries treatment from both, so assigning it to either one biases that arm.
3. Define the population as is_in_analysis_population = TRUE AND unit_type = 'user' AND first_exposed_at_utc < planned_end_utc AND first_exposed_at_utc + interval '7 days' <= :data_cut_utc. The horizon filter is what makes the readout reproducible next week instead of drifting with every re-run; the data-cut filter is the one that actually buys seven days of follow-up, since a unit exposed an hour before the horizon otherwise contributes an hour of observation to a seven-day rate.
4. Measure the outcome on a per-unit relative window: LEFT JOIN fct_event on user_id with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc, first_exposed_at_utc + interval '7 days'). LEFT JOIN so units with no outcome stay in the denominator at zero rather than being deleted by an inner join.
5. Check the sample ratio before reading the effect: variant share of exposed units against the intended split, tested as a binomial. Run it on the truncated population as well as on the full exposed set, because if one arm exposes later on average the data-cut filter removes more of that arm and can manufacture a ratio mismatch the randomisation did not have. A mismatch on the full set means the exposure data is not a valid randomisation and invalidates the readout rather than being a footnote under it.
6. Report the per-variant rate, the absolute difference, and the fact that the variance unit is unit_id. That is straightforward here only because the grain is already one row per user; a per-session outcome under user randomisation would need a delta-method or bootstrap standard error instead.

Worked solution (35 min)

1. Count units per variant and units with more than one variant; report the contaminated count before anything else.
2. Build the exposed population CTE with both the horizon filter and the seven-days-before-data-cut filter, assert one row per unit_id, and record per variant how many units the second filter removed.
3. LEFT JOIN the bounded outcome events and flag conversion with a boolean aggregate.
4. Aggregate per variant to exposed, converted, rate and share, and run the binomial check on the share, both before and after the truncation.
5. Re-run with a fixed calendar window from experiment_start_utc and record the difference as the size of the dilution you avoided.

Expected result: One row per variant with exposed, converted, rate and share, plus a scalar count of excluded contaminated units. Every retained unit was first exposed at least seven days before the data cut, so each one has a full seven days of observation and the rates are directly comparable.

Verify it yourself

• Exposed counts across variants sum to the population count minus the contaminated units.
• MAX(first_exposed_at_utc) among retained units is at least seven days before :data_cut_utc.
• Variant shares sum to 1.
• No unit_id contributes to two variant rows.
• The truncation drops a similar share of each arm; if it does not, note the imbalance alongside the SRM result rather than reading the effect.
• Swapping the relative window for a calendar one changes the rates; if it does not, exposure timing is balanced and you can say so.

Follow-up

• Some units were assigned days before they were exposed. What does analysing the assigned set instead do to the estimated effect, and in which direction?
• The split is 51/49 on 400,000 exposed units. Do you read the result?
• The treatment arm exposes on average two days later than control. What does that do to a fixed calendar outcome window, and which arm does it favour?

Given a dataset with various features, how would you determine the most important predictors for a healthcare outcome?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Fix the population and the time window before naming any metric.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Describe your approach to solving a complex business problem using data.

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

If tasked with improving a specific metric, what steps would you take to analyze the situation?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How would you design an experiment to test the efficacy of a new healthcare product?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Decide the analysis before seeing data, including how long it runs and when you look.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

What statistical methods do you use for data analysis?

Medium

Approach

1. Say what you would check first and why it is the highest-information step.
2. State your assumptions explicitly before working the problem.
3. Work from the decision backwards to the evidence you would need.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Measure a paid channel without a user-level experiment

Hard · difference-in-differences · geo holdout · incrementality · clustered inference

Finance needs the incremental contribution of paid social before next year's budget. It carries 18% of last-touch-attributed signups in dim_user.last_touch_channel, but the ad platform will not randomise users and unexposed users are unobservable. The ad platform targets at market level, your warehouse resolves a market label onto fct_session, and 40 markets are eligible with 12 months of weekly signups derivable from dim_user. Design the read, size it on total signups rather than attributed ones, and state the identifying assumption and the check that would falsify it.

Approach

1. Say why the attributed number cannot answer the question. Last-touch assigns full credit to whichever touch was recorded nearest the conversion, which systematically rewards channels that intercept demand that already existed, and the counterfactual of not running the channel is nowhere in the touchpoint data.
2. Choose a geo holdout: randomise markets into a paused arm and a spend-as-usual arm, and measure total signups per market. The metric has to be the total, because the question is whether demand is created or merely relabelled, and attributed signups fall to zero in a paused market by construction.
3. Size on markets, since the market is the randomisation unit and therefore the variance unit. Forty markets give at most twenty per arm, so form matched pairs on twelve months of pre-period weekly signups and randomise within pair, which removes the between-market level differences that would otherwise dominate the residual variance.
4. Fix the identifying assumption in plain words: absent the pause, treated and control markets' signup trends would have moved in parallel. Falsify it by running the same estimator on placebo cut dates inside the pre-period; a non-zero placebo estimate or visibly diverging pre-trends kills the design as specified.
5. Estimate with market and week fixed effects and cluster standard errors on market. Below about forty clusters the cluster-robust error is biased downward, so use a wild cluster bootstrap or randomisation inference over the pair assignment. If only one or two markets can be paused, switch to synthetic control: weight untreated markets to match the treated market's pre-period path and use in-space placebos for inference.

Worked solution (40 min)

1. Aggregate weekly signups per market for twelve months from dim_user with is_internal = FALSE, and drop markets whose history is too short or too volatile to pair.
2. Rank markets on pre-period mean and trend, form twenty matched pairs, and randomise one of each pair into the paused arm.
3. Derive the MDE from the between-pair residual standard deviation of weekly signups, not the within-market one, because power is set by the number of pairs; express it as a percentage of total signups and compare it with the effect finance cares about.
4. Run placebo estimates at three pre-period cut dates before committing; re-pair or abandon if any is significant.
5. After the pause, estimate difference-in-differences with market and week fixed effects, report a wild cluster bootstrap interval, and convert the point estimate into incremental cost per signup against the spend withheld.

Expected result: A matched-pair geo holdout across twenty pairs, read on total signups with market and week fixed effects and market-clustered inference, yielding an incrementality ratio of incremental signups to attribution-credited signups. The identifying assumption is parallel counterfactual trends, checked with pre-period placebo cuts. The MDE is set by twenty pairs rather than by millions of sessions, and is usually far coarser than the requester expects.

Verify it yourself

• Pre-period placebo estimates should centre on zero with intervals covering it. A systematically non-zero placebo means the pairing failed and the main estimate inherits that bias.
• Treated plus control market signups in the pre-period must reconcile with the total the finance dashboard reports, otherwise the market label is silently dropping sessions.
• An incrementality ratio above 1 implies the channel drives conversions it never touched. That is possible through view-through effects the model never sees, but it demands a stated mechanism rather than a shrug.

Follow-up

• Campaign commitments mean the pause starts in different weeks in different markets. What does staggered timing do to a two-way fixed effects estimate when effects vary across cohorts?
• How long do you pause, given that paid social has a delayed conversion tail, and how do you keep that tail out of the pre-period?
• The estimate says the channel is 40% incremental. What happens to the last-touch number on the dashboard after that?

Tell a pipeline outage from a collapse in usage

Medium · pipeline-failure · instrumentation · event-volume

Weekly active accounts completing a core action fell 9%, and almost the entire fall sits in accounts whose events carry surface = 'ios'. App crash rates and store reviews are unchanged. You have fct_event with occurred_at_utc, received_at_utc, event_name, is_core_action, app_version and surface, plus fct_session and the ingestion job run log. Establish within the hour whether iOS engagement fell or iOS events stopped arriving, name the evidence that distinguishes them, and say what you would publish on the dashboard in the meantime.

Approach

1. Compare the event-name composition inside surface = 'ios' against the prior four weeks as shares, not counts. A behaviour collapse scales most event names together; a dropped event definition or a broken downstream filter hits specific event_name values while page_view and session-opening events hold steady. That shape difference is the fastest discriminator available.
2. Profile the received_at_utc minus occurred_at_utc distribution per day for surface = 'ios'. A stalled-then-backfilling pipeline shows a fat upper tail and a recovering p99; a silently dropped stream shows an unchanged lag distribution over a smaller volume. The two failure modes have different remedies and different histories.
3. Cut by app_version. A logging SDK change arrives with one build and ramps with its adoption curve; an infrastructure fault arrives across every build within the same hour. Checking this costs one group-by and rules out half the hypothesis space.
4. Cross-check against a signal that does not travel the suspect path: server-emitted events with session_id NULL, and subscription or billing activity for the same accounts. If those accounts are still transacting, the users did not leave.
5. Publish an ex-iOS total with an explicit annotated break rather than a blended total. A blended number during a known ingestion gap is wrong in a direction you can already name, and republishing it daily spreads the artefact into every downstream report.

Follow-up

• Suppose the events do eventually backfill. What is your policy for restating the published weekly numbers, and who needs to be told?
• What monitor would have caught this before a human noticed the weekly metric, and what would it alert on?
• If is_core_action is maintained in the tracking plan, what governance would stop a change to that list from silently moving a north-star metric?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Can you explain a time when you derived insights from data that led to a significant business decision?

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Name the disagreement or constraint, and how you resolved it with evidence.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• What did you decide not to do, and why?
• How did you know the outcome was caused by your change?

Walk through an analysis you got wrong and what changed

Easy · postmortem · data quality · self-assessment

Describe an analysis of yours that turned out to be wrong after somebody had already acted on it. You have four minutes. The account must name the defect mechanically, the join, the filter, the window or the identity key, rather than describing it as a communication problem. It must also say who did what because of the wrong number, how the error surfaced, how long it stood, and what control you put in place so that class of error cannot reach a decision again. Do not pick an error nobody acted on.

Approach

1. Recognise what is being probed: whether you can be specific about your own failure without minimising it or performing contrition. The discriminator is whether the defect has a mechanism the listener could reproduce in their own warehouse.
2. Choose the case by blast radius rather than by comfort. An error nobody acted on tests nothing, and picking one signals that you are managing the interview instead of answering it.
3. Structure the account in six beats: the number, the decision it drove, the defect, the detection, the correction, the control. Keep the defect to one reproducible sentence, for example an inner join to fct_subscription_period that dropped accounts with no subscription row and so computed retention over payers only.
4. State the direction of the bias, not only its existence. A filter or join that removes rows usually moves a metric predictably, and knowing which way shows you diagnosed the mechanism rather than patched the symptom.
5. Be exact about detection and elapsed time. 'A colleague noticed' and 'the row-count assertion failed before publication' are different answers about the same organisation, and the second one is the one your control is supposed to produce next time.
6. End on the control, its cost, whether it has fired since, and one thing it does not cover.

Follow-up

• What did the control cost, and has it fired since? If it never has, how do you know it works?
• How long did the wrong number stand before anyone questioned it, and what does that say about the review path it went through?
• What is the equivalent mistake you are most likely to make in this role, given the tables you would be working in?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

1. Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
2. Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
3. Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
4. Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
5. Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

• The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
• Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
• Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?