Interview guides · EPAM India

EPAM India Data Scientist Interview Questions & Guide 2026

Practice explaining analysis before a EPAM India data scientist interview: the recommendation, the uncertainty, the follow-ups.

The 60-second brief

As a Data Scientist at EPAM India, you serve as a pivotal bridge between complex data landscapes and actionable business strategy. EPAM India operates at the intersection of digital engineering and consulting, meaning your work directly influences the digital transformation journeys of global enterprise clients. You will not merely build models in isolation; you will design solutions that address real-world challenges in sectors such as finance, healthcare, and retail, often operating within cross-functional teams that include software engineers, product managers, and business stakeholders.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Size an experiment before anyone launches it
Pick a randomisation unit that respects interference
Define numerator, denominator and window precisely

Overview

The role demands a rare blend of technical rigor and product intuition. You will be responsible for the entire lifecycle of data-driven products—from initial problem framing and metric design to the deployment of scalable machine learning solutions. Success here requires the ability to communicate sophisticated statistical concepts to non-technical partners, ensuring that the insights you generate translate into measurable business growth. It is a high-impact position where your ability to diagnose performance drops and design robust experimentation frameworks is just as critical as your coding proficiency.

The EPAM India interview process

EPAM India candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Online Technical Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Deep-Dive Technical Rounds

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Business-Focused Interview

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Where candidates lose points

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Treating last-touch attribution as the causal value of a channel

The attribution label on dim_user is the output of a rule that assigns full credit to whichever touch happened to be recorded last inside a lookback window, and that rule systematically rewards channels that sit close to the conversion, especially branded search and retargeting, which largely intercept demand that already existed. Reallocating spend on those labels moves budget toward the channels that are best at being last, which is why attributed return on ad spend often improves while total signups do not. Nothing in the touchpoint data can settle this, because the counterfactual of not running the channel was never observed. The credible reads are a geo holdout or a scheduled pause, sized in advance on the total-signups metric rather than on the attributed one, and the honest framing in the meantime is that the label describes correlation with conversion and not incremental contribution.

Generalising beyond the population the sample actually supports

State the frame the sample was drawn from and where it diverges from the population you want to talk about: time window, platform, geography, opt-in. If a group is excluded from the frame, either weight to known margins or narrow the claim rather than quietly extending it.

Reaching for a model before the target metric exists

Before naming an algorithm, write down the label, the prediction time, and the action that changes when the score crosses a threshold. If you cannot say what decision the output drives, any modelling choice is guesswork dressed up as method.

Questions & practice

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Worked solution (25 min)

Aggregate to one row per (week, segment tuple) with summed visitors and signups, then split into w0 and w1 frames and align with an outer join, filling missing visitors and signups with 0.
Compute w_s = visitors / visitors.sum() within each week, and r_s = signups / visitors only where visitors > 0. Where a week's visitors are 0, set that week's r_s to that week's pooled rate, the stated convention; never leave it NaN.
rate_effect = (w0 * (r1 - r0)).sum(); mix_effect = ((w1 - w0) * r0).sum(); interaction = ((w1 - w0) * (r1 - r0)).sum().
contribution = w0*(r1-r0) + (w1-w0)r0 + (w1-w0)(r1-r0) per segment, which reduces to w1r1 - w0r0; sort by abs and return the head.
assert abs(rate + mix + interaction - (pooled1 - pooled0)) < 1e-12.

Expected result: Three scalars summing exactly to the observed pooled change, plus a segment table whose contribution column also sums to that change. In the stated case rate and mix carry opposite signs: several segments converted better while volume moved toward a weaker-converting one, which is what makes the pooled move look inexplicable until it is split.

Verify it yourself

On synthetic data where every segment appears in both weeks, every segment rate is held constant and only the weights move, rate_effect and interaction are exactly 0.
On synthetic data where every segment appears in both weeks, weights are held constant and only rates move, mix_effect and interaction are exactly 0.
The per-segment contribution column sums to the pooled change, matching the sum of the three scalars.
Swapping the two weeks negates the total, as it must, but not the parts. Assert the three identities that actually hold: the interaction term is exactly invariant, because both of its factors change sign; the swapped rate effect equals -(rate_effect + interaction); and the swapped mix effect equals -(mix_effect + interaction). They still sum to the negated pooled change. Expecting all three to flip sign is the common wrong assertion.
Plant a segment present only in week 1, holding 3 percent of week-1 visitors at a rate well below pooled. Under the documented pooled-rate convention its mix contribution is w_s1 * r_pooled0 and its interaction contribution is w_s1 * (r_s1 - r_pooled0), with a rate contribution of exactly 0; under a zero-fill convention its mix contribution is exactly 0 and the whole w_s1 * r_s1 lands in interaction. Assert whichever one the function documents, and fail if mix is asserted non-zero while the code zero-fills.

Follow-up

The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Cluster bootstrap for a per-session rate randomised on users

Hard · cluster bootstrap · randomisation unit · variance

An experiment randomised on user_id reports a per-session conversion rate, so sessions inside a user are correlated. Input: one row per session with user_id, variant in {control, treatment} and converted in {0,1}. Write a cluster bootstrap from scratch: resample users with replacement within each arm, keep every session of a drawn user, recompute each arm's ratio of converted sessions to sessions, and take the difference. Return the point estimate, a 95 percent percentile interval from at least 2,000 resamples, the naive session-level interval that ignores clustering, and the ratio of their widths.

Approach

Name the estimand precisely: it is a ratio of sums, sum(converted) over sum(sessions) within an arm, not the mean of per-user rates. Those differ whenever session counts vary across users, and the ratio is what the reported metric is.
Resample the cluster, not the row. Draw n_users user ids with replacement inside each arm and take every session belonging to each draw, including duplicate draws of the same user. Keeping the user count fixed per arm rather than the session count is what preserves the sampling design.
Precompute per-user (converted_sum, session_count) once, so each resample is two vector lookups and a division rather than a repeated filter over the session frame. That turns 2,000 resamples from minutes into under a second.
Take the 2.5th and 97.5th percentiles of the 2,000 differences for the interval, and report the point estimate from the full data rather than from the bootstrap mean, since the bootstrap mean carries the resampling bias.
Compute the naive interval from the session-level binomial standard error and compare widths. The expected inflation is roughly sqrt(1 + (m-1)*rho), with m the mean sessions per user and rho the intraclass correlation of converted within users, so a computed ratio far from that value points at a bug in one of the two intervals.

Follow-up

Users average 3.4 sessions and the intraclass correlation is 0.12. What width ratio do you predict before running it, and does your bootstrap land there?
Give the delta-method standard error for this ratio and say when you would prefer it to the bootstrap.
Half the users in the treatment arm have exactly one session. What does that do to the cluster bootstrap's coverage, and how would you check it?

Rebuild per-visitor ordering without groupby convenience methods

Easy · pandas · vectorisation · window logic

You have a DataFrame of 2 million fct_event rows with visitor_id, occurred_at_utc and event_id, unsorted and containing duplicate timestamps within a visitor. Produce three new columns: event_rank, the 1-based position of the event within its visitor ordered by occurred_at_utc; seconds_since_prev, the gap to that visitor's previous event, NULL for the first; and is_first_for_visitor. You may use sort_values, shift, cumsum, numpy and boolean masking. You may not use groupby.transform, groupby.apply, groupby.cumcount, groupby.rank or merge_asof. Break timestamp ties on event_id.

Approach

Sort once by ['visitor_id', 'occurred_at_utc', 'event_id'] and reset the index. The whole exercise reduces to row arithmetic on a sorted frame, and the tiebreak on event_id is what makes the result reproducible across runs.
Mark visitor boundaries with is_first = df['visitor_id'].ne(df['visitor_id'].shift()). This is the single fact every other column derives from.
Compute seconds_since_prev as the diff of the timestamp column, then overwrite it with NaT/NaN wherever is_first is True. The shift crosses the boundary between visitors and will otherwise hand the first row of each visitor the last event of the previous one.
Build event_rank from a running counter that resets at boundaries: take a global cumulative position (np.arange(len(df))) and subtract, per row, the global position at which that visitor started. Get the start position by forward-filling the positions where is_first is True, which is a cumsum-free reset and is O(n).
Verify against the forbidden method once, as a test rather than as the implementation, and confirm the two agree on every row.

Follow-up

The frame does not fit in memory. How does your approach change if you can only process one visitor-partitioned chunk at a time?
occurred_at_utc is client-supplied and sometimes runs backwards within a visitor. Does your seconds_since_prev go negative, and should it?
How would you extend this to reset the counter at every change of surface as well as visitor?

Expect to demonstrate your proficiency in querying large, complex data…

Medium · sql and data manipulation

Expect to demonstrate your proficiency in querying large, complex datasets efficiently. Write a query using SQL window functions to calculate a rolling average of sales over the last 30 days. How do you handle missing or null values when joining large datasets from disparate sources? Explain how you would optimize a slow-running query on a massive database table.

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
Write the active-account spine and the 28-day activity CTE.
Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
No account_id appears twice in the output.
Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

This category focuses on your ability to design valid experiments and …

Medium · a/b testing and experimentation

This category focuses on your ability to design valid experiments and identify potential biases. Describe the process of setting up an A/B test for a new UI change; how do you determine the sample size? What are the most common experimentation pitfalls that lead to false positives? How do you ensure statistical significance when your traffic volume is low?

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

Diagnose a 50.5 to 49.5 split before reading the result

Easy · sample ratio mismatch · exposure logging · chi-square

An experiment randomised on visitor_id ran for 11 days. fct_experiment_exposure holds 104,320 control rows and 102,180 treatment rows for the experiment_id, all with is_in_analysis_population = TRUE, against a configured 50/50 split. The treatment arm shows a 3.1% relative lift in visit-to-signup conversion at p = 0.004, and the team wants to ship. Decide whether that lift can be reported, show the arithmetic behind your decision, and name the three checks you would run first on the exposure logging.

Approach

Test the split instead of eyeballing it. Chi-square goodness of fit against 103,250 expected per arm gives 2 x 1070^2 / 103,250 = 22.2 on 1 degree of freedom, p about 2.5e-6. The conventional alarm threshold is p < 0.001, so this is a sample ratio mismatch.
Treat the mismatch as invalidating, not as a caveat. The arms are no longer exchangeable, so the 3.1% lift has no causal reading and rebalancing on covariates afterwards does not restore it.
Walk the logging path in the order units flow through it: assignment service, then the exposure event fired on the treated surface, then the filters that build the analysis population. Each stage can drop one arm preferentially.
Name the three suspects this stack actually has. Exposure emitted by client code the treatment arm reaches later or not at all when its bundle errors; an extra redirect or render in one arm that loses slow connections; and a post-assignment filter (is_bot_flagged, is_internal, consent_state) that correlates with variant.
Check whether the imbalance is uniform by splitting counts on session_date, surface, app_version and device_type. A mismatch confined to one client build or to day one points at a deployment fault that can be fixed and re-run, not at a design fault.

Worked solution (15 min)

N = 206,500, expected 103,250 per arm, observed deviation 1,070, which is 0.518% of the total pushed into one arm.
Chi-square = 2 x 1070^2 / 103,250 = 22.18 on 1 df, p about 2.5e-6.
Compare with the p < 0.001 threshold and stop: the readout is void.
Re-pull the same counts grouped by session_date and app_version to localise where the rows were lost.
Deliver one line: the split is off by 1,070 visitors, a roughly one-in-400,000 event under correct randomisation, so the experiment is being fixed and re-run rather than read.

Expected result: Chi-square(1) = 22.2, p about 2.5e-6, far past the p < 0.001 mismatch threshold. The correct deliverable is an exposure-logging investigation and a re-run, not a caveated 3.1% lift.

Verify it yourself

Cross-check the statistic as a z on one arm count: under a fair split that count has SD sqrt(N/4) = sqrt(51,625) = 227 visitors, so the 1,070 deviation is 4.71 SD, and 4.71^2 = 22.2 reproduces the chi-square.
Chi-square scales linearly with N at a fixed ratio, so the same 50.518% split at N = 2,000 returns 22.18 / 103.25 = 0.21, p = 0.64. That reconciles the arithmetic and nothing more; it is not evidence that the alarm is independent of N. What rules out the large N is that the p-value is uniform under correct randomisation at every sample size, so p = 2.5e-6 is a one-in-400,000 draw whether N is 2,000 or 2,000,000. Large N buys the power to resolve a 0.5% skew; it does not manufacture one.
The mismatch p-value should be orders of magnitude smaller than the lift's p-value; if they are comparable, recheck the arithmetic.
Arm counts must sum to the total exposure rows for that experiment_id. If they do not, the analysis-population filter is already dropping rows before you looked.

Follow-up

The imbalance is entirely in one app_version on android. Can you salvage the other surfaces, and under what argument?
How would you monitor this across every running experiment without drowning in false alarms from a daily chi-square at alpha 0.05?
The missing rows turn out to be disproportionately visitors with consent_state = 'unknown'. What does that do to the estimate even after the logging bug is fixed?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Walk through an analysis you got wrong and what changed

Easy · postmortem · data quality · self-assessment

Describe an analysis of yours that turned out to be wrong after somebody had already acted on it. You have four minutes. The account must name the defect mechanically, the join, the filter, the window or the identity key, rather than describing it as a communication problem. It must also say who did what because of the wrong number, how the error surfaced, how long it stood, and what control you put in place so that class of error cannot reach a decision again. Do not pick an error nobody acted on.

Approach

Recognise what is being probed: whether you can be specific about your own failure without minimising it or performing contrition. The discriminator is whether the defect has a mechanism the listener could reproduce in their own warehouse.
Choose the case by blast radius rather than by comfort. An error nobody acted on tests nothing, and picking one signals that you are managing the interview instead of answering it.
Structure the account in six beats: the number, the decision it drove, the defect, the detection, the correction, the control. Keep the defect to one reproducible sentence, for example an inner join to fct_subscription_period that dropped accounts with no subscription row and so computed retention over payers only.
State the direction of the bias, not only its existence. A filter or join that removes rows usually moves a metric predictably, and knowing which way shows you diagnosed the mechanism rather than patched the symptom.
Be exact about detection and elapsed time. 'A colleague noticed' and 'the row-count assertion failed before publication' are different answers about the same organisation, and the second one is the one your control is supposed to produce next time.
End on the control, its cost, whether it has fired since, and one thing it does not cover.

Follow-up

What did the control cost, and has it fired since? If it never has, how do you know it works?
How long did the wrong number stand before anyone questioned it, and what does that say about the review path it went through?
What is the equivalent mistake you are most likely to make in this role, given the tables you would be working in?

Disagree with a product manager's roadmap claim using data

Medium · causal inference · selection bias · stakeholder

A product manager proposes building a feature on the argument that accounts connecting an integration in week one retain three times better at week four. The figure is correctly computed from dim_user and fct_event, and it has already been shown to leadership. You have one scheduled 1:1 before the roadmap locks. Deliver the specific analysis you would run to test whether the relationship is causal, the result that would change your own mind, and how you open the conversation so that the PM is not put in the position of defending the number in public.

Approach

Recognise what is being probed: whether you can separate a number being right from an inference being wrong, and do it without costing the PM face. The generic answer recites that correlation is not causation; the strong one names the specific confound and proposes the cheapest design that could distinguish the explanations.
State the alternative concretely. Accounts that connect an integration in week one are accounts that already have a workflow and a technical owner, so week-one intent plausibly drives both the connection and week-four retention. The selection is on intent, which no amount of post-hoc adjustment observes.
Order the discriminating analyses by cost. First, condition on pre-connection activity by comparing retention within strata of week-one core-action count, which removes the crude version of the confound but not unobserved intent. Second, look for variation in integration availability that was unrelated to intent, such as a staggered release or an outage window. Third, an encouragement design that randomises a prompt to connect and reads the intent-to-treat effect on week-four retention, which is the only version that identifies an effect.
Run the timing check, because it is nearly free and it is the most persuasive single piece of evidence. If the retention advantage among connectors is already visible before any of them connected, the causal story is largely finished.
Pre-commit to what would change your mind and say it before you show anything: if the gap survives stratification and the encouragement arm moves week-four retention at all, the feature has a case and you will say so.
Open the 1:1 by agreeing with the true part, that the correlation is real and worth chasing, then ask what effect size the roadmap plan assumes. That makes the size of the claim the topic instead of its authorship.

Follow-up

The encouragement test needs six weeks and the roadmap locks in two. What do you recommend in the interim?
Stratifying on week-one activity closes half the gap. What do you conclude, and what do you still not know?
How would you word this in the roadmap document so the PM's original number is reframed rather than deleted?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

The executive says to give the midpoint and they will manage the risk. What do you do?
How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?