Interview guides · ConstructConnect

ConstructConnect Data Scientist Interview Questions & Guide 2026

Causal inference drills for ConstructConnect data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

As a Data Scientist at ConstructConnect, you will play a pivotal role in transforming raw data into actionable insights that drive business decisions and enhance product offerings. This role is vital to the company’s mission of providing accurate and timely information to the construction industry, impacting everything from project bids to market trends. You will be involved in developing predictive models, analyzing customer behavior, and working with large datasets to uncover patterns that can inform strategic initiatives.

Ask early whether the loop includes an asynchronous take-home or a timed live case, because the two are graded on different things. A take-home is read as an artifact: the question you decided to answer, what you did about missing or malformed records, and a conclusion stated plainly enough for someone to act on. A reviewer who cannot rerun your notebook discounts the result whatever score is printed in it. Hold to the stated time box and write down what you would have done with more of it, since the follow-up round is usually a live defence of the same work.

Focus areas

Define numerator, denominator and window precisely
Separate novelty effects from durable behaviour change
Size an experiment before anyone launches it

Overview

The complexity and scale of data at ConstructConnect offers an intellectually stimulating environment where you can apply advanced analytics and machine learning techniques. You will collaborate with cross-functional teams, including product managers, engineers, and business analysts, to understand the challenges faced by users and translate these into data-driven solutions. Expect to contribute to delivering value not only to the company but also to the users who rely on your insights for critical decision-making.

The ConstructConnect interview process

ConstructConnect candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interviews

Rounds of this kind usually include one question about work that did not go well, and it is the part that carries the most information. Anyone can narrate a shipped win. What the interviewer learns from a project that stalled is how you behave without a result to hide behind: whether you noticed the problem yourself, how long it took, and who you told. Answers that route the failure onto a data pipeline or a reorganisation close the topic without answering it, and the follow-up comes back to your own part.

What to demonstrate

Whether you found the error yourself or someone else found it, and how long it sat before anyone knew
What you changed afterwards, stated as a check you now run rather than a lesson you now believe
Whether the mistake you choose has real cost attached, such as a quarter of misdirected roadmap or a metric that was reported upward, instead of one that flatters you

How to prepare

Choose a failure you caught yourself and be ready to say what tipped you off. A story where someone else caught it is still usable, but you will be asked why you missed it.
Write down the check you added afterwards and where it lives now, so the correction is a concrete artefact rather than a resolution.
Rehearse saying the cost out loud. Candidates shrink the number by instinct once the interviewer is in the room.

Interviews with Hiring Managers

Much of this round runs on your own history, but the manager is not collecting a project list. They are working out what it is like when something goes wrong on your watch: how late the bad news tends to arrive, and whether a number you hand over has been checked by anyone including you. That is why the strongest material is a project where you can describe the part that did not work and what it cost. A result you cannot take full responsibility for, however clean, gives them nothing to trust you with afterwards.

What to demonstrate

Whether you volunteer the limits of a result you are proud of, or wait to be pushed onto them
How errors surfaced in your past work, and whether you or somebody else found them
Whether the scope you claim matches the level of detail you can still produce about it
What you did the first time a stakeholder acted on something of yours that turned out to be wrong

How to prepare

Rebuild one headline figure from memory down to the join and the filter, so a question about the denominator does not stall the conversation
For each project you raise, write the sentence you would say to someone who had already acted on a number that later turned out wrong
Mark which parts of a project were yours and which belonged to other people, and state that boundary yourself before anyone asks

Interviews with Team Members

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Where candidates lose points

Treating last-touch attribution as the causal value of a channel

The attribution label on dim_user is the output of a rule that assigns full credit to whichever touch happened to be recorded last inside a lookback window, and that rule systematically rewards channels that sit close to the conversion, especially branded search and retargeting, which largely intercept demand that already existed. Reallocating spend on those labels moves budget toward the channels that are best at being last, which is why attributed return on ad spend often improves while total signups do not. Nothing in the touchpoint data can settle this, because the counterfactual of not running the channel was never observed. The credible reads are a geo holdout or a scheduled pause, sized in advance on the total-signups metric rather than on the attributed one, and the honest framing in the meantime is that the label describes correlation with conversion and not incremental contribution.

Slicing a flat experiment until a segment reaches significance

Testing one metric across twenty segments at a nominal 5% level produces a significant result about two thirds of the time when nothing is happening anywhere, and the segment that surfaces is by construction the one with the most favourable noise. The reported effect in that slice is then badly overstated, because selection on significance conditions the estimate on being large. What makes it dangerous rather than merely wrong is that a post-hoc segment always has a plausible story attached, so it survives the meeting. The controls are declaring the small number of segments of interest before launch, correcting across the ones tested, and treating anything discovered afterwards as a hypothesis that needs its own adequately-powered test rather than a finding.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Naming a model class before naming the deployment constraints

Set out the latency budget, the label delay, the retraining cadence, the interpretability requirement and the number of labelled examples, then pick the model that fits them. A boosted-tree answer to a problem where each decision must be explained to the affected user is a well-executed answer to the wrong question.

Questions & practice

Write a function to calculate the correlation coefficient between two variables.

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Write down the assumption the method needs before you use the method.
Sanity-check the answer against a simple bound or a simulated case.

Follow-up

How would you explain this result to someone who does not know statistics?
Which assumption here is most likely to be violated in practice?

How would you implement a decision tree from scratch?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Cluster bootstrap for a per-session rate randomised on users

Hard · cluster bootstrap · randomisation unit · variance

An experiment randomised on user_id reports a per-session conversion rate, so sessions inside a user are correlated. Input: one row per session with user_id, variant in {control, treatment} and converted in {0,1}. Write a cluster bootstrap from scratch: resample users with replacement within each arm, keep every session of a drawn user, recompute each arm's ratio of converted sessions to sessions, and take the difference. Return the point estimate, a 95 percent percentile interval from at least 2,000 resamples, the naive session-level interval that ignores clustering, and the ratio of their widths.

Approach

Name the estimand precisely: it is a ratio of sums, sum(converted) over sum(sessions) within an arm, not the mean of per-user rates. Those differ whenever session counts vary across users, and the ratio is what the reported metric is.
Resample the cluster, not the row. Draw n_users user ids with replacement inside each arm and take every session belonging to each draw, including duplicate draws of the same user. Keeping the user count fixed per arm rather than the session count is what preserves the sampling design.
Precompute per-user (converted_sum, session_count) once, so each resample is two vector lookups and a division rather than a repeated filter over the session frame. That turns 2,000 resamples from minutes into under a second.
Take the 2.5th and 97.5th percentiles of the 2,000 differences for the interval, and report the point estimate from the full data rather than from the bootstrap mean, since the bootstrap mean carries the resampling bias.
Compute the naive interval from the session-level binomial standard error and compare widths. The expected inflation is roughly sqrt(1 + (m-1)*rho), with m the mean sessions per user and rho the intraclass correlation of converted within users, so a computed ratio far from that value points at a bug in one of the two intervals.

Worked solution (35 min)

per_user = df.groupby(['variant','user_id'])['converted'].agg(['sum','size']); split into two arrays per arm.
point = (t_sum.sum() / t_n.sum()) - (c_sum.sum() / c_n.sum()).
For b in range(B): idx = rng.integers(0, len(t_sum), len(t_sum)); ratio_t = t_sum[idx].sum() / t_n[idx].sum(); same for control; store the difference. Vectorise by drawing a (B, n) index matrix if memory allows.
ci = np.percentile(diffs, [2.5, 97.5]); naive_se = sqrt(p_t*(1-p_t)/n_sessions_t + p_c*(1-p_c)/n_sessions_c); naive_ci = point +/- 1.96*naive_se.
width_ratio = (ci[1]-ci[0]) / (naive_ci[1]-naive_ci[0]).

Expected result: A point estimate identical to the full-data ratio difference, a percentile interval containing that point estimate, and a width ratio greater than 1 whose value approximates sqrt(1 + (m-1)*rho) for the data's mean cluster size m and intraclass correlation rho. A ratio of approximately 1.0 means sessions were resampled instead of users.

Verify it yourself

Construct data where every user has exactly one session: the cluster and naive intervals then agree to within Monte Carlo error, since m = 1 makes the design effect 1.
Construct data where converted is constant within each user: rho is 1, and the width ratio should approach sqrt(m).
The bootstrap distribution's mean sits close to the point estimate; a large offset means the resampled arms have drifted in size.
Doubling B narrows the Monte Carlo noise in the percentile endpoints without shifting the interval's centre.

Follow-up

Users average 3.4 sessions and the intraclass correlation is 0.12. What width ratio do you predict before running it, and does your bootstrap land there?
Give the delta-method standard error for this ratio and say when you would prefer it to the bootstrap.
Half the users in the treatment arm have exactly one session. What does that do to the cluster bootstrap's coverage, and how would you check it?

Given a dataset, how would you find anomalies?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

What libraries do you commonly use for data analysis?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Read an experiment from first exposure, not assignment

Hard · experiments · window-functions · srm

fct_experiment_exposure holds experiment_id, unit_type, unit_id, variant, user_id, assigned_at_utc, first_exposed_at_utc, is_in_analysis_population and planned_end_utc. fct_event holds user_id, occurred_at_utc, is_core_action, and carries events up to a known data cut, :data_cut_utc. For one experiment randomised on unit_type = 'user', return per variant: exposed units, units with at least one core action in the seven days after that unit's own first exposure, the rate, and the variant share of exposed units. Only units whose seven-day window has fully elapsed as of the data cut belong in the readout. Units appearing under more than one variant are excluded from both arms and counted separately.

Approach

Run the contamination pass as an aggregate, not a window: SELECT unit_id FROM fct_experiment_exposure WHERE experiment_id = :exp GROUP BY unit_id HAVING COUNT(DISTINCT variant) > 1, then anti-join it away. PostgreSQL rejects COUNT(DISTINCT variant) OVER (PARTITION BY unit_id) outright, since DISTINCT is not implemented for window functions; if you want the test inline, MIN(variant) OVER (PARTITION BY unit_id) <> MAX(variant) OVER (PARTITION BY unit_id) is the equivalent that does run.
Do not resolve contamination by keeping the earliest variant. A unit that saw both arms carries treatment from both, so assigning it to either one biases that arm.
Define the population as is_in_analysis_population = TRUE AND unit_type = 'user' AND first_exposed_at_utc < planned_end_utc AND first_exposed_at_utc + interval '7 days' <= :data_cut_utc. The horizon filter is what makes the readout reproducible next week instead of drifting with every re-run; the data-cut filter is the one that actually buys seven days of follow-up, since a unit exposed an hour before the horizon otherwise contributes an hour of observation to a seven-day rate.
Measure the outcome on a per-unit relative window: LEFT JOIN fct_event on user_id with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc, first_exposed_at_utc + interval '7 days'). LEFT JOIN so units with no outcome stay in the denominator at zero rather than being deleted by an inner join.
Check the sample ratio before reading the effect: variant share of exposed units against the intended split, tested as a binomial. Run it on the truncated population as well as on the full exposed set, because if one arm exposes later on average the data-cut filter removes more of that arm and can manufacture a ratio mismatch the randomisation did not have. A mismatch on the full set means the exposure data is not a valid randomisation and invalidates the readout rather than being a footnote under it.
Report the per-variant rate, the absolute difference, and the fact that the variance unit is unit_id. That is straightforward here only because the grain is already one row per user; a per-session outcome under user randomisation would need a delta-method or bootstrap standard error instead.

Worked solution (35 min)

Count units per variant and units with more than one variant; report the contaminated count before anything else.
Build the exposed population CTE with both the horizon filter and the seven-days-before-data-cut filter, assert one row per unit_id, and record per variant how many units the second filter removed.
LEFT JOIN the bounded outcome events and flag conversion with a boolean aggregate.
Aggregate per variant to exposed, converted, rate and share, and run the binomial check on the share, both before and after the truncation.
Re-run with a fixed calendar window from experiment_start_utc and record the difference as the size of the dilution you avoided.

Expected result: One row per variant with exposed, converted, rate and share, plus a scalar count of excluded contaminated units. Every retained unit was first exposed at least seven days before the data cut, so each one has a full seven days of observation and the rates are directly comparable.

Verify it yourself

Exposed counts across variants sum to the population count minus the contaminated units.
MAX(first_exposed_at_utc) among retained units is at least seven days before :data_cut_utc.
Variant shares sum to 1.
No unit_id contributes to two variant rows.
The truncation drops a similar share of each arm; if it does not, note the imbalance alongside the SRM result rather than reading the effect.
Swapping the relative window for a calendar one changes the rates; if it does not, exposure timing is balanced and you can say so.

Follow-up

Some units were assigned days before they were exposed. What does analysing the assigned set instead do to the estimated effect, and in which direction?
The split is 51/49 on 400,000 exposed units. Do you read the result?
The treatment arm exposes on average two days later than control. What does that do to a fixed calendar outcome window, and which arm does it favour?

Discuss how you would develop a recommendation system for construction suppliers.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Given a dataset from a construction project, how would you analyze its success factors?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you approach a problem where the data is biased?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How do you prioritize tasks when faced with multiple deadlines?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you design an experiment to test a new product feature aimed at contractors?

Medium · a/b testing and experimentation

Approach

Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer would contain.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Diagnose a 50.5 to 49.5 split before reading the result

Easy · sample ratio mismatch · exposure logging · chi-square

An experiment randomised on visitor_id ran for 11 days. fct_experiment_exposure holds 104,320 control rows and 102,180 treatment rows for the experiment_id, all with is_in_analysis_population = TRUE, against a configured 50/50 split. The treatment arm shows a 3.1% relative lift in visit-to-signup conversion at p = 0.004, and the team wants to ship. Decide whether that lift can be reported, show the arithmetic behind your decision, and name the three checks you would run first on the exposure logging.

Approach

Test the split instead of eyeballing it. Chi-square goodness of fit against 103,250 expected per arm gives 2 x 1070^2 / 103,250 = 22.2 on 1 degree of freedom, p about 2.5e-6. The conventional alarm threshold is p < 0.001, so this is a sample ratio mismatch.
Treat the mismatch as invalidating, not as a caveat. The arms are no longer exchangeable, so the 3.1% lift has no causal reading and rebalancing on covariates afterwards does not restore it.
Walk the logging path in the order units flow through it: assignment service, then the exposure event fired on the treated surface, then the filters that build the analysis population. Each stage can drop one arm preferentially.
Name the three suspects this stack actually has. Exposure emitted by client code the treatment arm reaches later or not at all when its bundle errors; an extra redirect or render in one arm that loses slow connections; and a post-assignment filter (is_bot_flagged, is_internal, consent_state) that correlates with variant.
Check whether the imbalance is uniform by splitting counts on session_date, surface, app_version and device_type. A mismatch confined to one client build or to day one points at a deployment fault that can be fixed and re-run, not at a design fault.

Worked solution (15 min)

N = 206,500, expected 103,250 per arm, observed deviation 1,070, which is 0.518% of the total pushed into one arm.
Chi-square = 2 x 1070^2 / 103,250 = 22.18 on 1 df, p about 2.5e-6.
Compare with the p < 0.001 threshold and stop: the readout is void.
Re-pull the same counts grouped by session_date and app_version to localise where the rows were lost.
Deliver one line: the split is off by 1,070 visitors, a roughly one-in-400,000 event under correct randomisation, so the experiment is being fixed and re-run rather than read.

Expected result: Chi-square(1) = 22.2, p about 2.5e-6, far past the p < 0.001 mismatch threshold. The correct deliverable is an exposure-logging investigation and a re-run, not a caveated 3.1% lift.

Verify it yourself

Cross-check the statistic as a z on one arm count: under a fair split that count has SD sqrt(N/4) = sqrt(51,625) = 227 visitors, so the 1,070 deviation is 4.71 SD, and 4.71^2 = 22.2 reproduces the chi-square.
Chi-square scales linearly with N at a fixed ratio, so the same 50.518% split at N = 2,000 returns 22.18 / 103.25 = 0.21, p = 0.64. That reconciles the arithmetic and nothing more; it is not evidence that the alarm is independent of N. What rules out the large N is that the p-value is uniform under correct randomisation at every sample size, so p = 2.5e-6 is a one-in-400,000 draw whether N is 2,000 or 2,000,000. Large N buys the power to resolve a 0.5% skew; it does not manufacture one.
The mismatch p-value should be orders of magnitude smaller than the lift's p-value; if they are comparable, recheck the arithmetic.
Arm counts must sum to the total exposure rows for that experiment_id. If they do not, the analysis-population filter is already dropping rows before you looked.

Follow-up

The imbalance is entirely in one app_version on android. Can you salvage the other surfaces, and under what argument?
How would you monitor this across every running experiment without drowning in false alarms from a daily chi-square at alpha 0.05?
The missing rows turn out to be disproportionately visitors with consent_state = 'unknown'. What does that do to the estimate even after the logging bug is fixed?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Bring numbers, including the ones you are unsure of

An answer without a quantity is hard to interrogate, so interviewers keep probing until they find one. Come with the baseline, the change, the window it was measured over, and how confident you were. If the effect never got measured, say so and say what you would have measured. Fabricated precision is worse than an honest gap.

Describe your experience with statistical modeling.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that project again?
What did you decide not to do, and why?

Choose between three teams' requests with one analyst-week

Medium · prioritisation · stakeholder · judgement

You have one analyst-week. Three requests land the same morning. A growth team wants a paid-channel readout before a Friday spend decision. A billing team wants gross monthly revenue churn rebuilt, because the current figure recognises cancellation at canceled_at_utc rather than period_end_utc and is therefore wrong. A product team wants a dashboard for a feature launching in six weeks. All three sponsors are peers of your manager. Deliver your ranking, the explicit rule that produced it, and the message you send to the two teams you defer.

Approach

Recognise what is being probed: whether you prioritise on decision value and reversibility or on who asked most recently and most loudly. The generic answer sorts by importance; the strong one states a rule, applies it, and accepts the ranking it produces even where that is uncomfortable.
Score each request on three statable things: the decision it unblocks and the date that decision is made, the cost of being wrong in the meantime, and whether the work is one-off or compounding. A wrong published churn figure compounds, because it is quoted downstream and enters forecasts; the channel readout has a fixed date that cannot move; the dashboard has six weeks of slack.
Notice the tension between value and urgency rather than resolving it by feel. The churn defect is the most valuable item and the least urgent one, which is exactly the shape of work that never gets done.
Break the churn item in two. A one-hour severity check, sizing the gap between the two recognition points in MRR, is cheap enough to do before ranking anything and may promote the item outright. Do that first, then rank.
Make the deferrals concrete. Each deferred team gets a date, a reason expressed as another team's decision deadline rather than as relative importance, and the smallest thing you can hand them immediately.

Follow-up

The dashboard team escalates to your manager. What do you say in that conversation?
Your severity check shows churn is overstated by 15%. Does the ranking change, and does anybody need to be told today regardless of the ranking?
A fourth request arrives Wednesday with a Thursday deadline. What comes off the list, and who do you tell first?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?