Interview guides · ArcelorMittal Long Products Canada

ArcelorMittal Long Products Canada Data Scientist Interview Questions & Guide 2026

Causal inference drills for ArcelorMittal Long Products Canada data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

The Data Scientist role at ArcelorMittal Long Products Canada is a high-impact position situated at the intersection of heavy industrial operations and advanced analytical rigor. In an environment defined by massive scale, complex logistics, and precision manufacturing, your work directly informs how the organization optimizes production cycles, enhances safety protocols, and drives operational efficiency. You are not just building models; you are translating physical-world industrial data into actionable business intelligence that impacts the bottom line.

Product-sense cases reward reasoning from a mechanism to a testable prediction. Reciting every metric you can name reads as pattern matching; naming the single quantity that would move if your explanation were true reads as thinking.

Focus areas

Decompose a metric move by segment and mix
Define numerator, denominator and window precisely
Turn a vague request into a measurable question

Overview

This role requires a unique blend of technical depth and product-sense. You will collaborate with cross-functional teams, including engineering, production, and supply chain experts, to solve problems that are often ambiguous and highly technical. Whether you are diagnosing a drop in a specific production metric or designing experiments to test new process improvements, your ability to communicate complex statistical findings to non-technical stakeholders is as critical as your coding proficiency.

The ArcelorMittal Long Products Canada interview process

ArcelorMittal Long Products Canada candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Interviews with Department Heads

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Interviews with Subject Matter Experts

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Collaborative Problem-Solving

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Final Assessment

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Slicing a flat experiment until a segment reaches significance

Testing one metric across twenty segments at a nominal 5% level produces a significant result about two thirds of the time when nothing is happening anywhere, and the segment that surfaces is by construction the one with the most favourable noise. The reported effect in that slice is then badly overstated, because selection on significance conditions the estimate on being large. What makes it dangerous rather than merely wrong is that a post-hoc segment always has a plausible story attached, so it survives the meeting. The controls are declaring the small number of segments of interest before launch, correcting across the ones tested, and treating anything discovered afterwards as a hypothesis that needs its own adequately-powered test rather than a finding.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Over-explaining the method and under-explaining the implication

Lead with the answer and what you would do about it, then give the approach when asked. Roughly one sentence of method per three of implication is the right ratio for a stakeholder-facing answer; the interviewer already knows what a regression is.

Questions & practice

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Follow-up

The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Implement seven-day activation from its written definition

Medium · metric definition · cohorts · pandas · joins

Implement the seven-day activation rate. Inputs: dim_user with user_id, account_created_at_utc and is_internal; fct_event with user_id, occurred_at_utc and is_core_action. A user activates when core-action events carrying a non-NULL user_id fall on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). The denominator is every non-internal user whose account_created_at_utc lands in the cohort week, including users with no events at all. Return one row per cohort week with numerator, denominator and rate, publishing only weeks whose last signup is at least eight days old.

Approach

Build the denominator first, from dim_user alone, filtered on is_internal = False. Deriving it from the join is the standard way to lose every user who never fired an event, which is exactly the population the metric is about.
Join events to users on user_id with a left join from the user side, then apply the window as a half-open interval: occurred_at >= created AND occurred_at < created + 7 days. The right bound is exclusive, so an event at exactly created + 7 days does not count.
Count distinct UTC dates per user, not distinct events. Floor occurred_at_utc to date before the nunique, and do it in UTC rather than local time so the threshold does not move with the user's country.
Apply the >= 2 threshold, aggregate to cohort week, and compute the rate by re-summing numerator and denominator per week rather than averaging any per-user or per-day rate. Fix the week anchor explicitly: cohort_week is the Monday of the signup week in UTC, which is what Postgres DATE_TRUNC('week') returns and what any SQL version of this metric will produce. In pandas, subtract dt.weekday days from the floored timestamp. If you reach for periods instead, the anchor that matches is to_period('W') (equivalently 'W-SUN'), whose weeks end Sunday and therefore start Monday; to_period('W-MON') labels weeks that end on Monday, so it runs Tuesday through Monday and its start_time is a Tuesday. Mixing the two shifts every cohort label by one day and silently moves Mondays into the previous week.
Suppress immature weeks: drop any cohort week whose maximum account_created_at_utc is within 8 days of the data cut, and return them as absent rather than as a partial number.

Follow-up

The threshold is 2 distinct days. What changes in the reported history if someone moves it to 3, and how would you publish that change?
Invited seats and SSO-provisioned users have no pre-signup session. Should they be in this denominator at all, and what does including them do to the rate for sales-assisted accounts?
How would you produce the same metric at account grain, and which of the two would you put on the dashboard?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
Set ref = df['received_at_utc'].max() and build the five masks against it.
Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Describe a time you had to optimize a slow-running SQL query. What was your approach?

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

How do you ensure data integrity when merging datasets from disparate production-line systems?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

How would you use SQL window functions to calculate a moving average of production output over the last 30 days?

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
Write the active-account spine and the 28-day activity CTE.
Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
No account_id appears twice in the output.
Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

If we observe a sudden metric drop in our daily production efficiency, what is your step-by-step process for diagnosing the root cause?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

What framework do you use for product metric design when starting a project from scratch?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design an A/B test to determine if a new machine calibration setting improves yield?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

What are the most common experimentation pitfalls you have encountered, and how do you mitigate them?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

Estimate a threshold-triggered programme with regression discontinuity

Hard · regression discontinuity · instrumental variables · identification

Accounts reaching seats_licensed >= 25 in dim_account are automatically assigned a dedicated onboarding specialist; below 25 they are not. Leadership wants the programme's effect on 12-month net revenue retention and will not randomise coverage away from any account. Three years of dim_account and fct_subscription_period rows are available, including mrr_cents_constant_fx. Specify the design, the estimand it identifies, two threats that would invalidate it, and what changes when 8% of accounts below the threshold received a specialist anyway.

Approach

Set up a regression discontinuity on the running variable seats_licensed with a cutoff at 25, comparing accounts just below with accounts just above. The identifying assumption is continuity: absent the programme, expected 12-month NRR would be a continuous function of seat count through 25.
Name the estimand honestly and early. This is a local average treatment effect at 25 seats. It says nothing about a 5-seat or a 200-seat account, and that belongs in the first line of the answer rather than a footnote.
Estimate with local linear regression on each side, a triangular kernel, an MSE-optimal bandwidth and robust bias-corrected confidence intervals. Do not fit a high-order global polynomial; it imports weight from observations far from the cutoff and is known to manufacture discontinuities.
Test the two threats that actually apply. Manipulation: an account that wants the specialist can buy a 25th seat, which piles density just above the cutoff, so run a density test on seats_licensed around 25 and look for a spike at exactly 25. Bundling: if a price break, a plan tier or a support SLA also switches at 25 seats, the discontinuity measures the whole bundle, so check current_plan_tier and the price schedule at the cutoff.
Treat the 8% crossover as a fuzzy design. Treatment probability jumps at the cutoff without going from 0 to 1, so divide the jump in NRR by the jump in the probability of receiving a specialist. That is a Wald instrumental-variables estimator with the cutoff indicator as the instrument; it needs exclusion, which is exactly what the bundling check is about, plus monotonicity, and it narrows the estimand further to compliers at the cutoff.

Worked solution (45 min)

Fix the running variable as seats_licensed at the moment the assignment rule was evaluated, and fix the outcome as 12-month NRR computed from fct_subscription_period on mrr_cents_constant_fx for the account's cohort.
Plot mean NRR in one-seat bins on each side of 25 with a local linear fit. The picture comes before the estimate, because a discontinuity invisible in the binned plot is rarely real.
Run the density test at 25 and a continuity check on pre-cutoff characteristics such as billing_country, account_type and pre-programme MRR; these must be smooth through the cutoff.
Estimate the sharp RD with an MSE-optimal bandwidth and robust bias-corrected intervals, then repeat at half and double the bandwidth as a sensitivity check.
Estimate the first stage, the jump in specialist assignment at 25, and report the fuzzy estimate as the ratio with the estimand stated as the complier effect at the cutoff.

Expected result: A fuzzy regression discontinuity at 25 seats, estimated as the ratio of the NRR jump to the specialist-probability jump, identifying the local complier average effect at the cutoff under continuity, exclusion and monotonicity. With 8% crossover below the cutoff and near-full compliance above it, the first stage is about 0.92, so the fuzzy estimate is roughly 1.09 times the reduced-form jump and its standard error inflates by the same factor.

Verify it yourself

Characteristics fixed before the cutoff is crossed must not jump at 25. If account_type mix or billing_country jumps, the local comparison is not as-good-as-random.
The estimate should be stable across half and double the MSE-optimal bandwidth. Strong bandwidth sensitivity means the functional form is doing the work, not the discontinuity.
The reduced-form jump divided by the first-stage jump must reproduce the fuzzy point estimate exactly; if it does not, the two are being computed on different samples.
The effective sample inside the bandwidth, not the full account count, drives the standard error. Local linear RD typically needs on the order of three to four times the sample a randomised comparison would for the same power.

Follow-up

The density test shows a spike at exactly 25 seats. Is the design dead, and what would you do next?
You have 40,000 accounts but the bandwidth keeps 900. How does the detectable effect compare with a randomised comparison of the same nominal size?
Seat counts change over time. Which value of seats_licensed is the running variable, and what breaks if you pick the wrong one?

A conversion rate that fell in one regulatory region

Medium · consent · identity-keys · denominator · geography

Visit-to-signup conversion fell 1.3 points over six weeks. Signups cut by dim_user.country_code put the fall in one regulatory region where a consent banner shipped in week one, but absolute signups from that region are flat. fct_session carries consent_state, visitor_id, is_bot_flagged and session_date and no country column, so the denominator cannot be cut the same way. Using fct_session and fct_event, decide whether behaviour changed or the denominator did, state what these tables cannot settle, and name the one column that would settle it.

Approach

Name the asymmetry before computing anything. The numerator is user-keyed and therefore cuttable by country; the denominator is visitor-keyed and is not. Dividing a region-filtered numerator by an unfiltered denominator produces a quantity that is not a rate, and presenting it as a regional conversion rate is the first mistake available here.
Attack the denominator on the dimension you do have. Compute distinct visitor_id per week and sessions per distinct visitor_id per week: a consent banner that blocks or shortens the identity cookie raises the distinct-visitor count and lowers sessions per visitor, which depresses any visitor-keyed rate with no behaviour behind it.
Split on consent_state. Sessions with consent_state = 'denied' can enter the denominator but can never be joined forward to a signup, so a rising denied share mechanically drives the pooled rate down by roughly its own share. Report the granted-only rate and the denied share as two separate numbers rather than one blended figure.
Cross-check with measures that do not depend on the visitor key at all: absolute weekly signups, which are given as flat, and signups per session rather than per visitor.
State the limit honestly. Without country on the session or on its entry event, the regional attribution rests on the numerator alone, and the correct request is that one column, not a more elaborate model on top of the data you have.

Follow-up

If granted-only conversion is the metric going forward, what selection bias have you accepted, and in which direction does it point?
How would you handle the six weeks of already-published history once the new definition is adopted?
What is the smallest instrumentation change that restores a cuttable denominator without collecting more personal data than before?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

Given a table of sensor readings, how do you handle missing values or outliers before feeding the data into a model?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

The executive says to give the midpoint and they will manage the risk. What do you do?
How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?

Disagree with a product manager's roadmap claim using data

Medium · causal inference · selection bias · stakeholder

A product manager proposes building a feature on the argument that accounts connecting an integration in week one retain three times better at week four. The figure is correctly computed from dim_user and fct_event, and it has already been shown to leadership. You have one scheduled 1:1 before the roadmap locks. Deliver the specific analysis you would run to test whether the relationship is causal, the result that would change your own mind, and how you open the conversation so that the PM is not put in the position of defending the number in public.

Approach

Recognise what is being probed: whether you can separate a number being right from an inference being wrong, and do it without costing the PM face. The generic answer recites that correlation is not causation; the strong one names the specific confound and proposes the cheapest design that could distinguish the explanations.
State the alternative concretely. Accounts that connect an integration in week one are accounts that already have a workflow and a technical owner, so week-one intent plausibly drives both the connection and week-four retention. The selection is on intent, which no amount of post-hoc adjustment observes.
Order the discriminating analyses by cost. First, condition on pre-connection activity by comparing retention within strata of week-one core-action count, which removes the crude version of the confound but not unobserved intent. Second, look for variation in integration availability that was unrelated to intent, such as a staggered release or an outage window. Third, an encouragement design that randomises a prompt to connect and reads the intent-to-treat effect on week-four retention, which is the only version that identifies an effect.
Run the timing check, because it is nearly free and it is the most persuasive single piece of evidence. If the retention advantage among connectors is already visible before any of them connected, the causal story is largely finished.
Pre-commit to what would change your mind and say it before you show anything: if the gap survives stratification and the encouragement arm moves week-four retention at all, the feature has a case and you will say so.
Open the 1:1 by agreeing with the true part, that the correlation is real and worth chasing, then ask what effect size the roadmap plan assumes. That makes the size of the claim the topic instead of its authorship.

Follow-up

The encouragement test needs six weeks and the roadmap locks in two. What do you recommend in the interim?
Stratifying on week-one activity closes half the gap. What do you conclude, and what do you still not know?
How would you word this in the roadmap document so the PM's original number is reframed rather than deleted?