Interview guides · Kinaxis

Kinaxis Data Scientist Interview Questions & Guide 2026

Causal inference drills for Kinaxis data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

A Data Scientist at Kinaxis plays a pivotal role in transforming complex supply chain data into actionable intelligence. By leveraging advanced analytics and machine learning, you will help global organizations navigate the volatility and uncertainty inherent in modern supply chains. Your work affects how Kinaxis customers optimize planning, reduce waste, and improve agility through the Kinaxis platform.

SQL is seldom the hardest round and is often the one that eliminates people. The working bar is usually window functions, correct deduplication, and joins that do not silently fan out rows, rather than obscure syntax.

Focus areas

Set fraud thresholds by expected cost
Reconcile amounts in minor units and currency
Report only matured cohorts for loss metrics

Overview

This role is highly collaborative and requires a blend of technical rigor and product-oriented thinking. You will sit at the intersection of data engineering, software development, and business strategy, often working on projects that require both deep statistical analysis and the ability to articulate findings to non-technical stakeholders. Whether you are building predictive models or analyzing large-scale datasets, you are expected to be a problem-solver who can navigate ambiguity and drive tangible value for the business.

The Kinaxis interview process

Kinaxis candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Discussions

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Project Deep Dives

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Conversational Interviews

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Structured Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Where candidates lose points

Averaging delinquency across a book that is growing

A loan three months old cannot be 90 days past due, so a portfolio with many recent originations reports a low blended 90+ rate purely from age mix. The blended rate falls fastest exactly when originations grow fastest, which is precisely when credit quality most needs watching, so the metric moves in the reassuring direction during the riskiest period. Only comparisons at equal months on book are valid, which is what a vintage or roll-rate view enforces.

Using written premium as the denominator of a loss ratio

Premium is written at inception and earned pro rata across the exposure period, so in a growing book written premium runs ahead of earned premium and the loss ratio comes out too low, with the error reversing when the book shrinks. The numerator has the mirror-image problem if it omits incurred-but-not-reported reserves, since recent accident periods then look profitable twice over. Both sides must refer to the same exposure period, which is what an accident-period view at a fixed development age enforces.

Interpreting a change before checking data quality and logging

Spend the first pass on row volume by day, null rates, duplicate keys, and whether the step change lands on a release or tracking-migration date. A discontinuity that coincides with a deploy is an instrumentation hypothesis before it is a behavioural one.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Questions & practice

Collapse retry chains and compute a dollar-weighted approval rate

Medium · sessionisation · window functions · dollar-weighted rates

fct_payment_authorization gives auth_id, card_token_id, merchant_id, amount_minor, transaction_currency, requested_at, auth_result, is_reversal, channel and issuer_country. Two reference frames give the minor-unit exponent per currency and a daily rate to one reporting currency. Collapse retry chains first: attempts sharing card_token_id, merchant_id and amount_minor whose consecutive gaps are under 15 minutes form a single attempt, whose outcome is its last row. Exclude reversals and zero-amount verifications. Return a 7-day rolling dollar-weighted approval rate by channel and issuer_country.

Approach

Filter before grouping: drop is_reversal rows and zero-amount verifications, since neither is a purchase attempt and both would otherwise sit in the denominator.
Sort by card_token_id, merchant_id, amount_minor and requested_at, take the gap to the previous row within that key, mark a chain start where the gap exceeds 15 minutes or the key changes, and label chains with a cumulative sum of that flag. This is a gap rule between consecutive attempts, not a fixed clock bucket, so a chain may span more than 15 minutes in total.
Keep each chain's terminal row by requested_at. If a retry was approved, the purchase was approved; keeping the first row reports the decline that caused the retry as the outcome.
Convert amounts exactly once: amount_minor divided by 10 to the power of the currency exponent, multiplied by the reference rate for the authorization date. Do not reach for settlement_fx_rate, which is null on precisely the declined rows the denominator needs.
Build the rolling window as a ratio of two rolling sums, approved value over total value, per channel and issuer_country. A rolling mean of daily ratios weights a quiet Sunday the same as a busy Friday.

Follow-up

The count-weighted rate is flat while the dollar-weighted rate falls 80 basis points. What do you look at first?
How would you choose the 15-minute window rather than inheriting it?
A merchant moves from two retries to five. Which of your two rates moves, and is that a real change in approval quality?

Measure calibration of a twelve-month default probability from scratch

Hard · calibration · brier score · binning

fct_loan_application gives application_id, model_pd_12m, model_version, decision, funded_at and loan_id. fct_loan_performance_monthly gives loan_id, months_on_book, days_past_due and charge_off_flag. Define the outcome as ever 90 or more days past due, or charged off, by months_on_book = 12. Without sklearn or scipy, build an equal-count binned reliability table, the expected calibration error, the Brier score and its reliability, resolution and uncertainty components, and report the residual the binned identity leaves behind. Restrict to cohorts that have actually reached 12 months on book.

Approach

Build the label first and name the population it covers out loud: only funded loans have outcomes, so this measures calibration on the approved population. The declined region is unmeasured, and no binning scheme repairs that.
Restrict to applications whose loans have reached months_on_book = 12. A cohort observed at 8 months has a mechanically lower default rate and will read as systematic over-prediction that is really just immaturity.
Bin by equal count, deciles of model_pd_12m through a rank-based cut, not equal width. The PD distribution is heavily right-skewed, so equal-width bins put most of the mass in the first bin and leave the risky bins with single-digit counts whose observed rates mean nothing.
Per bin compute n, mean predicted, observed rate, and the binomial standard error sqrt(o(1-o)/n) so a gap can be read against noise. ECE is the count-weighted mean absolute gap between mean predicted and observed.
Compute Brier directly as the mean squared error, then reliability = sum of n_k (pbar_k - obar_k)^2 over N, resolution = sum of n_k (obar_k - obar)^2 over N, uncertainty = obar(1 - obar). Report residual = Brier - (reliability - resolution + uncertainty). That identity is exact only for discrete forecasts, so with binned continuous scores the residual is the within-bin spread of the score; a large one means the bins are too wide to support the decomposition.
Split by model_version. A mixed-version population can look well calibrated in aggregate while each version is biased in opposite directions.

Follow-up

AUC is unchanged after a population shift but the reliability curve has moved. What happened, and what do you do about it?
How would you recalibrate without retraining, and what would you check afterwards?
The top decile shows observed default well above predicted. Is that a calibration problem or a policy problem?

Build a vintage delinquency table without pivot or unstack

Easy · vintage analysis · cohorts · groupby · cumulative max

fct_loan_performance_monthly gives loan_id, origination_month, months_on_book, days_past_due, charge_off_flag and restructured_flag. Produce a DataFrame with one row per origination_month and columns for months_on_book 0 through 12, each cell holding the share of that vintage's funded loans that had ever reached 90 or more days past due, or charge-off, by that age. You may not use pivot, pivot_table, crosstab or unstack. Cells for ages a cohort has not yet reached must be NaN rather than zero.

Approach

Define the per-row indicator as days_past_due >= 90 or charge_off_flag, then take a cumulative maximum of it per loan ordered by months_on_book, because the metric is reached-by-age-m, not in-that-state-at-age-m.
Deal with restructuring before the cumulative max. Restructuring resets days_past_due, so a restructured loan re-enters at current and, without the cumulative maximum carrying its pre-restructure worst state, reads as a cure.
Fix the denominator once as the count of distinct loan_id per origination_month across the whole cohort. Prepaid and charged-off loans stop producing rows, so a denominator recomputed at each age silently shrinks exactly where losses land.
Aggregate with groupby(['origination_month','months_on_book'])['ever_90'].sum(), then pre-build the output frame indexed by sorted origination months with integer columns 0 to 12 and assign from the grouped Series by .loc on its index.
Mask cells beyond each cohort's maximum observed months_on_book so an immature cell reads NaN instead of an artificially low rate.

Worked solution (30 min)

Sort by loan_id and months_on_book, build the ever_90 indicator, then apply groupby('loan_id')['ever_90'].cummax().
Compute cohort_size as the distinct loan_id count per origination_month, before any filtering on age.
Group the cumulative indicator by origination_month and months_on_book and sum it to get the numerator per cell.
Create the output frame with the sorted origination months as index and range(0, 13) as columns, fill it from the grouped Series, and divide each row by its cohort_size.
Compute each cohort's maximum observed months_on_book and set every cell to the right of it to NaN.

Expected result: A DataFrame with one row per origination_month and columns 0 to 12, values non-decreasing left to right within every row, near zero in column 0, and NaN forming an upper-right triangle where cohorts have not yet aged.

Verify it yourself

Every row is non-decreasing across columns; a decrease means the cumulative maximum was not applied or a restructure reset leaked through.
The denominator per row equals the cohort's distinct loan count and does not change across columns.
Column 0 is at or near zero for every vintage, since a loan cannot be 90 days past due at its first month end.
origination_month plus months_on_book is constant along an anti-diagonal, so one anti-diagonal should reproduce a direct calendar-month filter.

Follow-up

Two adjacent vintages diverge at months_on_book 6. How would you separate seasoning, mix shift and a genuine credit-quality change?
The three most recent vintages look best on this table. What do you check before saying so?
How does the table change if charge-off policy moved from 180 to 120 days past due partway through the series?

Write a query using SQL window functions to calculate a rolling average of supply chain metrics.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Describe a scenario where you had to optimize a slow-running SQL query.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Explain how you would perform a conditional aggregation to categorize customer segments.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

Accident-quarter loss ratio on earned rather than written premium

Hard · loss ratio · earned vs written · development age

From fct_policy_period_monthly, compute the accident-quarter loss ratio by product_line: incurred losses, being paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor, over earned_premium_minor for the same accident quarter. State explicitly whether loss_adjustment_expense_minor is included and apply that choice consistently. Also output the same ratio computed on written_premium_minor so the two can be compared. The table holds current values with no valuation-date snapshot. Say in one line which comparison this schema cannot support and what you would need to support it.

Approach

Derive the accident quarter from as_of_month with date_trunc, and note that the table already attributes losses to the month of the loss event while earning premium pro rata into the same month, which is what makes the two sides comparable at all.
Aggregate earned_premium_minor, written_premium_minor and the three loss components to product_line and accident quarter in one pass, keeping loss adjustment expense as its own column so the inclusion choice is a final-select decision rather than something buried in a CTE.
Compute both ratios side by side and a third column for their difference, because the size and sign of that difference is a direct read on whether the book grew or shrank in the quarter.
State the limitation plainly: every row carries today's reserve estimate, so each accident quarter is observed at a different development age and a cross-quarter comparison mixes development with underwriting. A fixed development age needs a valuation-date dimension, that is one row per accident period per valuation, which this table does not have.
Guard against the mirror-image error on the numerator by confirming ibnr_reserve_minor is non-zero on recent quarters; if it is null or zero there, the recent periods are understated twice over and the series is not usable.

Worked solution (40 min)

CTE quarterly: group fct_policy_period_monthly by product_line and date_trunc('quarter', as_of_month), summing earned_premium_minor, written_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor.
Final SELECT: build incurred_minor as the three loss components plus the LAE column, with the LAE inclusion written as a named expression so the choice is visible on the page.
Emit loss_ratio_earned and loss_ratio_written, both cast to numeric, plus their difference and the written-to-earned premium ratio.
Order by product_line and accident quarter, and append the one-line note about the missing valuation dimension to the query as a comment.

Expected result: One row per product_line and accident quarter with earned and written premium, incurred losses, both ratios and their difference. In quarters where written premium exceeds earned, loss_ratio_written is strictly below loss_ratio_earned, and the gap widens with the growth rate.

Verify it yourself

For a mature product line with flat volume, summed earned premium over a full policy term is close to summed written premium, so the two ratios converge there.
The sign of loss_ratio_earned minus loss_ratio_written matches the sign of written premium minus earned premium in every quarter, which is an arithmetic identity and not a data finding.
Recent accident quarters have non-zero ibnr_reserve_minor; a zero there means the numerator is missing its tail and the ratio is understated.
Within a quarter, loss_ratio_earned divided by loss_ratio_written equals written_premium_minor divided by earned_premium_minor exactly, because both ratios share one numerator. Any deviation means the LAE term was built into one ratio and not the other, which is the real test that the inclusion choice was applied consistently.
Recomputing with LAE excluded scales each quarter's ratio by incurred_minor / (incurred_minor + lae_minor). That factor is not constant across quarters unless LAE happens to be booked as a fixed fraction of incurred losses, so a uniform shift is evidence about how LAE is loaded, not evidence that the query is self-consistent — use the previous check for that.

Follow-up

Written premium exceeds earned premium by 18 percent this quarter and by 3 percent two years ago. What happened to the book, and what does it do to each ratio?
How would you build a development triangle from a valuation-dated version of this table, and what would you use the chain-ladder factors for?
Statutory presentation conventionally takes the expense ratio on written premium while the loss ratio uses earned. How do you avoid a combined ratio that quietly mixes the two bases?

If you observe a sudden metric drop after a product release, what is your systematic approach to diagnosing the cause?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How do you design a product metric to measure the success of a new supply chain forecasting feature?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What are the common experimentation pitfalls that lead to false positives?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Name the randomisation unit first; it decides the variance and what the test can detect.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

Explain the trade-offs between a frequentist and Bayesian approach in an A/B test.

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Say whether units interfere with each other, and switch design if they do.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

Measure an underwriting automation programme without hiding cost

Medium · metric gaming · swap set · operations metrics

An operations programme aims to cut manual underwriting. The proposed metric is manual review rate: decided_by = 'manual' over applications with decision in ('approve','decline') in fct_loan_application. The team controls the referral rules recorded in policy_rule_hits. Explain how they can halve that metric in one release while making the book worse, then design the readout that catches it: primary metric, the cuts it must always be reported in, and the guardrails. Say what evidence you would require before declaring a batch of referrals safe to auto-decide.

Approach

Name the exploit precisely. Referrals can be converted to automatic declines by a rule change, which removes them from the numerator at no operational cost while they stay in the denominator, so the rate falls and applicants a human would have approved are rejected instead. Auto-approving the same population moves the metric identically with the mirror-image failure mode.
Refuse the bare rate as a primary. Make it conditional on outcomes: manual review rate measured at a held-constant approval rate within each model_pd_12m band, so a reduction is creditable only when the decisions themselves did not move.
Require fixed cuts on every report: by model_pd_12m band and by the rule identifier inside policy_rule_hits. A single blended operations number cannot be decomposed after the fact, and the decomposition is the entire diagnostic.
Set guardrails on the things the exploit disturbs: approval rate within each score band, the override rate where manual decisions disagree with the automated recommendation, and the vintage 90+ rate at months_on_book = 12 for the newly auto-decided population against the referrals of the prior regime.
Fix the evidence bar for retiring a referral rule: a shadow period where the rule fires and the case is still worked manually, recording what the automation would have decided, what the human decided, and, for the disagreement group alone, the realised performance. Agreement on the easy majority establishes nothing about the disputed minority.
State the conflict honestly rather than designing around it. The referral population exists because the model is least certain there, so automating it trades loss for cost by construction, and the readout's job is to price that trade rather than to make it disappear.

Worked solution (30 min)

Compute manual review rate for a recent month overall, then by model_pd_12m band, then by rule identifier extracted from policy_rule_hits.
For each rule, compute the approve share and decline share of the applications it touches, so the rules whose removal would actually change decisions are identified before anyone proposes removing them.
Simulate the exploit: reclassify one high-volume referral rule's applications as automatic declines, then recompute the blended rate and the band-level approval rates side by side.
Write the readout spec: the primary with its conditioning, the two mandatory cuts, the three guardrails, and the shadow-period evidence bar.

Expected result: A table in which the simulated rule change lowers the blended manual review rate materially while approval rate in the affected model_pd_12m bands falls at the same time, together with a readout spec that makes it impossible to publish the first effect without the second.

Verify it yourself

The simulated change lowers the manual review rate and the band-level approval rate together. If band approval rate is unchanged, that rule was not binding and its removal is genuinely free.
Rule-level counts reconcile to the blended rate once applications hitting several rules are counted once. A jsonb array of rule identifiers is the usual place this double-counts.
The newly auto-decided population and the prior manual referrals are compared at equal months on book, not at equal calendar dates.

Follow-up

The override rate falls to near zero after the change. Is that good?
How long must the shadow period run, given the outcome matures at twelve months on book?
The rule fires on 200 applications a month. What can and cannot be established at that volume?

Manual review rate doubles on the day two changes ship

Hard · definition change · model version · population stability

Manual review rate on fct_loan_application, defined as decided_by = 'manual' over decisions in ('approve','decline'), stepped from 9 to 18 percent on a single day. That day a new model_version went live, and the same release changed which referral paths are recorded as manual. Columns: submitted_at, channel, bureau_score, model_pd_12m, model_version, policy_rule_hits, decision, decided_by, decision_at. Attribute the step across definition, model and population, with a number against each.

Approach

Establish the shape before the cause. A definition or code change produces a step at the deploy timestamp; a population change produces a ramp. Plot at hourly granularity either side of the release and read which one this is.
Dual-run the definition. Recompute the metric under both the old and the new recording rules across several months of history. If the historical series under the new rule sits near 18 percent throughout, that part of the step is definitional and should be restated, not investigated.
Hold the model constant and decompose the residual by policy_rule_hits identifier. A referral rule keyed to a score threshold shows its own step, and naming the rule is what makes the finding actionable to the policy owner.
Test the score, not the applicants. Compare model_pd_12m distributions by model_version on the same applications where shadow scores exist; otherwise compare adjacent weeks with a population stability index, treating the customary 0.1 and 0.25 marks as heuristics rather than tests, and assess calibration separately from ranking.
Test the applicants, not the score. Compare channel mix, bureau_score distribution and null-bureau share across the boundary. If those are stable, population is not a driver and should be reported as approximately zero rather than left unquantified.
Deliver three components that sum to the observed step, each with the team that owns it.

Follow-up

If the new model is better calibrated, is a higher referral rate the correct outcome or a threshold nobody retuned?
How would you have caught the recording change before it reached the dashboard?
What is the swap set here, and which swap group would you inspect first?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Describe a project where you faced a significant technical obstacle; how did you overcome it?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Recommend a decision whose true outcome matures a year later

Hard · decision under censoring · leading indicators · staged rollout

An underwriting rule change must be decided in six weeks. Its real outcome, the vintage 90-plus rate at months_on_book 12 in fct_loan_performance_monthly, matures in a year. The executive wants a yes or no, not a range. Randomising the credit decision across the whole population is not available. Name the leading indicator you would accept, state its bias and the direction of that bias, define the decision rule and stopping condition before any rollout starts, and say what reading would make you recommend reversing the change.

Approach

Fix the readout before the rollout, because a readout chosen after the data arrives is a story rather than a decision rule: indicator, window, threshold and reversal condition all go in writing first.
Choose the leading indicator on its measured relationship to the matured outcome in historical vintages rather than on availability. Early delinquency, typically the share reaching dpd_1_29 or missing a first scheduled payment by months_on_book 3, is the usual candidate, and you quantify how well it predicted the 12-month rate across past cohorts.
State the bias and its direction plainly: early delinquency under-represents default that emerges later and is contaminated by servicing and payment-date effects, so treat it as a floor on risk rather than an estimate of it.
Buy identification where full randomisation is unavailable: a narrow randomised approval band around the cutoff, or a staged rollout by channel or region read as a difference-in-differences, with the parallel-trends assumption stated and checked in the pre-period rather than assumed.
Give the executive the binary they asked for with the trigger attached in the same sentence: yes, conditional on the month-3 indicator staying inside a stated band, with an automatic hold if it breaches.

Follow-up

How would you validate that the month-3 indicator predicts the 12-month outcome, and what evidence would invalidate it mid-rollout?
Compliance refuses a randomised band. What is your next-best identification strategy, and what precision do you lose by taking it?

Turn a one-line fraud-number request into a scoped brief

Easy · scoping · metric definition · denominators

A stakeholder messages: what is our fraud rate, and is it going up? You have fct_payment_authorization, fct_card_dispute and dim_customer. At least four defensible answers exist: count-weighted or value-weighted, attributed to the transaction month or to the dispute filing month, and gross or net of recoveries and successful representments. You get one reply before someone else produces an uncaveated number. Write that reply: the clarifying questions you ask, the single default you will produce if nobody answers, and what the default excludes.

Approach

Establish the decision behind the question first, because a risk-rule change, a board number and a merchant contract negotiation need different denominators, and asking which one is not stalling.
Offer a short menu rather than an open question: a stakeholder can choose between two named options but cannot specify a denominator from scratch.
Commit to a default so the reply is useful even if nobody answers, for example net fraud loss in basis points of settled volume, attributed to the requested_at month, matured months only.
State the exclusions in the same breath as the default: non-fraud dispute categories, transaction months with less than 120 days of maturity, and first-party abuse that arrives coded as consumer_dispute.
Give a delivery time for the default and a longer one for the fuller cut, so the choice between them carries a visible cost.

Follow-up

They come back wanting it by merchant for a contract negotiation. What changes in the definition and in the maturity rule?
How would you separate first-party abuse from third-party fraud in this data, and what would you refuse to conclude from the split?