Interview guides · KNIME

KNIME Data Scientist Interview Questions & Guide 2026

Diagnose a metric move for KNIME data scientist prep: split by segment, rule out a logging change, separate mix shift from behaviour.

The 60-second brief

As a Data Scientist at KNIME, you are at the intersection of advanced analytics and user empowerment. KNIME is synonymous with its open-source platform, which allows users to create data science workflows without deep coding requirements. In this role, your mission is not just to build models, but to demonstrate the power of the KNIME platform to a global community of data practitioners, customers, and partners.

Ask early whether the loop includes an asynchronous take-home or a timed live case, because the two are graded on different things. A take-home is read as an artifact: the question you decided to answer, what you did about missing or malformed records, and a conclusion stated plainly enough for someone to act on. A reviewer who cannot rerun your notebook discounts the result whatever score is printed in it. Hold to the stated time box and write down what you would have done with more of it, since the follow-up round is usually a live defence of the same work.

Focus areas

Separate authorization, settlement and dispute outcomes cleanly
Read vintage curves, not blended portfolio averages
Reconcile amounts in minor units and currency

Overview

You will contribute to a product-centric environment where your technical expertise directly influences the toolsets used by thousands of analysts. Whether you are optimizing internal data processes, assisting customers with complex analytical challenges, or refining the features of the KNIME platform, you are expected to bridge the gap between high-level machine learning theory and practical, scalable application.

The role demands a balance of intellectual curiosity and pragmatism. You will work within a collaborative, international team that values clear communication and the ability to explain complex concepts—such as or methodologies—to stakeholders with varying levels of technical expertise. Expect an environment that is rigorous in its evaluation of your technical depth but deeply invested in your ability to communicate your findings effectively.

The KNIME interview process

KNIME candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Presentation

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Cross-Functional Interviews

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Senior Leadership Meeting

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Where candidates lose points

Reading the most recent months of fraud and dispute rates as final

Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, and several reason codes run considerably longer, so the disputes belonging to a recent transaction month have simply not been filed yet. Any chart attributed by transaction date therefore slopes down at the right edge regardless of what is happening. The fix is to report only matured cohorts, or to apply development factors estimated from completed months and to show the estimate as an estimate.

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Reading experiment results before checking the arm split

Compare observed arm counts against the intended allocation ratio, not an assumed even split, and set the alarm far below the conventional 0.05: at 0.05 roughly one healthy experiment in twenty trips it, which is why sample-ratio checks usually run at p < 0.001 or stricter. The test's power scales with sample size, so it misses a real diversion on a small experiment and fires on an imbalance too small to move the estimate on a very large one. A flag means go find the assignment or logging fault before reading any outcome, not report a mismatch.

Never asking what decision the analysis will inform

Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.

Questions & practice

What are Decision Trees in machine learning, and how exactly do they work?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Write integrity checks for the authorization and settlement lifecycle

Easy · data quality · minor units · fx reconciliation

You are given fct_payment_authorization as a pandas DataFrame with auth_id, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, is_reversal, parent_auth_id, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency and settlement_fx_rate. Write a function returning one row per integrity check with the check name, failing row count, failing share and up to five example auth_id values. Cover at least six checks, one of which reconciles captured_amount_minor against settlement_amount_minor through settlement_fx_rate. Partial capture, zero-amount verification and a decline with no capture are all legitimate and must not be flagged.

Approach

Separate contract violations from observations before writing any code: an approved row carrying a decline_reason_code is structurally impossible, while a capture two days after requested_at is merely slow and belongs in a different severity tier.
Express each check as a boolean mask over the whole frame and collect the masks in a dict, so the summary table is one comprehension over mask.sum() rather than a row loop.
For the reconciliation, leave minor units before comparing: expected = captured_amount_minor / 10exponent[transaction_currency] * settlement_fx_rate * 10exponent[settlement_currency]. Build the exponent table covering zero-decimal and three-decimal currencies instead of assuming two everywhere.
Guard the legitimate cases explicitly so each mask fires only on the genuine contradiction: captured_amount_minor below amount_minor is partial capture, amount_minor of zero on an approved row is account verification, a null captured_at on a declined row is correct.
Sort the output by failing share times a stated severity weight, because a check firing on 0.01 percent of rows can still be the one that breaks a ledger reconciliation.

Worked solution (25 min)

Assert auth_id is unique, then build a currency exponent lookup that includes the zero-decimal and three-decimal currencies present in the data.
Define masks for: approved with non-null decline_reason_code; declined with non-null captured_at; captured_amount_minor above amount_minor with parent_auth_id null; captured_at before requested_at; settled_at before captured_at; is_reversal true with parent_auth_id null; settlement_currency differing from transaction_currency while settlement_fx_rate is null.
Add the exponent-aware reconciliation mask with a tolerance of one minor unit plus a small relative term.
Assemble a frame of check_name, n_failing, pct_failing and up to five sample auth_id values, ordered by severity then share.
Read five flagged rows per check by hand and confirm each is genuinely contradictory before reporting any counts.

Expected result: A DataFrame of at least eight rows, one per check, each with n_failing, pct_failing and example auth_id values. The reconciliation check should fire on a whole currency at once rather than on scattered rows, because an exponent error is systematic while an FX error is not.

Verify it yourself

Every mask returns false on three hand-constructed legitimate rows: a partial capture, a zero-amount verification and a decline with no capture.
Each pct_failing uses the denominator that check actually applies to, with reversals and verifications excluded where they are out of scope.
The reconciliation residual is centred at zero with spread under one minor unit; residuals clustered at exactly 100x or 0.01x mean an exponent is wrong, not a rate.

Follow-up

Which of these would you run as a blocking pipeline assertion and which as a monitored metric, and why?
The FX check fails on 3 percent of rows, all in one settlement currency. How do you decide between a data bug and a rounding convention?
How would you detect that a currency's minor-unit exponent is wrong in your reference table, using only the transaction data?

Implement accident-quarter loss ratio at twelve months development

Medium · loss ratio · development age · earned premium

fct_policy_period_monthly arrives as a stack of month-end snapshots: each row carries valuation_month alongside as_of_month, policy_id, product_line, written_premium_minor, earned_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor. Compute the accident-quarter loss ratio at exactly 12 months of development: incurred losses over earned premium, both taken from rows whose as_of_month falls in the accident quarter, read from the snapshot 12 months after that quarter closes. Report quarters that cannot reach that age as incomplete rather than dropping them.

Approach

Derive accident_quarter from as_of_month, then define the evaluation snapshot per quarter as valuation_month equal to the quarter's final month plus twelve months. Every figure in the ratio comes from that one snapshot, not from whichever snapshot happens to be newest.
Numerator is paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor over the accident quarter's rows in that snapshot. Loss adjustment expense may be included or not, but the choice applies to every quarter and is named in an output column.
Denominator is earned_premium_minor over the same rows. Written premium is booked in full at inception, so in a growing book it runs ahead of earned premium and drags the ratio down, with the error reversing when the book shrinks.
Left-join the full quarter list against available valuation months so a quarter with no 12-month snapshot yields status incomplete and a null ratio, instead of disappearing and shortening the series without saying so.
Split by product_line, since both the loss ratio level and the speed of development differ by line, and a blended series moves with mix as much as with experience.

Follow-up

The most recent complete quarter came in four points better than the one before. What do you check before calling it an improvement?
How would you estimate the 12-month figure for a quarter that is only 6 months developed, and how would you label the estimate?
Why can an expense ratio legitimately use a different denominator from the loss ratio in the same presentation?

How do you determine if a result is statistically significant, and what actions do you take if it is not?

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Explain how you would use SQL window functions to calculate a rolling average or identify trends over time.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Count-weighted and dollar-weighted approval rates on one currency

Easy · aggregation · denominators · minor units

Using fct_payment_authorization, report the trailing 7-day authorization approval rate two ways for transaction_currency = 'EUR': count-weighted, and dollar-weighted on amount_minor. Exclude is_reversal = true, exclude incremental authorizations (parent_auth_id not null), and exclude zero-amount account verifications. auth_result = 'approved' is the numerator; the four declined_* values make up the rest of the denominator. Return channel, attempts, approved_attempts, approval_rate_count and approval_rate_value. State every exclusion and its reason before you write the SELECT.

Approach

Say the denominator out loud first: attempts on a single transaction currency, excluding reversals, incremental authorizations and zero-amount verifications, because none of those is a purchase attempt a merchant is trying to get approved.
Filter requested_at against a half-open interval (>= start AND < end) so the boundary day is neither dropped nor double counted.
Compute both rates in one pass with FILTER clauses: COUNT() FILTER (WHERE auth_result = 'approved') over COUNT(), and SUM(amount_minor) FILTER (WHERE auth_result = 'approved') over SUM(amount_minor).
Cast one side of each ratio to numeric before dividing, since amount_minor and the counts are integers and integer division silently truncates to zero.
Group by channel and sort by the value-weighted rate, then read the gap between the two rates as a statement about where the declines sit rather than as noise.

Worked solution (20 min)

Write the exclusion list as comments above the query: is_reversal = false, parent_auth_id is null, amount_minor > 0, transaction_currency = 'EUR'.
Build a single aggregate query over fct_payment_authorization with a half-open requested_at predicate and those four filters.
Emit attempts, approved_attempts, approval_rate_count and approval_rate_value with FILTER clauses and a numeric cast on the numerator.
Group by channel, order by approval_rate_value ascending so the worst channel is on top.

Expected result: One row per channel present in the window, with attempts and approved_attempts as integers and both rates as numerics strictly between 0 and 1. approval_rate_value is typically lower than approval_rate_count on ecommerce and moto, because declines concentrate in higher-value card-not-present traffic.

Verify it yourself

SUM(attempts) across channels equals a separate COUNT(*) run with the same four filters and no GROUP BY.
Dropping the amount_minor > 0 filter raises attempts and moves approval_rate_count while leaving approval_rate_value unchanged, which confirms the zero-amount rows were only ever affecting the count-weighted figure.
Both rates are strictly less than 1 unless a channel had no declines at all in the window.

Follow-up

The two rates diverge by four points on the ecommerce channel but agree on card_present. What does that tell you, and what would you cut next?
How would you extend this to all currencies without summing amount_minor across them?
Which of the four decline reasons belong in the denominator of a rate you would put in front of a risk team, and which are really the network's problem?

How do you design metrics to track the success of a new feature in the KNIME platform?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you balance user growth metrics with platform stability?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

What are the primary indicators you would monitor if you noticed a sudden drop in user activity?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

What are the most common experimentation pitfalls you encounter when running A/B tests?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Say whether units interfere with each other, and switch design if they do.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

What is the difference between correlation and causation in the context of A/B testing?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
State the primary metric and the minimum effect worth shipping, then size the test.
Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Test a referral rule when reviewers are a shared queue

Medium · interference · switchback · cluster randomisation

A proposed risk rule raises the share of ecommerce authorizations routed to manual review. Reviewers work a shared pool of queues serving both arms, so extra referrals from treatment lengthen the wait for control cases too. The current plan randomises by customer_id and reads decision latency plus matured fraud basis points. Explain why that design is biased and in which direction, propose a design that is not, and say what governs its power.

Approach

Name the violated assumption precisely: a unit's outcome depends on other units' assignments through the shared reviewer capacity, so the stable unit treatment value assumption fails. Customer-level randomisation then estimates a contrast between a degraded treatment and a degraded control, not between treatment and the status quo.
State the direction. Treatment pushes work into the shared queue, control absorbs part of that wait, so the measured latency difference understates the true effect of full rollout. The test can look acceptable while the rolled-out state is materially worse, which is the expensive failure mode here.
Move randomisation up to a unit that contains the interference. Either cluster-randomise whole queues or sites, or run a switchback that flips the rule for an entire queue over time blocks. Switchback is usually the better choice here because queue count is small and each queue serves as its own control, removing between-queue heterogeneity.
Specify the switchback concretely. Block length should be several times the queue sojourn time; discard a burn-in after each switch equal to the 95th percentile sojourn so carryover cases from the previous regime do not contaminate the new block; balance assignment within each day so the daily arrival pattern cannot correlate with arm.
Analyse at the randomisation unit. Aggregate to queue-block means, include queue and time-block fixed effects, and use randomisation inference or a wild cluster bootstrap rather than cluster-robust standard errors, which are anti-conservative with few clusters. Check residual autocorrelation between adjacent blocks; if it is material, widen the blocks or model it.
Separate the two readouts by maturity. Latency and referral precision at case close are available inside the test window; matured fraud basis points require at least 120 days of dispute maturity from the transaction month, so it is a deferred confirmatory read and must not be presented as a low number on immature cohorts.

Worked solution (30 min)

Write the interference channel explicitly: reviewer capacity is fixed per queue per hour, arrivals are the sum of both arms, so control's wait is a function of treatment's assignment.
Choose a queue-by-four-hour-block switchback across 14 queues for 30 days, giving 14 * 6 * 30 = 2,520 blocks, with within-day balanced assignment and a burn-in equal to the 95th percentile sojourn discarded at each switch.
Compute power at the block level. With a residual block-level standard deviation of decision latency of 6 minutes after removing queue and day-of-week fixed effects, and half the blocks per arm, MDE = 2.8016 * sqrt(2 * 36 / 1260).
State the caveat that adjacent blocks are autocorrelated, so the effective block count is below 2,520 and the analytic MDE is a floor; estimate the inflation from historical block-to-block autocorrelation before committing.
Split the readout: latency and referral precision as the in-window primary, matured fraud basis points as a deferred read at 120 days with immature months marked incomplete rather than plotted.

Expected result: A queue-by-time-block switchback with burn-in exclusion and within-day balanced assignment, analysed on block means with queue and time fixed effects and randomisation inference. With 2,520 blocks and a 6-minute residual block standard deviation, the analytic MDE on decision latency is 2.8016 * sqrt(72 / 1260) = 2.8016 * 0.2390 = 0.67 minutes, stated as a floor pending the autocorrelation correction. Power is governed by block count and between-block residual variance, not by transaction volume.

Verify it yourself

An A/A switchback on historical data should produce block-level differences centred on zero with a standard deviation matching the residual block standard deviation used in the power calculation.
Plot the effect against time since the last switch. If it is not flat after the burn-in window, the burn-in is too short and carryover is still present.
Confirm that arm assignment is orthogonal to hour of day and day of week in the realised schedule, since queue load is strongly periodic.
Cluster count, not row count, should appear in the degrees of freedom of the final inference; if the standard error implies thousands of independent observations, the analysis is at the wrong grain.

Follow-up

How do you set the block length when the queue sojourn time itself changes under treatment?
You have 14 queues and 30 days. Compare a queue-level cluster design against a queue-day switchback on power and on what each can estimate.
The shared resource is capacity. What happens to your estimate if reviewers work faster when the queue is long?

Fraud losses appear to halve in recent transaction months

Medium · right censoring · development factors · maturity

A weekly chart attributes fct_card_dispute cases to the requested_at month of the linked fct_payment_authorization row. The two most recent months show the first-chargeback rate falling by half, and a risk rule shipped six weeks ago. Columns: dispute_id, auth_id, dispute_category, dispute_stage, opened_at, disputed_amount_minor, liability_shift_flag, outcome, net_loss_minor, resolved_at. Decide whether the rule worked, and produce the version of the chart you would sign your name to.

Approach

Separate the two dates explicitly. opened_at is when a case was filed, requested_at is when the transaction happened. Attributing by transaction month is the right causal choice and is exactly what makes the newest months structurally incomplete.
Measure the filing lag rather than assuming it: the distribution of opened_at minus requested_at over fully developed months, split by dispute_category, and the age at which around 95 percent of cases have arrived.
Build a development triangle of transaction month by months of development on cumulative case counts, and estimate age-to-age factors from the columns that are complete.
Develop the immature months with those factors and plot the result as an estimate with a visible band, kept visually distinct from the matured series rather than blended into it.
State the assumption the method needs: a stable development pattern across cohorts. A change in filing behaviour, merchant mix or the dispute team's own backlog breaks it, so inspect factor stability down each column before relying on the estimate.
Only then evaluate the rule, comparing pre-change and post-change cohorts at equal development age.

Follow-up

What leading indicator would you accept while the cohort matures, and what is its known bias?
How does liability_shift_flag change which disputes you should expect to see in the first place?
If the rule also blocked good transactions, where does that cost appear, and is any of it in this chart?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ownership when the result is inconvenient

Sometimes the honest read is that the initiative did not work, and the person who commissioned the analysis was hoping otherwise. Interviewers want to know whether you softened it. Prepare the case where you delivered an unwelcome result, how you presented the uncertainty without hiding behind it, and what the team did next.

Describe a time you had to explain a complex technical concept to a non-technical audience.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Explain an incomplete dispute chart to a non-technical executive

Easy · dispute maturity · stakeholder communication · right-censoring

A finance lead is looking at first-chargeback rate by transaction month, built from fct_card_dispute joined to fct_payment_authorization on auth_id and attributed to requested_at. The last three months slope sharply down and the lead wants to announce a fraud improvement at tomorrow's review. Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, so those months are not complete. In five minutes, with no statistics vocabulary, explain why the decline is not yet evidence and say exactly what you would put on the slide instead.

Approach

Lead with the mechanism in the listener's own terms, not with the statistical name for it: a dispute is attributed to the month the transaction happened, but it can be filed up to roughly 120 days later, so recent months contain only the disputes filed so far.
Show completeness rather than arguing about the rate: for each transaction month, plot the share of its eventual disputes already filed, estimated from months that are fully matured. The last three months will sit visibly below 100 percent.
Replace the chart with two artefacts: a matured series that stops 120 days back and is labelled final, and a development-factor estimate for the immature months drawn as a dashed range and labelled an estimate.
Hand over one sentence the executive can repeat without you in the room: the recent months look better because the disputes have not arrived yet, not because fewer will arrive.
Offer a weekly signal they can watch instead, such as the risk-score mix of approved volume or the decline-rule hit rate, and state up front what it does and does not predict.

Follow-up

The deck ships tomorrow regardless. What exactly goes on the slide, and what wording do you insist on?
How would you estimate the development factors, and how would you notice if they had shifted?

Defend a vintage finding that contradicts the portfolio dashboard

Medium · vintage analysis · mix shift · stakeholder pushback

The lending dashboard shows blended 90-plus days-past-due falling for four consecutive quarters while originations grew 60 percent. Using fct_loan_performance_monthly, you build a vintage view keyed on origination_month by months_on_book and find the three most recent vintages are worse than their predecessors at the same age. The business lead presents that dashboard weekly and pushes back hard, suggesting you picked favourable cohorts. You get one meeting and the vintage table. Present the finding so it survives the cherry-picking objection and ends in a decision.

Approach

Reconcile before you contradict: show that aggregating your vintage table along the calendar diagonal reproduces the published blended series, so the disagreement is about age mix rather than about data quality.
Make the mechanism arithmetic rather than rhetorical: a loan cannot reach 90 days past due before it is 90 days old, so rapid origination growth shifts weight onto young months-on-book where the rate is structurally near zero.
Show every vintage rather than a selected pair, all indexed at months_on_book equal to 12, with cohort sizes printed beside each curve so nobody can claim the divergence rests on a thin cohort.
Handle restructuring explicitly, because restructured_flag resets days_past_due: count each loan on its worst pre-restructure state, or recent vintages will look better than they are.
Close on the decision rather than the chart: state what the divergence implies for the cutoff or the channel mix, and state in advance what evidence would make you withdraw the claim.

Follow-up

Two cohorts differ at month 12. How do you separate a seasoning effect from a genuine credit-quality effect?
Someone argues the recent vintages are simply a broker-channel mix shift. How do you test that, and what would confirm it?