Interview guides · CodeSignal

CodeSignal Data Scientist Interview Questions & Guide 2026

Case practice for CodeSignal data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

As a Data Scientist at CodeSignal, you sit at the intersection of high-stakes technical assessment and product innovation. Your work is fundamental to the company’s core value proposition: providing a fair, accurate, and scalable way to measure technical talent. You are not just analyzing data; you are refining the very metrics that define how thousands of companies evaluate their engineering candidates.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

Decompose expected loss into PD, LGD, EAD
Set fraud thresholds by expected cost
Separate authorization, settlement and dispute outcomes cleanly

Overview

The role requires a blend of rigorous statistical analysis and product-sense. You will influence the evolution of the CodeSignal assessment platform by designing experiments that test new question formats, diagnosing shifts in user behavior, and ensuring the reliability of CodeSignal's scoring models. You will work closely with product managers and engineering teams to translate complex data into actionable product roadmaps for CodeSignal's technical interviewing platform.

The CodeSignal interview process

CodeSignal candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Deep-Dive Interviews

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Averaging delinquency across a book that is growing

A loan three months old cannot be 90 days past due, so a portfolio with many recent originations reports a low blended 90+ rate purely from age mix. The blended rate falls fastest exactly when originations grow fastest, which is precisely when credit quality most needs watching, so the metric moves in the reassuring direction during the riskiest period. Only comparisons at equal months on book are valid, which is what a vintage or roll-rate view enforces.

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Questions & practice

If you observe a high variance in assessment scores, how do you determine if this is due to test difficulty or population bias?

Medium · statistics and probability

Approach

Write down the assumption the method needs before you use the method.
Say what the estimate is of, and over what population it generalises.
Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

What sample size would you need to detect an effect half this size?
How would you explain this result to someone who does not know statistics?

How would you design an A/B test to validate that a change in our scoring algorithm doesn't negatively impact the candidate experience?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Measure calibration of a twelve-month default probability from scratch

Hard · calibration · brier score · binning

fct_loan_application gives application_id, model_pd_12m, model_version, decision, funded_at and loan_id. fct_loan_performance_monthly gives loan_id, months_on_book, days_past_due and charge_off_flag. Define the outcome as ever 90 or more days past due, or charged off, by months_on_book = 12. Without sklearn or scipy, build an equal-count binned reliability table, the expected calibration error, the Brier score and its reliability, resolution and uncertainty components, and report the residual the binned identity leaves behind. Restrict to cohorts that have actually reached 12 months on book.

Approach

Build the label first and name the population it covers out loud: only funded loans have outcomes, so this measures calibration on the approved population. The declined region is unmeasured, and no binning scheme repairs that.
Restrict to applications whose loans have reached months_on_book = 12. A cohort observed at 8 months has a mechanically lower default rate and will read as systematic over-prediction that is really just immaturity.
Bin by equal count, deciles of model_pd_12m through a rank-based cut, not equal width. The PD distribution is heavily right-skewed, so equal-width bins put most of the mass in the first bin and leave the risky bins with single-digit counts whose observed rates mean nothing.
Per bin compute n, mean predicted, observed rate, and the binomial standard error sqrt(o(1-o)/n) so a gap can be read against noise. ECE is the count-weighted mean absolute gap between mean predicted and observed.
Compute Brier directly as the mean squared error, then reliability = sum of n_k (pbar_k - obar_k)^2 over N, resolution = sum of n_k (obar_k - obar)^2 over N, uncertainty = obar(1 - obar). Report residual = Brier - (reliability - resolution + uncertainty). That identity is exact only for discrete forecasts, so with binned continuous scores the residual is the within-bin spread of the score; a large one means the bins are too wide to support the decomposition.
Split by model_version. A mixed-version population can look well calibrated in aggregate while each version is biased in opposite directions.

Worked solution (45 min)

Reduce fct_loan_performance_monthly to one row per loan_id with the maximum days_past_due and any charge_off_flag over months_on_book 0 to 12, plus the maximum months_on_book observed, and keep only loans reaching 12.
Inner-join to approved and funded applications, and record how many approved applications were dropped for immaturity and how many decisions were declines that never enter the measurement at all.
Assign deciles with a rank-based cut on model_pd_12m, then aggregate n, mean predicted, observed rate and standard error per bin.
Compute ECE, Brier, reliability, resolution, uncertainty and the residual, and print all six.
Repeat the whole computation split by model_version and compare the per-version reliability against the pooled figure.

Expected result: A ten-row reliability table with n, mean predicted, observed and standard error; an ECE of a few tenths of a percentage point to a couple of points; a Brier score close to the uncertainty term, because a rare-event model has little resolution to subtract; and a residual small relative to the reliability term.

Verify it yourself

Bin counts sum to the filtered row count, and the count-weighted mean of bin observed rates equals the overall observed rate.
Uncertainty equals obar(1 - obar) computed directly, and reliability is zero on a synthetic input where observed is set equal to predicted in every bin.
The residual shrinks as the bin count rises from 5 to 20; if it does not, the label or the binning is wrong.
Multiplying every PD by 1.5 must raise reliability and ECE while leaving resolution, uncertainty and AUC unchanged, since equal-count bins are invariant to a monotone transform.

Follow-up

AUC is unchanged after a population shift but the reliability curve has moved. What happened, and what do you do about it?
How would you recalibrate without retraining, and what would you check afterwards?
The top decile shows observed default well above predicted. Is that a calibration problem or a policy problem?

How would you identify "power users" in our database using self-joins or CTEs?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Write a query using SQL window functions to calculate the rolling 7-day average of assessment completions per user.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Given a table of event logs, how would you isolate the sequence of actions that leads to a candidate abandoning an assessment?

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Customers with no credit application, avoiding the NOT IN trap

Medium · null semantics · anti-join · scd type 2

Count current customers who have never submitted a credit application, broken out by segment. dim_customer is a slowly changing dimension type 2, so restrict to is_current = true, kyc_status = 'verified' and closed_at null. In fct_loan_application, customer_id is null for applicants who were not customers when they applied. Write the anti-join, return segment and customer_count, and state in one line what NOT IN (SELECT customer_id FROM fct_loan_application) returns against this table and why.

Approach

Pin the dimension to one row per customer first: is_current = true already guarantees that, but say so out loud, because forgetting it multiplies every count by the number of attribute versions a customer has accumulated.
Write the anti-join as NOT EXISTS with a correlated predicate on customer_id, which evaluates per row and is unaffected by nulls anywhere in the applications table.
Name the failure explicitly: NOT IN against a nullable column compares each candidate to a set containing NULL, the comparison yields UNKNOWN rather than TRUE, and the whole predicate is therefore never satisfied, so the query returns zero rows.
If NOT IN is required for some reason, add WHERE customer_id IS NOT NULL inside the subquery, which restores the intended semantics, and note that a LEFT JOIN with an IS NULL filter is equally safe.
Group by segment and sanity-check the total against the unfiltered current-customer count minus the count of distinct applying customers.

Worked solution (20 min)

SELECT segment, COUNT(*) FROM dim_customer c WHERE c.is_current AND c.kyc_status = 'verified' AND c.closed_at IS NULL.
Add AND NOT EXISTS (SELECT 1 FROM fct_loan_application a WHERE a.customer_id = c.customer_id).
Group by segment and order by the count descending.
Run the NOT IN variant alongside it and record that it returns zero rows, then run it again with IS NOT NULL added to the subquery and confirm the counts match the NOT EXISTS version.

Expected result: One row per segment present in the current dimension (retail, sme, corporate), each with a non-zero customer_count. The NOT EXISTS and the null-guarded NOT IN forms agree exactly; the unguarded NOT IN form returns no rows at all.

Verify it yourself

COUNT(DISTINCT customer_id) equals COUNT(*) on the filtered dimension, which confirms is_current really collapsed the type 2 versions.
The total across segments plus COUNT(DISTINCT customer_id) from fct_loan_application restricted to current verified open customers equals the full current verified open customer count.
Swapping NOT EXISTS for the unguarded NOT IN changes the result from a populated set to an empty one, which is the whole point of the exercise.

Follow-up

Rewrite it as a LEFT JOIN with IS NULL and say when you would prefer that form to NOT EXISTS.
How does the answer change if you want customers who never applied as of a historical date rather than today?
The applications table has 40,000 rows with a null customer_id. What are those rows, and are they a data quality problem or a product fact?

What metrics would you track to ensure that our platform remains fair and unbiased for all candidates?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you decide between prioritizing a feature that increases candidate volume versus one that improves assessment accuracy?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you measure the success of a new feature designed to reduce candidate anxiety during a live coding assessment?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

We are testing a new UI for the coding interface. How do you determine if the difference in completion rates is statistically significant?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Name the guardrails that would stop a launch even on a positive primary result.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

What are the most common experimentation pitfalls you have encountered, and how do you mitigate them?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

How would you handle interference between treated and control units?
What would you conclude if the result is positive but the test is underpowered?

If we notice a sudden 10% drop in completion rates for our Python assessments, how would you investigate the cause?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Test a referral rule when reviewers are a shared queue

Medium · interference · switchback · cluster randomisation

A proposed risk rule raises the share of ecommerce authorizations routed to manual review. Reviewers work a shared pool of queues serving both arms, so extra referrals from treatment lengthen the wait for control cases too. The current plan randomises by customer_id and reads decision latency plus matured fraud basis points. Explain why that design is biased and in which direction, propose a design that is not, and say what governs its power.

Approach

Name the violated assumption precisely: a unit's outcome depends on other units' assignments through the shared reviewer capacity, so the stable unit treatment value assumption fails. Customer-level randomisation then estimates a contrast between a degraded treatment and a degraded control, not between treatment and the status quo.
State the direction. Treatment pushes work into the shared queue, control absorbs part of that wait, so the measured latency difference understates the true effect of full rollout. The test can look acceptable while the rolled-out state is materially worse, which is the expensive failure mode here.
Move randomisation up to a unit that contains the interference. Either cluster-randomise whole queues or sites, or run a switchback that flips the rule for an entire queue over time blocks. Switchback is usually the better choice here because queue count is small and each queue serves as its own control, removing between-queue heterogeneity.
Specify the switchback concretely. Block length should be several times the queue sojourn time; discard a burn-in after each switch equal to the 95th percentile sojourn so carryover cases from the previous regime do not contaminate the new block; balance assignment within each day so the daily arrival pattern cannot correlate with arm.
Analyse at the randomisation unit. Aggregate to queue-block means, include queue and time-block fixed effects, and use randomisation inference or a wild cluster bootstrap rather than cluster-robust standard errors, which are anti-conservative with few clusters. Check residual autocorrelation between adjacent blocks; if it is material, widen the blocks or model it.
Separate the two readouts by maturity. Latency and referral precision at case close are available inside the test window; matured fraud basis points require at least 120 days of dispute maturity from the transaction month, so it is a deferred confirmatory read and must not be presented as a low number on immature cohorts.

Worked solution (30 min)

Write the interference channel explicitly: reviewer capacity is fixed per queue per hour, arrivals are the sum of both arms, so control's wait is a function of treatment's assignment.
Choose a queue-by-four-hour-block switchback across 14 queues for 30 days, giving 14 * 6 * 30 = 2,520 blocks, with within-day balanced assignment and a burn-in equal to the 95th percentile sojourn discarded at each switch.
Compute power at the block level. With a residual block-level standard deviation of decision latency of 6 minutes after removing queue and day-of-week fixed effects, and half the blocks per arm, MDE = 2.8016 * sqrt(2 * 36 / 1260).
State the caveat that adjacent blocks are autocorrelated, so the effective block count is below 2,520 and the analytic MDE is a floor; estimate the inflation from historical block-to-block autocorrelation before committing.
Split the readout: latency and referral precision as the in-window primary, matured fraud basis points as a deferred read at 120 days with immature months marked incomplete rather than plotted.

Expected result: A queue-by-time-block switchback with burn-in exclusion and within-day balanced assignment, analysed on block means with queue and time fixed effects and randomisation inference. With 2,520 blocks and a 6-minute residual block standard deviation, the analytic MDE on decision latency is 2.8016 * sqrt(72 / 1260) = 2.8016 * 0.2390 = 0.67 minutes, stated as a floor pending the autocorrelation correction. Power is governed by block count and between-block residual variance, not by transaction volume.

Verify it yourself

An A/A switchback on historical data should produce block-level differences centred on zero with a standard deviation matching the residual block standard deviation used in the power calculation.
Plot the effect against time since the last switch. If it is not flat after the burn-in window, the burn-in is too short and carryover is still present.
Confirm that arm assignment is orthogonal to hour of day and day of week in the realised schedule, since queue load is strongly periodic.
Cluster count, not row count, should appear in the degrees of freedom of the final inference; if the standard error implies thousands of independent observations, the analysis is at the wrong grain.

Follow-up

How do you set the block length when the queue sojourn time itself changes under treatment?
You have 14 queues and 30 days. Compare a queue-level cluster design against a queue-day switchback on power and on what each can estimate.
The shared resource is capacity. What happens to your estimate if reviewers work faster when the queue is long?

Approval rate fell but approved value did not

Easy · approval rate · retry chains · denominators

Over ten days the count-weighted 7-day approval rate on fct_payment_authorization fell from 91 to 86 percent, while captured value in the reporting currency is flat. Available columns: auth_id, card_token_id, merchant_id, mcc, channel, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, risk_score, is_reversal, parent_auth_id, captured_at, captured_amount_minor, issuer_country. In twenty minutes, decide whether approved value is actually at risk, and hand back a corrected rate together with its denominator and every exclusion written down.

Approach

Restate the denominator before querying anything. The current one counts every row, so reversals (is_reversal = true), incremental authorizations (parent_auth_id not null) and zero-amount verification attempts (amount_minor = 0) are all sitting in it.
Chart numerator and denominator separately by day. If approved counts are flat and total attempts rose, the rate moved because the denominator grew, which is a different investigation from a rule change and points at a different owner.
Group declines by decline_reason_code and merchant_id. Retry-driven inflation concentrates in a few soft decline codes at a few merchants; a genuine policy change spreads across merchants within one code family.
Collapse retry chains: partition by (card_token_id, merchant_id, amount_minor), keep one attempt per 15-minute window taking the best outcome, and recompute both count-weighted and dollar-weighted rates on the collapsed set.
Convert amount_minor to one reporting currency using each currency's ISO 4217 exponent before any dollar weighting, then cut by channel and issuer_country to confirm nothing is hiding underneath a flat total.
Report both rates side by side with the exclusion list attached, and state which definition the alert should have been built on.

Follow-up

How would you pick the retry-collapsing window when merchants retry on different schedules?
Flat captured value could itself be hiding a mix shift. How do you rule that out?
What monitor would have caught denominator inflation on the day it started, rather than ten days later?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Tell me about a time you identified a flaw in a metric you were tracking. How did you fix it?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Recommend a decision whose true outcome matures a year later

Hard · decision under censoring · leading indicators · staged rollout

An underwriting rule change must be decided in six weeks. Its real outcome, the vintage 90-plus rate at months_on_book 12 in fct_loan_performance_monthly, matures in a year. The executive wants a yes or no, not a range. Randomising the credit decision across the whole population is not available. Name the leading indicator you would accept, state its bias and the direction of that bias, define the decision rule and stopping condition before any rollout starts, and say what reading would make you recommend reversing the change.

Approach

Fix the readout before the rollout, because a readout chosen after the data arrives is a story rather than a decision rule: indicator, window, threshold and reversal condition all go in writing first.
Choose the leading indicator on its measured relationship to the matured outcome in historical vintages rather than on availability. Early delinquency, typically the share reaching dpd_1_29 or missing a first scheduled payment by months_on_book 3, is the usual candidate, and you quantify how well it predicted the 12-month rate across past cohorts.
State the bias and its direction plainly: early delinquency under-represents default that emerges later and is contaminated by servicing and payment-date effects, so treat it as a floor on risk rather than an estimate of it.
Buy identification where full randomisation is unavailable: a narrow randomised approval band around the cutoff, or a staged rollout by channel or region read as a difference-in-differences, with the parallel-trends assumption stated and checked in the pre-period rather than assumed.
Give the executive the binary they asked for with the trigger attached in the same sentence: yes, conditional on the month-3 indicator staying inside a stated band, with an automatic hold if it breaches.

Follow-up

How would you validate that the month-3 indicator predicts the 12-month outcome, and what evidence would invalidate it mid-rollout?
Compliance refuses a randomised band. What is your next-best identification strategy, and what precision do you lose by taking it?

Disagree with a product manager over an approval-rate target

Medium · metric design · denominators · influence without authority

A product manager proposes a quarterly goal of raising card authorization approval rate by 150 basis points, measured as approved authorizations divided by all authorizations in fct_payment_authorization. You believe that metric can be hit with no customer benefit, because merchant retry chains, zero-amount verification authorizations, incremental authorizations and reversals all sit in the denominator, and declines skew toward high-value cross-border ecommerce. You support the underlying goal. In one working session, change the metric without killing the initiative, and name the guardrail you would accept.

Approach

Separate the goal from the metric out loud and agree with the goal first, so the disagreement stays narrow and technical rather than becoming positional.
Demonstrate the failure rather than asserting it: compute the proposed metric and the dollar-weighted collapsed version over the same recent window, and find a period where they moved in opposite directions.
Propose the replacement precisely: sum of approved amount_minor over sum of attempted amount_minor, after collapsing retries to one attempt per card_token_id, merchant_id and amount_minor within a 15-minute window, excluding is_reversal rows and zero-amount verifications, with everything converted to one reporting currency before summing.
Attach the guardrail that makes the target honest: matured first-chargeback rate and net fraud loss in basis points of settled volume, read only on transaction months carrying at least 120 days of maturity.
Give the product manager something back: the replacement metric cuts cleanly by channel and issuer_country, which makes a roadmap of merchant-specific and authentication fixes legible in a way the blended rate never was.

Follow-up

How do you identify a retry chain when the merchant varies the amount slightly between attempts?
The product manager wants a weekly read on the guardrail. What is the earliest defensible signal, and how do you label it?