Interview guides · Elevate Credit

Elevate Credit Data Scientist Interview Questions & Guide 2026

Causal inference drills for Elevate Credit data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

The role of a Data Scientist at Elevate Credit is pivotal in driving data-informed decision-making and enhancing product offerings. As a Data Scientist, you'll leverage analytical skills to extract insights from complex datasets, ultimately influencing product development and user experience. Your work will impact not only the internal processes of Elevate Credit but also the financial wellbeing of its users, by developing models and algorithms that optimize lending decisions and risk assessments.

Product-sense cases reward reasoning from a mechanism to a testable prediction. Reciting every metric you can name reads as pattern matching; naming the single quantity that would move if your explanation were true reads as thinking.

Focus areas

Decompose expected loss into PD, LGD, EAD
Separate authorization, settlement and dispute outcomes cleanly
Read vintage curves, not blended portfolio averages

Overview

In this role, you will be part of a collaborative team that addresses strategic challenges within the financial technology landscape. You will engage with various data sources, utilizing advanced statistical methods and machine learning techniques to identify trends and solutions that can lead to innovative financial products. The complexity and scale of problems you will tackle, such as credit scoring and fraud detection, make this position both critical and intellectually stimulating, offering an opportunity to contribute to meaningful advancements in the industry.

The Elevate Credit interview process

Elevate Credit candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Interviews

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Behavioral Interviews

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

Whether you can state the other side's argument accurately before you explain why you disagreed
What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Where candidates lose points

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

SQL that silently fans out on a one-to-many join

State the grain of each table and the grain you want in the result before writing the join. Pre-aggregate the many side to the join key, or use EXISTS or a window function, and verify with a row count against COUNT(DISTINCT id) rather than trusting that the numbers look plausible.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

Can you explain the bias-variance tradeoff?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Translate the result into the decision it informs, in one plain sentence.
Say what the estimate is of, and over what population it generalises.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

What are some common metrics used to evaluate model performance?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Implement accident-quarter loss ratio at twelve months development

Medium · loss ratio · development age · earned premium

fct_policy_period_monthly arrives as a stack of month-end snapshots: each row carries valuation_month alongside as_of_month, policy_id, product_line, written_premium_minor, earned_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor. Compute the accident-quarter loss ratio at exactly 12 months of development: incurred losses over earned premium, both taken from rows whose as_of_month falls in the accident quarter, read from the snapshot 12 months after that quarter closes. Report quarters that cannot reach that age as incomplete rather than dropping them.

Approach

Derive accident_quarter from as_of_month, then define the evaluation snapshot per quarter as valuation_month equal to the quarter's final month plus twelve months. Every figure in the ratio comes from that one snapshot, not from whichever snapshot happens to be newest.
Numerator is paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor over the accident quarter's rows in that snapshot. Loss adjustment expense may be included or not, but the choice applies to every quarter and is named in an output column.
Denominator is earned_premium_minor over the same rows. Written premium is booked in full at inception, so in a growing book it runs ahead of earned premium and drags the ratio down, with the error reversing when the book shrinks.
Left-join the full quarter list against available valuation months so a quarter with no 12-month snapshot yields status incomplete and a null ratio, instead of disappearing and shortening the series without saying so.
Split by product_line, since both the loss ratio level and the speed of development differ by line, and a blended series moves with mix as much as with experience.

Worked solution (30 min)

Add accident_quarter and a target_valuation column equal to the quarter end plus twelve months.
Filter rows to those where valuation_month equals the row's target_valuation, then assert each accident_quarter has exactly one distinct valuation_month left.
Aggregate incurred and earned premium by accident_quarter and product_line and take the ratio.
Reindex against the full list of accident quarters and product lines, marking rows with no matching snapshot as incomplete with a null ratio.
Recompute one quarter by hand on a five-policy subset and confirm it matches.

Expected result: One row per accident_quarter and product_line with incurred, earned_premium, loss_ratio, an lae_included flag and status in complete or incomplete. Complete quarters end twelve months before the latest valuation_month, so the four or five most recent quarters carry null ratios.

Verify it yourself

Assert exactly one distinct valuation_month per accident quarter after filtering; more than one means the snapshot join fanned out.
For a growing book, earned premium in a quarter is below written premium booked in that quarter; if it is above, either the book is shrinking or the accident-quarter derivation is wrong.
Incomplete quarters carry null ratios and never appear on the same line as complete ones in any chart.
Swapping the denominator to written premium shifts a quarter by L(W - E)/(EW), so the sign of the change is the sign of written minus earned premium in that quarter: the ratio falls where the book grew and rises where it shrank. Verify the swap per quarter against sign(W - E) rather than expecting a uniform move; a series that moves in one direction everywhere only means the book grew in every quarter.

Follow-up

The most recent complete quarter came in four points better than the one before. What do you check before calling it an improvement?
How would you estimate the 12-month figure for a quarter that is only 6 months developed, and how would you label the estimate?
Why can an expense ratio legitimately use a different denominator from the loss ratio in the same presentation?

Write a SQL query to find the top 10 customers by transaction volume.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Write a function to calculate the mean and median of a given dataset.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Reconcile captured authorizations against the daily settlement total

Easy · reconciliation · currency · rounding

fct_payment_authorization holds captured_amount_minor in transaction_currency, and settlement_amount_minor in settlement_currency with settlement_fx_rate applied at settlement rather than at authorization. The rate is quoted in major units of settlement_currency per major unit of transaction_currency, and dim_currency.minor_unit_exponent carries the ISO 4217 exponent for each code (0, 2 or 3 depending on the currency). Produce a daily reconciliation: for each settled_at date and settlement_currency, return settled_count, total settlement_amount_minor, and the sum of captured_amount_minor converted into settlement minor units. Flag any date and currency pair whose two totals differ by more than one minor unit per settled authorization. Do not sum amounts across currencies anywhere in the output.

Approach

Restrict to rows that actually settled: settled_at is not null and settlement_amount_minor is not null, which is a smaller population than captured rows because a capture can still be in flight.
Truncate settled_at to a date with an explicit time zone so the cut matches the ledger's cut, since settled_at is timestamptz and date_trunc on timestamptz silently uses the session time zone.
Join dim_currency twice, once on transaction_currency and once on settlement_currency, so both exponents are on the row. Minor units are not a common scale: a bare captured_amount_minor * settlement_fx_rate is correct only when the two exponents are equal, and a zero-decimal currency settling into a two-decimal one is wrong by a factor of 100.
Convert per row as ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle)) — minor units to major in the transaction currency, apply the major-per-major rate, then back to minor units in the settlement currency. The collapsed form ROUND(captured_amount_minor::numeric * settlement_fx_rate * POWER(10::numeric, exp_settle - exp_txn)) is the same expression. Round per row and then sum, not SUM(...) * an average rate, because the rate varies row by row and rounding per row is what the settlement file did.
Group by the settlement date and settlement_currency together, never by date alone, and carry the currency into every output column name or row.
Compare the two totals with a tolerance scaled by settled_count, since per-row rounding accumulates linearly in the number of rows rather than being a fixed constant.

Worked solution (25 min)

Filter to settled rows and derive settlement_date from settled_at with an explicit time zone.
Join dim_currency on transaction_currency and again on settlement_currency to pick up exp_txn and exp_settle; fail the run if either is null rather than defaulting to 2.
Aggregate by settlement_date and settlement_currency: COUNT(*), SUM(settlement_amount_minor), and SUM(ROUND(captured_amount_minor::numeric / POWER(10::numeric, exp_txn) * settlement_fx_rate * POWER(10::numeric, exp_settle))).
Add a derived difference column and a boolean flag where ABS(difference) > settled_count.
Order by the flag first and then by settlement_date so the exceptions surface at the top.

Expected result: One row per settlement_date and settlement_currency, carrying settled_count, settlement_total_minor, converted_capture_total_minor, their signed difference, and a boolean exception flag. On clean days the difference is small relative to settled_count and has no consistent sign. Pairs with unequal exponents reconcile exactly as well as equal-exponent pairs; if they do not, the exponent terms are missing rather than the FX rate being wrong.

Verify it yourself

The difference divided by settled_count stays well under one minor unit on unflagged days, which is the expected size of per-row rounding.
Restricted to rows where exp_txn = exp_settle, the exponent-aware conversion and a bare captured_amount_minor * settlement_fx_rate agree exactly; restricted to rows where the exponents differ by 2, the bare form is smaller by exactly a factor of 100. That is the direct test that the dim_currency join is load-bearing.
Re-running with SUM over all currencies at once produces a number that changes when a zero-decimal currency's volume moves, which demonstrates why the cross-currency total is meaningless.
settled_count summed over all dates equals COUNT(*) on the same settled-row filter with no grouping.

Follow-up

A partial capture means captured_amount_minor is less than amount_minor. Where does that show up in this reconciliation, and where does it not?
On one currency pair the converted total is consistently about one hundredth of the settlement total, on every date, while the other pairs reconcile. Which two columns do you inspect first, and what single change fixes it?
The rate is documented as major-per-major. If a feed started publishing it minor-per-minor instead, which pairs would still reconcile and which would break?
How would you present a total across currencies to a finance partner who has asked for one number?

Analyze this data and propose a strategy for improving user engagement.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Given a dataset, how would you approach extracting actionable insights?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What approach would you take to identify potential fraud patterns in transactions?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize your tasks when working on multiple projects?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you handle missing data in a dataset?

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer would contain.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

A dispute rate that does not fall at the right edge

Medium · right-censoring · development factors · cohort maturity

Your chargeback chart slopes downward over the last four months every time it is refreshed. It is built from fct_card_dispute counted by opened_at, divided by settled transactions in the same month. Using auth_id, requested_at, dispute_stage, opened_at, network_reason_code and outcome, explain what the chart is actually showing, then specify the metric you would publish: attribution, denominator, maturity rule, and how immature periods appear on the chart. State the condition under which you would publish a development-factor estimate instead of a blank.

Approach

Separate the two dates. A dispute belongs to the month of the transaction that caused it, which is requested_at on the linked authorization, not to the month the case was filed. Counting by opened_at mixes old transactions into new months and makes the series unreadable in both directions at once.
Once attribution is fixed, name the censoring. Recent transaction months have not had time to accumulate their disputes, so the right edge falls by construction. The slope will still be there next month, one month further right, which is the diagnostic that distinguishes it from a real improvement.
Set the maturity threshold empirically rather than by folklore. From completed transaction months, compute the share of the final dispute count filed by 30, 60, 90, 120 and 180 days, split by reason-code family. Consumer dispute rights commonly run around 120 days from the transaction or the expected delivery date, but several reason codes run materially longer, so one global cut-off under-covers the slow families.
Specify the published metric: disputes reaching dispute_stage 'first_chargeback' or later, attributed to the transaction month, over settled transactions in that same month, with only months past the measured threshold shown as final.
Decide how immature months appear. Either omit them, or show a chain-ladder style estimate with an interval in visibly different styling, never as a plain point on the same line as final months. Publish the estimate only when the development pattern is stable across recent completed cohorts; if the reason-code mix or the dispute process has shifted, the factors are estimated on a pattern that no longer applies.
Note that the count series and the value series mature at different speeds, since net_loss_minor is signed and known only after the case resolves, so a loss-value chart is stale for longer than a case-count chart of the same data.

Worked solution (30 min)

Join fct_card_dispute to fct_payment_authorization on auth_id, rebuild the series by requested_at month, and plot both versions together.
For transaction months older than twelve months, compute the cumulative share of final disputes filed within 30, 60, 90, 120 and 180 days, split into fraud and non-fraud categories.
Choose the maturity threshold as the age at which cumulative share reaches a stated coverage level, for example 95 percent, for the slowest family you intend to report, and write that level into the definition.
Recompute the last twelve months, marking anything below the threshold as incomplete, and write one line saying what the chart no longer claims.

Expected result: Two series over the same data, one by opened_at and one by transaction month, plus a maturity table of cumulative filing share by age and reason-code family from which a single threshold is chosen and justified, with the most recent months marked incomplete rather than plotted low.

Verify it yourself

Cumulative filing share is non-decreasing in age. If it falls anywhere, cases are being re-dated or re-staged and the stage history log needs checking.
A mature transaction month's rate does not move when the query is re-run a month later, while an immature one rises. If a month you declared mature still moves materially, the threshold is too short.
Fraud-category disputes reach the coverage level at a younger age than delivery-related consumer disputes, so a threshold calibrated on the fast family leaves the right edge still sloping.

Follow-up

Your development factors were estimated over a period containing an authentication policy change. What do you do?
How do you present a month that is 60 days mature to an audience that wants a single number?
Would you attribute a dispute to the authorization month or the settlement month, and when does the difference matter?

A single day of settled volume comes in thirty percent low

Medium · pipeline failure · null rates · data freshness

Yesterday's settled volume is 30 percent below the same weekday across the previous eight weeks, while authorization counts and approved counts for the day look normal. You have fct_payment_authorization with requested_at, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency, created_at and updated_at, plus warehouse load metadata. Decide within the hour whether this is a real drop or a load problem, and say what you would post to the on-call channel in either case.

Approach

Check the data before the business. Compare max(updated_at) and row counts per load partition for the affected day against the prior eight same weekdays; a truncated load usually shows as a count deficit confined to a specific hour range.
Measure null rates per column by requested_at date. Authorization fields present while settled_at and settlement_amount_minor are null points at the settlement feed; a uniform deficit across all columns points at ingestion.
Separate late arrival from loss. Settlement lands after authorization by design, so re-measure the same day 24 and 48 hours later. A deficit that closes on the next load is lag, not a drop.
Reconcile against the independent source: tie the day's settled total to the settlement ledger or acquirer file. If the ledger has the money and the warehouse does not, the business number was never wrong.
Only after those four steps consider a business explanation, and then only if a segment cut produces a coherent story rather than a flat haircut spread evenly across every merchant and channel.
Close by splitting the alert: a freshness and completeness check on the settlement feed, separate from the volume alert, so the next occurrence is classified automatically.

Follow-up

How would you make the daily volume alert immune to settlement lag without also hiding a genuine drop?
What is the right way to restate a published number after a backfill lands?
Which check catches a partial load that is uniform across the day rather than concentrated in one window?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

How do you handle feedback and criticism?

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
Quantify the outcome, including what you would not claim credit for.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Explain an incomplete dispute chart to a non-technical executive

Easy · dispute maturity · stakeholder communication · right-censoring

A finance lead is looking at first-chargeback rate by transaction month, built from fct_card_dispute joined to fct_payment_authorization on auth_id and attributed to requested_at. The last three months slope sharply down and the lead wants to announce a fraud improvement at tomorrow's review. Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, so those months are not complete. In five minutes, with no statistics vocabulary, explain why the decline is not yet evidence and say exactly what you would put on the slide instead.

Approach

Lead with the mechanism in the listener's own terms, not with the statistical name for it: a dispute is attributed to the month the transaction happened, but it can be filed up to roughly 120 days later, so recent months contain only the disputes filed so far.
Show completeness rather than arguing about the rate: for each transaction month, plot the share of its eventual disputes already filed, estimated from months that are fully matured. The last three months will sit visibly below 100 percent.
Replace the chart with two artefacts: a matured series that stops 120 days back and is labelled final, and a development-factor estimate for the immature months drawn as a dashed range and labelled an estimate.
Hand over one sentence the executive can repeat without you in the room: the recent months look better because the disputes have not arrived yet, not because fewer will arrive.
Offer a weekly signal they can watch instead, such as the risk-score mix of approved volume or the decline-rule hit rate, and state up front what it does and does not predict.

Follow-up

The deck ships tomorrow regardless. What exactly goes on the slide, and what wording do you insist on?
How would you estimate the development factors, and how would you notice if they had shifted?

State honestly what your cutoff change actually contributed

Hard · impact attribution · swap set · counterfactual

Six months ago your recommendation moved a credit cutoff, using fct_loan_application and fct_loan_performance_monthly. Since then approval rate rose four points and the 12-month vintage 90-plus rate on affected cohorts is flat. In the same window the bureau changed a score attribute, marketing shifted channel mix toward broker, and the internal funding rate moved. Your performance review asks for impact in currency terms. Give the number you would stand behind, the counterfactual it rests on, and the part of the observed movement you would not claim.

Approach

Define the counterfactual before computing anything: the claim is not what happened after the change, it is what would have happened had the old cutoff scored the same applications, which means replaying the old threshold on the post-change population.
Build the swap set: applications the new cutoff approves that the old one declined, applications the old one approved that the new one declines, and everyone else held out as unaffected. Only the swap groups carry your effect. Note that the swap-out group has no outcome under the new rule, because those loans were never funded, so its forgone margin must be estimated from matched pre-change approvals rather than observed.
Price each swap group at months_on_book equal to 12 on the measure the cutoff was meant to move: interest and fees collected, minus net charge-offs, minus funding cost at the internal transfer rate.
Strip the confounders explicitly. Cohorts affected by the bureau attribute change are either recomputed on the old attribute or excluded; channel mix is held fixed by reweighting to the pre-change mix; funding cost is charged at the rate in force each month rather than one blended average.
State the residual you will not claim, with its size, and give a range rather than a point wherever cohorts have not yet reached 12 months on book.

Follow-up

The swap-in group is only 6 percent of applications. How does that change the way you present the number, and to whom?
What would you have needed to set up at launch to make this attribution clean, and why was a randomised band around the cutoff not used?