Interview guides · Cognistx

Cognistx Data Scientist Interview Questions & Guide 2026

Case practice for Cognistx data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

A Data Scientist at Cognistx plays a pivotal role in shaping data-driven solutions that enhance the company's products and services. This position involves leveraging advanced analytical techniques, machine learning models, and statistical methods to extract insights from complex data sets. As a Data Scientist, your work directly impacts user experiences and informs strategic business decisions, making this role both critical and rewarding.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

Separate authorization, settlement and dispute outcomes cleanly
Decompose expected loss into PD, LGD, EAD
Set fraud thresholds by expected cost

Overview

At Cognistx, you will contribute to innovative projects that span various industries and applications, from improving machine learning algorithms to optimizing user engagement through data insights. You will collaborate with cross-functional teams, including product managers and engineers, to solve challenging problems that drive business value. This role is not only about technical expertise but also about understanding user needs and translating data into actionable insights.

Expect to work in a dynamic environment where your contributions will influence product strategy and success. You will have the opportunity to tackle complex problems that require creativity and analytical rigor, making this role both challenging and fulfilling.

The Cognistx interview process

Cognistx candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interview

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
The counterfactual: what would have been done without your analysis, and why that default was worse
How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Final Interviews

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Where candidates lose points

Counting authorizations instead of weighting them, and summing amounts across currencies

Declines skew toward high-value, cross-border and card-not-present transactions, so an unweighted approval rate can sit flat while approved value falls. Merchant retry logic also turns one declined purchase into several rows, inflating the denominator by an amount that varies by merchant and by decline reason. Amounts are held in the minor unit of the transaction currency and that unit is not always two decimals, since some currencies have none and some have three, so summing amount_minor across currencies produces a figure with no interpretation at all.

Reading the most recent months of fraud and dispute rates as final

Consumer dispute rights commonly run around 120 days from the transaction or expected delivery date, and several reason codes run considerably longer, so the disputes belonging to a recent transaction month have simply not been filed yet. Any chart attributed by transaction date therefore slopes down at the right edge regardless of what is happening. The fix is to report only matured cohorts, or to apply development factors estimated from completed months and to show the estimate as an estimate.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Questions & practice

Can you describe a machine learning project you have worked on and the challenges you faced?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Write a function to implement a logistic regression model from scratch.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

How would you optimize a machine learning model in production?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Estimate a delinquency roll-rate matrix and project twelve months

Hard · roll rates · markov chain · survivorship

fct_loan_performance_monthly gives loan_id, as_of_month_end, months_on_book, delinquency_bucket, charge_off_flag, prepaid_in_full_flag and restructured_flag. Build a month-to-month transition matrix over the five delinquency buckets plus absorbing charged_off and prepaid states. Loans that stop appearing must be routed to an absorbing state rather than dropped. Project the current book forward 12 months by repeated matrix multiplication and report the projected share reaching charge-off. Handle restructured_flag explicitly, and name one place the Markov assumption fails on this data.

Approach

Build consecutive month pairs per loan by shifting as_of_month_end within loan_id, then verify the shifted value is exactly one month later. A gap is not a transition, it is an exit you have not resolved yet.
Resolve exits before counting anything. A loan whose last row carries charge_off_flag moves to charged_off, one carrying prepaid_in_full_flag moves to prepaid, and one that disappears with neither is a data question to raise rather than silently discard, because discarding it is survivorship that inflates every cure rate.
Count pairs into a 7 by 7 matrix and row-normalise. Assert every row sums to one and the two absorbing rows are the identity; a row that does not sum to one means exits were dropped.
Decide and state the restructure rule. Restructuring resets days_past_due, so a dpd_60_89 to current move on a restructured loan is not a cure. Either give restructured loans their own state or carry the pre-restructure bucket, but do not let that move land in the cure cell.
Project by taking the current bucket distribution as a row vector and multiplying by the matrix twelve times. Report the charged_off entry, and report it again from an all-current starting vector so the reader can see how much of the projection comes from loans that are already delinquent today.
State the homogeneity failure plainly: transition rates depend strongly on months_on_book, so one pooled matrix applied to a book with a young mix understates early-life delinquency. If the mix is moving, estimate separate matrices by seasoning band.

Worked solution (45 min)

Sort by loan_id and as_of_month_end, shift to form (from_state, to_state) pairs, and flag pairs whose month gap is not exactly one.
For each loan's final row, assign the absorbing destination from charge_off_flag or prepaid_in_full_flag, and list loans that vanish with neither as an exception count to report.
Apply the restructure rule, then build the 7 by 7 count matrix with a cross-tabulation over ordered state categories and row-normalise it.
Assert row sums equal one and absorbing rows are the identity, then take the current month's bucket distribution as a row vector.
Multiply twelve times, report the charged_off component, and repeat from an all-current vector for comparison.

Expected result: A 7 by 7 row-stochastic matrix with identity rows for charged_off and prepaid, a current-to-current diagonal typically above 0.95, roll rates rising with bucket depth, and a 12-month projected charge-off share of a few percentage points from an all-current start and materially higher from the actual book.

Verify it yourself

Every row sums to one within floating-point tolerance, and both absorbing rows are exactly the identity.
No mass sits in cells that skip a bucket upward, such as dpd_1_29 to dpd_60_89 in one month; if there is, the pairs are not actually consecutive months.
The projection from an all-current start is strictly below the projection from the real book distribution.
Backtest by estimating on months up to t minus 12 and comparing the projected distribution with the realised one at t; a systematic optimistic gap points at dropped exits or at seasoning, not at noise.

Follow-up

How would you validate the projection against what actually happened, and over what window?
The cure rate out of dpd_30_59 rose five points last quarter. What are the candidate explanations and how would you separate them?
When would you prefer a vintage curve to a roll-rate projection, and why?

Collapse merchant retry chains before measuring authorization approval

Medium · window functions · gaps and islands · sessionisation

Merchant retry logic turns one declined purchase into several rows in fct_payment_authorization. Collapse attempts into chains: inside a partition of (card_token_id, merchant_id, amount_minor), consecutive rows whose requested_at gap is 15 minutes or less belong to the same chain. Return one row per chain with chain_started_at, attempt_count, amount_minor, channel, and chain_outcome set to 'approved' when any attempt in the chain was approved. Then report a 7-day rolling dollar-weighted approval rate by channel over chains rather than attempts. Exclude is_reversal rows.

Approach

In a CTE, take LAG(requested_at) OVER (PARTITION BY card_token_id, merchant_id, amount_minor ORDER BY requested_at) and set is_new_chain when the previous value is null or the gap exceeds 15 minutes.
Turn the flag into a chain identifier with SUM(is_new_chain::int) OVER (same partition, same order, rows unbounded preceding to current row), which is the standard gaps-and-islands construction.
Aggregate to one row per chain: MIN(requested_at), COUNT(*), and BOOL_OR(auth_result = 'approved') as the chain outcome, because the purchase succeeded if any attempt in the chain did.
Roll the chains to a daily grain per channel, then apply the rolling window as PARTITION BY channel ORDER BY chain_date RANGE BETWEEN INTERVAL '6 days' PRECEDING AND CURRENT ROW. An interval-offset RANGE frame is value-based: it admits every row in the partition whose chain_date falls in the closed window [D - 6 days, D], so a channel with no chains on some dates still gets the right seven-day sums — sparsity is exactly what this construct tolerates. ROWS BETWEEN 6 PRECEDING AND CURRENT ROW is the one that needs a dense grain, because it counts rows and on a sparse grain reaches back further than seven days.
Left-join a dense date spine only if the output needs a row on days with no chains. That is a presentation requirement, not a correctness one, and it leaves every rolling sum unchanged. The RANGE frame's preconditions are that chain_date is a date or timestamp (an interval offset needs an ordering column it can be added to) and that the aggregation left one row per (channel, chain_date).
Divide approved chain value by total chain value inside the rolling frame, never the average of daily rates, since averaging rates weights a quiet day the same as a heavy one.

Worked solution (40 min)

CTE 1: filter out is_reversal rows, then compute prev_requested_at with LAG and the is_new_chain flag.
CTE 2: build chain_id with a running SUM of the flag over the same partition and order.
CTE 3: group by partition keys plus chain_id to emit chain_started_at, attempt_count, amount_minor, channel and chain_outcome.
CTE 4: aggregate chains to (chain_date, channel) with approved value and total value. Add the date spine here only if the report must show empty days.
Final SELECT: two rolling SUMs over PARTITION BY channel ORDER BY chain_date RANGE BETWEEN INTERVAL '6 days' PRECEDING AND CURRENT ROW, divided at the end.

Expected result: A rolling series with one row per date and channel, where the chain count is strictly below the attempt count on ecommerce and recurring, and the chain-level approval rate is at or above the attempt-level rate for every channel, since collapsing removes failed attempts that later succeeded.

Verify it yourself

SUM(attempt_count) over all chains equals the filtered attempt row count exactly; if it does not, the chain identifier is not partitioned the same way as the LAG.
Every chain has exactly one row whose is_new_chain flag is true, so COUNT(DISTINCT chain_id) equals the number of flagged rows.
A chain whose attempt_count is 1 must have chain_outcome equal to that single row's approved status.
The rolling denominator on day D equals the sum of the daily denominators over every date present between D-6 and D inclusive — seven terms on a dense grain, fewer on a sparse one, and the same total either way.
Adding or removing the date spine leaves every rolling sum identical, because spine rows contribute zero. Swapping RANGE for ROWS changes them on any channel whose grain has a gap, which is the direct test of which construct tolerates sparsity.

Follow-up

This definition chains off the previous attempt, so a chain can span an hour if attempts arrive every 14 minutes. When is that right, and when would you anchor the window to the first attempt instead?
Two genuine purchases of the same amount at the same merchant, four minutes apart, collapse into one chain. How large is that error and how would you bound it?
A channel goes dark for three days mid-window. Which of the two frame types would silently widen, and what would the resulting rate look like?
What changes if the merchant retries with a slightly different amount to dodge a velocity rule?

Accident-quarter loss ratio on earned rather than written premium

Hard · loss ratio · earned vs written · development age

From fct_policy_period_monthly, compute the accident-quarter loss ratio by product_line: incurred losses, being paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor, over earned_premium_minor for the same accident quarter. State explicitly whether loss_adjustment_expense_minor is included and apply that choice consistently. Also output the same ratio computed on written_premium_minor so the two can be compared. The table holds current values with no valuation-date snapshot. Say in one line which comparison this schema cannot support and what you would need to support it.

Approach

Derive the accident quarter from as_of_month with date_trunc, and note that the table already attributes losses to the month of the loss event while earning premium pro rata into the same month, which is what makes the two sides comparable at all.
Aggregate earned_premium_minor, written_premium_minor and the three loss components to product_line and accident quarter in one pass, keeping loss adjustment expense as its own column so the inclusion choice is a final-select decision rather than something buried in a CTE.
Compute both ratios side by side and a third column for their difference, because the size and sign of that difference is a direct read on whether the book grew or shrank in the quarter.
State the limitation plainly: every row carries today's reserve estimate, so each accident quarter is observed at a different development age and a cross-quarter comparison mixes development with underwriting. A fixed development age needs a valuation-date dimension, that is one row per accident period per valuation, which this table does not have.
Guard against the mirror-image error on the numerator by confirming ibnr_reserve_minor is non-zero on recent quarters; if it is null or zero there, the recent periods are understated twice over and the series is not usable.

Follow-up

Written premium exceeds earned premium by 18 percent this quarter and by 3 percent two years ago. What happened to the book, and what does it do to each ratio?
How would you build a development triangle from a valuation-dated version of this table, and what would you use the chain-ladder factors for?
Statutory presentation conventionally takes the expense ratio on written premium while the loss ratio uses earned. How do you avoid a combined ratio that quietly mixes the two bases?

How would you approach a project where the outcome variable is highly imbalanced?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Given a dataset with user engagement metrics, how would you analyze it to improve product features?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize tasks when working on multiple projects?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a data pipeline for a real-time analytics application?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Price a false decline when the label does not exist

Hard · proxy metrics · selection bias · randomised holdout

Your expected-cost threshold needs a figure for what a false decline costs beyond the margin on the blocked transaction. fct_payment_authorization records the decline, nothing records what the customer did next elsewhere, and declined transactions never produce a fraud outcome, so neither side of the error is directly observable. Propose the measurement: the proxy you would build from the tables you have, the design that yields an unbiased estimate for at least part of the score range, the bias in each, and the single sentence you would attach to the number when it reaches a pricing decision.

Approach

Write down what is unobservable and why. The counterfactual spend of a customer who was not declined, and the fraud label on any transaction the rule blocked, are both missing because of the decision itself. Missingness that depends on the decision is not fixed by matching on observed covariates.
Build the observational proxy anyway and be specific. For customers receiving a first risk-rule decline in a window, compare settled volume and active status over the following 30 and 90 days against customers matched on pre-period settled volume, tenure, segment and channel mix who attempted a comparable transaction and were approved. The bias runs one way: matching conditions on having attempted something that scored near the cutoff, and part of the declined group are genuine fraudsters whose disappearance is a saving rather than a loss, so the estimate overstates the damage.
Buy one unbiased local estimate. Hold a small random share of authorizations inside a defined risk_score band out of the decline rule and approve them, sizing the sample in advance from the expected fraud rate in that band so the cost of the experiment is known before it runs. The result is unbiased for that band only, and it is simultaneously the only source of fraud labels in the declined region.
Bound the extrapolation instead of hiding it. Run the holdout in two or three adjacent bands and report the spread. If the effect is flat across bands a constant is defensible; if it is steep, quote band-specific figures and decline to supply a single number.
Handle the window. Attrition after a decline can resolve over months, so a 90-day window truncates it and the randomised estimate is a lower bound on long-run damage at the same time as the observational version is an upper bound. Stating both directions is what makes the number safe to use.
Write the sentence that travels with the number: what it is (an estimate from a randomised holdout in one score band over a 90-day window), what it is not (a measurement anywhere else on the score range), and which way it is likely to be wrong.

Worked solution (40 min)

Define the decline cohort and the matched comparison cohort precisely, including matching variables and the pre-period window, and produce the 30-day and 90-day settled-volume difference.
Decompose the declined cohort into customers who never transact again and customers who transact less, since fraudsters concentrate in the first group and that split tells you how much of the estimate is contamination.
Write the holdout design: score band, sample share, expected fraud rate in band, expected cost of running it, and the run length needed to detect an effect large enough to change the cutoff.
Recompute p* = C_FP / (C_FP + C_FN) at the top and bottom of your estimated cost range and state whether the range changes the cutoff you would set.
Write the one-sentence caveat that will be quoted alongside the number in the pricing decision.

Expected result: Two numbers with opposite biases, an observational estimate labelled as an upper bound and a randomised in-band estimate labelled unbiased for that band only, plus a sensitivity table showing how p* moves across the range and an explicit statement of whether the residual uncertainty changes the decision.

Verify it yourself

If the cost range moves p* by less than the granularity at which the rule can actually be set, further measurement is not worth buying and that is the finding.
The observational estimate should exceed the randomised one in magnitude. If it does not, the matching is capturing something other than the effect of the decline and needs re-examining.
Removing customers with zero prior settled volume from both cohorts should shrink the observational estimate, since new-account fraud concentrates there.

Follow-up

Compliance and finance both object to deliberately approving transactions you believe are fraudulent. What is your answer, and how do you size the holdout?
Your interval spans the decision boundary. What do you recommend?
How would you detect that this number has gone stale?

Approval rate rose in every band yet fell overall

Medium · mix shift · simpsons paradox · segmentation

Monthly application approval rate on fct_loan_application fell from 62 to 57 percent. Cut by bureau_score band, the rate rose in every band, including the null-bureau band. Columns: application_id, channel, submitted_at, requested_amount_minor, declared_annual_income_minor, bureau_score, model_pd_12m, model_version, policy_rule_hits, decision, decided_by, decision_at. The denominator is decision in ('approve','decline'). Explain the arithmetic, quantify how much of the five-point fall is mix versus within-band movement, and say what you would tell the team that owns acquisition.

Approach

Confirm the paradox is real rather than a banding artefact. Rebuild the bands on fixed cutpoints taken from the earlier period, because quantile bands re-cut each month move with the population and can manufacture this pattern on their own.
Compute the exact decomposition rather than describing it: within = sum of w_i0 * (r_i1 - r_i0), mix = sum of r_i0 * (w_i1 - w_i0), interaction = sum of (w_i1 - w_i0) * (r_i1 - r_i0). The three terms sum identically to the change in the blended rate, so the report can state the split.
Attribute the weight change by cutting the same fixed bands by channel. Keep null bureau_score as its own band; a thin file is a population signal, not missing data to be imputed away.
Check whether the new arrivals also changed the population inside a band, by comparing requested_amount_minor and declared_annual_income_minor distributions within one band across the two months.
Deliver two numbers rather than one story: policy is looser in every band, and the funnel is being fed a different population. Those have different owners and different fixes.

Follow-up

If the new channel is profitable at its own approval rate, is the blended fall a problem at all?
How would you present this so that nobody reads the blended series unaccompanied again?
What breaks if you fix the mix by reweighting to a frozen band distribution every month?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ownership when the result is inconvenient

Sometimes the honest read is that the initiative did not work, and the person who commissioned the analysis was hoping otherwise. Interviewers want to know whether you softened it. Prepare the case where you delivered an unwelcome result, how you presented the uncertainty without hiding behind it, and what the team did next.

Describe a time you solved a complex problem using data analysis.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What would you do differently if you ran that project again?
What did you decide not to do, and why?

Can you provide an example of how you communicated complex data findings to a non-technical audience?

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Recommend a decision whose true outcome matures a year later

Hard · decision under censoring · leading indicators · staged rollout

An underwriting rule change must be decided in six weeks. Its real outcome, the vintage 90-plus rate at months_on_book 12 in fct_loan_performance_monthly, matures in a year. The executive wants a yes or no, not a range. Randomising the credit decision across the whole population is not available. Name the leading indicator you would accept, state its bias and the direction of that bias, define the decision rule and stopping condition before any rollout starts, and say what reading would make you recommend reversing the change.

Approach

Fix the readout before the rollout, because a readout chosen after the data arrives is a story rather than a decision rule: indicator, window, threshold and reversal condition all go in writing first.
Choose the leading indicator on its measured relationship to the matured outcome in historical vintages rather than on availability. Early delinquency, typically the share reaching dpd_1_29 or missing a first scheduled payment by months_on_book 3, is the usual candidate, and you quantify how well it predicted the 12-month rate across past cohorts.
State the bias and its direction plainly: early delinquency under-represents default that emerges later and is contaminated by servicing and payment-date effects, so treat it as a floor on risk rather than an estimate of it.
Buy identification where full randomisation is unavailable: a narrow randomised approval band around the cutoff, or a staged rollout by channel or region read as a difference-in-differences, with the parallel-trends assumption stated and checked in the pre-period rather than assumed.
Give the executive the binary they asked for with the trigger attached in the same sentence: yes, conditional on the month-3 indicator staying inside a stated band, with an automatic hold if it breaches.

Follow-up

How would you validate that the month-3 indicator predicts the 12-month outcome, and what evidence would invalidate it mid-rollout?
Compliance refuses a randomised band. What is your next-best identification strategy, and what precision do you lose by taking it?