Interview guides · American Equity (AEL)

American Equity (AEL) Data Scientist Interview Questions & Guide 2026

Data scientist prep for American Equity (AEL): practice defining a payments and lending metric, picking its guardrails and defending the tradeoff.

The 60-second brief

The Data Scientist II role at American Equity (AEL) is a vital position within the Data Science team that drives the organization’s mission to empower clients in their retirement planning. This role combines advanced analytical skills with innovative problem-solving to develop data-driven solutions that support business initiatives across the company. By leveraging large datasets and applying machine learning techniques, you will create predictive models that inform strategic decisions, directly impacting the quality of products and services AEL offers to its clients.

Coding rounds for this role are usually data-manipulation shaped rather than data-structure shaped: group-bys, joins, time windows, ranking within a partition. Confirm the format before spending a week on graph traversal.

Focus areas

Read vintage curves, not blended portfolio averages
Set fraud thresholds by expected cost
Report only matured cohorts for loss metrics

Overview

In your role, you will collaborate with business partners to understand technical problem statements, develop efficient analytical models, and integrate these solutions into business processes. The complexity and scale of the data you will work with make this position not just challenging, but also crucial for the company’s success in an increasingly data-driven market. You will contribute significantly to the development of cutting-edge annuity products and other financial instruments that enhance the lives of retirees across the nation.

The American Equity (AEL) interview process

American Equity (AEL) candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Behavioral Interviews

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
The counterfactual: what would have been done without your analysis, and why that default was worse
How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Where candidates lose points

Assuming a model is fair because protected attributes are not among its inputs

Postcode, device, tenure, income proxies and even transaction patterns correlate with protected characteristics, so a model can produce a disparate outcome without ever reading the attribute. Credit decisions additionally carry an explainability obligation in many jurisdictions, since a denial has to be accompanied by its principal reasons, which constrains model form and feature engineering rather than being a reporting afterthought. Treating fairness testing and reason-code generation as design constraints from the first model version is far cheaper than retrofitting them to a deployed one.

Using written premium as the denominator of a loss ratio

Premium is written at inception and earned pro rata across the exposure period, so in a growing book written premium runs ahead of earned premium and the loss ratio comes out too low, with the error reversing when the book shrinks. The numerator has the mirror-image problem if it omits incurred-but-not-reported reserves, since recent accident periods then look profitable twice over. Both sides must refer to the same exposure period, which is what an accident-period view at a fixed development age enforces.

Ignoring interference between units in a marketplace experiment

Ask whether one unit's treatment can change another unit's outcome through shared inventory, a matching pool, a social graph or a common budget. Where it can, randomise at a level that contains the spillover, such as region or time slice, and say explicitly what that costs you in statistical power.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Questions & practice

Describe a project where you used machine learning to solve a business problem.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Check what information would not exist at prediction time, and exclude it.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

How would you approach feature selection for a predictive model?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

What are the key considerations when deploying a machine learning model into production?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Collapse retry chains and compute a dollar-weighted approval rate

Medium · sessionisation · window functions · dollar-weighted rates

fct_payment_authorization gives auth_id, card_token_id, merchant_id, amount_minor, transaction_currency, requested_at, auth_result, is_reversal, channel and issuer_country. Two reference frames give the minor-unit exponent per currency and a daily rate to one reporting currency. Collapse retry chains first: attempts sharing card_token_id, merchant_id and amount_minor whose consecutive gaps are under 15 minutes form a single attempt, whose outcome is its last row. Exclude reversals and zero-amount verifications. Return a 7-day rolling dollar-weighted approval rate by channel and issuer_country.

Approach

Filter before grouping: drop is_reversal rows and zero-amount verifications, since neither is a purchase attempt and both would otherwise sit in the denominator.
Sort by card_token_id, merchant_id, amount_minor and requested_at, take the gap to the previous row within that key, mark a chain start where the gap exceeds 15 minutes or the key changes, and label chains with a cumulative sum of that flag. This is a gap rule between consecutive attempts, not a fixed clock bucket, so a chain may span more than 15 minutes in total.
Keep each chain's terminal row by requested_at. If a retry was approved, the purchase was approved; keeping the first row reports the decline that caused the retry as the outcome.
Convert amounts exactly once: amount_minor divided by 10 to the power of the currency exponent, multiplied by the reference rate for the authorization date. Do not reach for settlement_fx_rate, which is null on precisely the declined rows the denominator needs.
Build the rolling window as a ratio of two rolling sums, approved value over total value, per channel and issuer_country. A rolling mean of daily ratios weights a quiet Sunday the same as a busy Friday.

Worked solution (35 min)

Filter out reversals and zero-amount rows, then sort by the chain key and requested_at.
Compute the within-key time difference, derive the chain start flag and the chain id, and take the last row per chain with groupby(chain_id).tail(1) after sorting.
Join the exponent and daily rate tables, compute value_reporting, and assert no nulls remain after the join.
Aggregate approved value and total value to a daily grain by channel and issuer_country, reindex to a complete date range per group so missing days are zero rather than absent.
Take 7-day rolling sums of both columns and divide, then confirm one hand-picked group-day against a direct filter.

Expected result: A DataFrame keyed by date, channel and issuer_country with approved_value, total_value and approval_rate. The collapsed attempt count is materially below the raw row count, with the gap concentrated in declined ecommerce rows, and ecommerce sits below card_present.

Verify it yourself

Within every chain, all consecutive requested_at gaps are under 15 minutes; across chains with the same key, at least one gap is not.
Chain count is strictly below raw attempt count, and the difference is concentrated in declined rows rather than spread evenly.
Total converted value on a single-currency single-day subset equals the raw amount_minor sum divided by that currency's exponent power, times that day's rate.
The rolling rate on one chosen day matches a direct filter-and-divide over the preceding seven days.

Follow-up

The count-weighted rate is flat while the dollar-weighted rate falls 80 basis points. What do you look at first?
How would you choose the 15-minute window rather than inheriting it?
A merchant moves from two retries to five. Which of your two rates moves, and is that a real change in approval quality?

Monthly delinquency roll rates with honest exiting-account denominators

Hard · roll rates · window functions · self joins

From fct_loan_performance_monthly, compute month-over-month roll rates: for each as_of_month_end and delinquency_bucket, the share of loans in that bucket that are in a worse state at the following month end. Use this disposition convention and apply it without exception: charge-off is the terminal state one rank past dpd_90_plus, so a loan that charges off has rolled forward, not exited; loans that leave the book any other way — prepaid in full, sold, matured, or simply absent at the next month end — are exited. Bucket order is current, dpd_1_29, dpd_30_59, dpd_60_89, dpd_90_plus, then charged_off. Return from_month, from_bucket, accounts, rolled_forward, cured, stayed, exited and roll_rate, and show that the four dispositions sum to accounts on every row.

Approach

Map the five delinquency_bucket values to ranks 0 through 4 in a small inline VALUES list, and reserve rank 5 for charged_off, because 'worse' is an ordering statement and enum text does not sort that way reliably. Read charge-off from charge_off_flag on the next month's row rather than from delinquency_bucket, which keeps carrying whatever delinquency value the loan held when it was written off.
Use LEAD(as_of_month_end), LEAD(bucket_rank) and LEAD(charge_off_flag) OVER (PARTITION BY loan_id ORDER BY as_of_month_end), then verify the leading month end is exactly one month after the current one, since a loan missing a month would otherwise appear to cure or roll across a gap it never traversed.
Classify each loan-month into exactly one of four labels off the adjacent next row, whose rank is 5 when its charge_off_flag is true and its mapped bucket rank otherwise. rolled_forward: the next rank is strictly greater. cured: strictly lower. stayed: equal. exited: there is no adjacent next row at all, which is where prepayment, sale, maturity and a gap in the monthly history land.
Define rolled_forward as 'any strictly worse rank', not 'rank plus one'. Under normal monthly aging the only reachable worse state is the next bucket, so the two definitions coincide — except at charge-off, where a loan can go from dpd_60_89 straight to rank 5. A strict plus-one test leaves those loan-months matching no branch and breaks the exhaustiveness invariant.
Be explicit about why charge-off is a roll rather than an exit: routed to exited, dpd_90_plus has no worse state left, so its roll_rate is identically zero by construction — arithmetic, not a credit finding. At rank 5 the dpd_90_plus row reports roll-to-loss, which is the number a loss forecast actually consumes.
Aggregate by from_month and from_bucket and require that rolled_forward + cured + stayed + exited equals accounts on every row, which is the invariant that proves the classification is exhaustive and mutually exclusive.

Worked solution (45 min)

CTE ranks: an inline VALUES mapping of the five delinquency_bucket values to ranks 0 through 4, joined onto the monthly rows. Rank 5 is not in the map; it is assigned during classification.
CTE leads: add LEAD(as_of_month_end), LEAD(bucket_rank) and LEAD(charge_off_flag) partitioned by loan_id ordered by as_of_month_end.
CTE classified: derive next_rank as CASE WHEN next_charge_off_flag THEN 5 ELSE next_bucket_rank END, then a CASE expression producing exactly one label per loan-month — exited when the next month end is null or is not the current month end plus one month, otherwise rolled_forward / cured / stayed by comparing next_rank to bucket_rank.
Aggregate to from_month and from_bucket with COUNT(*) and four FILTER counts, then compute roll_rate as rolled_forward over accounts with a numeric cast.
Assert the exhaustiveness invariant in a final HAVING or a separate verification query before trusting any number.

Expected result: One row per month and originating bucket, with the four disposition counts summing to accounts on every row and roll_rate between 0 and 1. Roll rates climb across current, dpd_1_29, dpd_30_59 and dpd_60_89, because each successive bucket holds loans that have already failed to cure. Do not expect that climb to continue into dpd_90_plus: that bucket is a catch-all holding several monthly cohorts at once until the charge-off policy retires them, so its roll-to-loss rate is set by the policy horizon — under a 180-day policy roughly a third of the bucket rolls in a given month — and usually sits below the dpd_60_89 rate. Cure out of dpd_90_plus is well under ten percent.

Verify it yourself

rolled_forward + cured + stayed + exited equals accounts for every row, with no exceptions.
SUM(accounts) for a given from_month equals the count of loan rows at that month end, so no loan-month was dropped or counted twice.
Removing the adjacency test changes the cure and roll counts, and the size of that change is exactly the number of loans with a gap in their monthly history.
The rolled_forward count for from_bucket dpd_90_plus in month M is at least the number of loans whose first charge_off_flag = true row falls at month end M+1. If it is materially lower, charge-off is being routed into exited.
roll_rate for dpd_90_plus is strictly greater than zero in any month with charge-offs. A flat zero on that bucket across every month is the signature of charge-off classified as an exit, not of an unusually stable book.
No loan-month is counted as both rolled_forward and exited: a loan with a charged-off row at M+1 has an adjacent next row, so it can never reach the exited branch.

Follow-up

Project the next three months of dpd_90_plus inflow from these roll rates. What assumption does that projection make, and when does it break?
A restructure resets days_past_due to zero. What does that do to the cure rate out of dpd_60_89, and how would you separate a real cure from a reset?
Argue the other convention: prepayment in exited but charge-off also in exited, with roll rates reported only for the four non-terminal buckets. What does that series answer better, and what does it lose?
The charge-off policy changed from 180 to 120 days. Show where that appears in this table and what it does to the dpd_90_plus roll rate specifically.

Customers with no credit application, avoiding the NOT IN trap

Medium · null semantics · anti-join · scd type 2

Count current customers who have never submitted a credit application, broken out by segment. dim_customer is a slowly changing dimension type 2, so restrict to is_current = true, kyc_status = 'verified' and closed_at null. In fct_loan_application, customer_id is null for applicants who were not customers when they applied. Write the anti-join, return segment and customer_count, and state in one line what NOT IN (SELECT customer_id FROM fct_loan_application) returns against this table and why.

Approach

Pin the dimension to one row per customer first: is_current = true already guarantees that, but say so out loud, because forgetting it multiplies every count by the number of attribute versions a customer has accumulated.
Write the anti-join as NOT EXISTS with a correlated predicate on customer_id, which evaluates per row and is unaffected by nulls anywhere in the applications table.
Name the failure explicitly: NOT IN against a nullable column compares each candidate to a set containing NULL, the comparison yields UNKNOWN rather than TRUE, and the whole predicate is therefore never satisfied, so the query returns zero rows.
If NOT IN is required for some reason, add WHERE customer_id IS NOT NULL inside the subquery, which restores the intended semantics, and note that a LEFT JOIN with an IS NULL filter is equally safe.
Group by segment and sanity-check the total against the unfiltered current-customer count minus the count of distinct applying customers.

Follow-up

Rewrite it as a LEFT JOIN with IS NULL and say when you would prefer that form to NOT EXISTS.
How does the answer change if you want customers who never applied as of a historical date rather than today?
The applications table has 40,000 rows with a null customer_id. What are those rows, and are they a data quality problem or a product fact?

Given a dataset, how would you identify trends and insights that could drive business decisions?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

If you were given a new dataset, what would be your first steps in analysis?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Explain how you would handle missing data in a dataset.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer would contain.
Work from the decision backwards to the evidence you would need.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Define an active customer for a card portfolio

Easy · denominator design · engagement · scd2

Leadership wants one weekly number for how many customers are active. You have fct_payment_authorization (auth_id, customer_id, auth_result, captured_at, amount_minor, is_reversal, channel) and dim_customer (customer_id, is_current, kyc_status, onboarded_at, closed_at). Write the metric definition you would publish: numerator, denominator, window, and every exclusion with its reason. Then name two ways the definition gives a wrong read, one that inflates it and one that deflates it, and say what you would report alongside it.

Approach

Fix the grain before writing anything. The number counts distinct customers, so the authorization table has to be reduced to one row per customer_id first; a customer with forty approvals counts once.
Build the numerator from money that actually moved: auth_result = 'approved' AND captured_at IS NOT NULL, excluding is_reversal = true and zero-amount verification authorizations. An approval that is never captured, or is reversed, has no economic content behind it.
Build the denominator from the population that could have transacted: dim_customer with is_current = true, kyc_status = 'verified', onboarded_at on or before the window start, and closed_at null or after the window start. Joining without is_current fans the type 2 history out and multiplies the denominator by the number of attribute versions.
State the window and its recompute cadence, trailing 30 days recomputed daily, and say plainly that this series is not comparable to a calendar-month version of the same definition.
Name the inflation path (recurring and subscription-only customers, and wallet top-ups, count as active with no user intent behind them) and the deflation path (a customer onboarded inside the window had less than 30 days of opportunity, and activity on products outside this table is invisible).
Pair it with a depth metric such as settled volume per active customer, so breadth cannot be reported on its own and an acquisition push cannot pass as engagement.

Worked solution (20 min)

Write the denominator query first and record both the raw dim_customer row count and the distinct customer_id count; their ratio is the fan-out factor the is_current filter removes.
Write the numerator as a distinct customer_id count over the trailing 30 days, applying the three exclusions one at a time and recording the count after each.
Compute the rate, then recompute with zero-amount verifications left in, and record the difference in basis points.
Write the two failure modes as one sentence each, naming the customer population affected by each.

Expected result: An eight-line definition: distinct customer_id with an approved, captured, non-reversed, non-zero-amount authorization in the trailing 30 days, over is_current, KYC-verified, open customers onboarded on or before the window start; accompanied by an exclusion log giving the customer count removed by each exclusion.

Verify it yourself

The numerator count never exceeds the denominator count. If it does, the numerator contains customers who are absent from dim_customer or whose only row is not the current version.
Total dim_customer rows divided by distinct customer_id is greater than one, which is the fan-out the is_current filter prevents.
Dropping the captured_at filter raises the numerator, and the increase concentrates in specific channels such as holds and verifications rather than spreading evenly. An even spread means the filter is removing the wrong rows.

Follow-up

A product team proposes counting any successful login as activity. What breaks?
The number jumps four percent overnight with no product change. What do you check first?
How would you report customers onboarded inside the window, given they had a shorter opportunity to transact?

Portfolio delinquency improving while the loan book doubles

Easy · vintage analysis · age mix · right censoring

The blended 90-plus days-past-due rate across fct_loan_performance_monthly fell from 3.1 to 2.2 percent over two quarters while monthly funded volume roughly doubled. Credit leadership wants to know whether underwriting improved. Columns: loan_id, as_of_month_end, origination_month, months_on_book, original_principal_minor, principal_balance_minor, days_past_due, delinquency_bucket, restructured_flag, charge_off_flag, charge_off_date. Produce the view that answers the question honestly, and state in one sentence what the blended rate can and cannot tell you.

Approach

Name the mechanical floor first. A first instalment falls due roughly a month after funding, so a loan cannot reach dpd_90_plus until around its fourth month on book. Every recent origination therefore enters the denominator with a numerator that is structurally zero.
Build a vintage table: rows origination_month, columns months_on_book, cell equal to the share of that cohort whose worst days_past_due reached 90 or more, or whose charge_off_flag became true, at or before that age.
Use each loan's worst state to date rather than its current bucket, and take the pre-restructure worst state, because restructuring resets days_past_due and would otherwise read as a cure.
Compare cohorts only at equal months_on_book, and render cells beyond a cohort's current maturity as absent rather than zero, so the table cannot be misread left to right.
Decompose the blended move into an age-mix component and a within-age component, so the write-up states how much of the 0.9 point improvement is arithmetic rather than asserting it.

Follow-up

What does the diagonal of a vintage table represent, and when is reading it the right thing to do?
How would a change in charge-off timing policy show up in this table, and how would you separate it from credit quality?
Which single chart goes in front of the credit committee, and what do you say when someone asks for the blended series anyway?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Discuss your experience with SQL and how you use it in data analysis.

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
Quantify the outcome, including what you would not claim credit for.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Share an example of how you have collaborated with cross-functional teams in the past.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

State honestly what your cutoff change actually contributed

Hard · impact attribution · swap set · counterfactual

Six months ago your recommendation moved a credit cutoff, using fct_loan_application and fct_loan_performance_monthly. Since then approval rate rose four points and the 12-month vintage 90-plus rate on affected cohorts is flat. In the same window the bureau changed a score attribute, marketing shifted channel mix toward broker, and the internal funding rate moved. Your performance review asks for impact in currency terms. Give the number you would stand behind, the counterfactual it rests on, and the part of the observed movement you would not claim.

Approach

Define the counterfactual before computing anything: the claim is not what happened after the change, it is what would have happened had the old cutoff scored the same applications, which means replaying the old threshold on the post-change population.
Build the swap set: applications the new cutoff approves that the old one declined, applications the old one approved that the new one declines, and everyone else held out as unaffected. Only the swap groups carry your effect. Note that the swap-out group has no outcome under the new rule, because those loans were never funded, so its forgone margin must be estimated from matched pre-change approvals rather than observed.
Price each swap group at months_on_book equal to 12 on the measure the cutoff was meant to move: interest and fees collected, minus net charge-offs, minus funding cost at the internal transfer rate.
Strip the confounders explicitly. Cohorts affected by the bureau attribute change are either recomputed on the old attribute or excluded; channel mix is held fixed by reweighting to the pre-change mix; funding cost is charged at the rate in force each month rather than one blended average.
State the residual you will not claim, with its size, and give a range rather than a point wherever cohorts have not yet reached 12 months on book.

Follow-up

The swap-in group is only 6 percent of applications. How does that change the way you present the number, and to whom?
What would you have needed to set up at launch to make this attribution clean, and why was a randomised band around the cutoff not used?