Interview guides · Citadel Securities

Citadel Securities Data Scientist Interview Questions & Guide 2026

Citadel Securities data scientist prep on data pitfalls: selection bias, target leakage, and checking an outlier's provenance before you drop it.

The 60-second brief

At Citadel Securities, a Data Scientist does not operate in a standard corporate tech capacity. Instead, you sit at the absolute center of global financial markets, where data is the primary leverage point for generating alpha and managing risk. The role is highly quantitative, blending elements of software engineering, statistical mechanics, machine learning, and financial modeling to extract signals from massive, noisy, and high-frequency datasets.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Separate forecast decay from execution cost
Build point-in-time panels without lookahead
Attach uncertainty to every Sharpe claim

Overview

You will be responsible for building and optimizing the predictive models that power automated market-making strategies and algorithmic execution. Because Citadel Securities accounts for a significant percentage of all U.S. equities and options volume, the systems you contribute to must process terabytes of market telemetry in real time. Your work directly impacts liquidity provision, transaction cost minimization, and the firm's ability to price risk across global asset classes.

This position demands an elite level of mathematical rigor and computational fluency. You will collaborate daily with Quantitative Researchers,, and portfolio managers to translate complex mathematical formulations into production-grade code. For those who thrive on immediate feedback, immense scale, and intellectually uncompromising environments, this role offers an unparalleled platform for impact.

The Citadel Securities interview process

Citadel Securities candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Call

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Online Assessment

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Technical Phone Screens

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Onsite Interview Loop

A loop is not scored one interview at a time. The people you meet compare notes afterwards, usually in a meeting you are not in, and the outcome turns on what each of them can say about you when asked. That rewards something other than survival: every room needs one specific thing worth repeating, and none of them can contradict another. The common way to lose is to tell the same project four times with different numbers in it, or to be uniformly fine in a way that leaves nobody with anything to argue for.

What to demonstrate

Whether your account of a project survives being told twice, with the same scale, the same metric definition and the same numbers each time
Whether each interviewer leaves with one concrete claim they could make on your behalf later, rather than an absence of complaints
Whether a question you already answered in an earlier room gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page fact sheet for your two or three main projects that fixes the numbers you will quote: rows of data, the metric as a single sentence, the effect you measured and how long the work took. Say them aloud from the sheet until they come out identical every time
For each kind of room you expect, decide the one sentence you want that interviewer repeating in a debrief, then check during the mock that you said it outright instead of implying it
Rehearse answering the same project question twice in one sitting, the second time as though you had not just answered it, because the thing that needs fixing is the flatness that creeps into a repeated story

Final Meeting

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Judging execution quality against interval VWAP and treating a favourable number as proof of good trading.

Interval VWAP is a benchmark the trader partly determines: trading in line with volume tracks VWAP almost by construction, and stretching an order over a longer interval makes the benchmark easier while exposing the position to price drift that the benchmark never charges. Arrival price is the benchmark aligned with the decision, because it charges both the spread and the drift between decision and completion, including the unfilled remainder. Reporting both, and reporting the opportunity cost of unfilled quantity, is what separates a real TCA from a flattering one.

Reporting the best backtest out of many trials as if it were a single pre-registered test.

The maximum of N noisy Sharpe estimates grows roughly like the standard error times sqrt(2 ln N) even when every underlying strategy has zero edge, so with a few hundred variants an in-sample Sharpe near 1 is the expected result of pure noise. Worse, the search is rarely counted honestly: parameter sweeps, universe changes, date-range choices and feature variants all count as trials. Quote the number of configurations tried, deflate the Sharpe for it, and keep a genuinely untouched holdout period. Note also that the asymptotic standard error of a Sharpe estimate is approximately sqrt((1 + SR^2/2)/T) for i.i.d. normal returns, which for three years of daily data is roughly 0.33, so two strategies differing by 0.3 in Sharpe are not distinguishable.

Averaging per-user rates to produce a population rate

Decide which quantity you want: the mean of per-user ratios and the ratio of summed numerator to summed denominator are different estimands, and heavy users dominate one but not the other. For a ratio metric, aggregate numerator and denominator separately and use the delta method for its variance.

Optimising accuracy on a heavily imbalanced target

State the base rate first, then choose the metric from the relative cost of a false positive against a false negative: precision and recall at the operating threshold, PR-AUC, or expected cost. At a 1 percent positive rate, predicting the majority class for everyone scores 99 percent accuracy and is worthless.

Questions & practice

Walk me through the mathematical derivation of the transition probabilities in a finite-state Markov Chain. How do you determine if a stationary distribution exists?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Write down the assumption the method needs before you use the method.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

Suppose you have three data groups. What is the mathematically possible range of their pairwise correlations, and how do you prove the boundaries of these correlations?

Medium · statistics and probability

Approach

Say what the estimate is of, and over what population it generalises.
Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

What is the exact mathematical difference between Ridge (L2) and Lasso (L1) regression regularization? Under what specific data conditions would you prefer one over the other?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Block bootstrap confidence interval for an annualized Sharpe

Medium · bootstrap · inference · numpy

daily_pnl has business_date, strategy_id and net_return: five strategies, 1,260 daily observations each, returns net of all costs. Build a 95% confidence interval for each strategy's annualized Sharpe using a moving-block bootstrap you write yourself, with no library resampler. Choose the block length from the data and justify it. Report each interval beside the i.i.d. normal approximation, sqrt((1 + SR_daily^2/2)/T) scaled by sqrt(252), and say which strategies the two methods disagree about and why.

Approach

Measure the dependence before resampling: compute lag-1 through lag-20 autocorrelation of net_return per strategy. The block bootstrap only earns its cost where that is non-zero, and on a strategy where it is flat the two intervals should agree, which is your implementation check.
Form the n - L + 1 overlapping blocks as a strided view of the return array, draw ceil(n/L) block starts with replacement, concatenate and truncate back to n. Recompute the annualized Sharpe on each resample; 5,000 resamples is enough for a 95% percentile interval.
Start at L near n^(1/3), about 11 for n = 1,260, then tabulate interval width against L over 5 to 30. A width still climbing at L = 30 means the dependence outruns the block and the interval is still too narrow, which is information about the strategy, not a bug.
Take either the percentile interval or the basic (reverse-percentile) interval 2*theta_hat minus the quantiles, and say which. The block bootstrap distribution centres on the sample statistic, so the two differ whenever the resample distribution is skewed, and for a ratio it is.
Cross-check against the closed form. For an AR(1) with coefficient rho, the long-run variance of the mean inflates by (1+rho)/(1-rho), so the interval should widen by about the square root of that. It is one line of arithmetic that tells you whether the block machinery is doing what the autocorrelation says it should.

Worked solution (35 min)

r = df.loc[df.strategy_id == s, 'net_return'].to_numpy(); n = r.size; sr = r.mean()/r.std(ddof=1)*sqrt(252).
blocks = np.lib.stride_tricks.sliding_window_view(r, L), shape (n-L+1, L).
Per resample: idx = rng.integers(0, n-L+1, size=ceil(n/L)); x = blocks[idx].ravel()[:n]; store its annualized Sharpe.
ci = np.percentile(boot_sr, [2.5, 97.5]); repeat for L in {5, 11, 21, 30} and tabulate the widths.
Put the i.i.d. approximation, the block interval and the lag-1 autocorrelation in one table per strategy.

Expected result: For a strategy with Sharpe 1.2 over five years (n = 1,260) the i.i.d. interval is 1.2 +/- 0.88, roughly [0.32, 2.08], since sqrt(252/1260) = 0.447. A strategy with lag-1 autocorrelation near +0.15 comes back about 16% wider (sqrt(1.15/0.85) = 1.16), roughly [0.18, 2.22]. A strategy with no serial dependence should reproduce the i.i.d. interval to within Monte Carlo noise.

Verify it yourself

Setting L = 1 must reproduce a plain i.i.d. bootstrap, which for a strategy with no serial dependence must land close to the closed-form interval.
Every resample must contain exactly n observations after truncation; a drifting length changes the Sharpe through both the ddof term and the sqrt(252) scaling.
The bootstrap mean of the Sharpe should sit near the sample Sharpe. A large gap is the ratio estimator's bias rather than a coding error, and it belongs in the write-up.

Follow-up

One strategy holds corporate bonds marked with mark_source = 'vendor_eval'. What does mark smoothing do to the lag-1 autocorrelation, to the annualized Sharpe itself, and to which of your two intervals you believe?
Would you use the same block length to bootstrap maximum drawdown? What breaks?
How does the answer change if you bootstrap 60 non-overlapping 21-day blocks instead of overlapping daily blocks?

Given an array of integers, write a dynamic programming solution to find the maximum sum of non-adjacent elements. What are the time and space complexities?

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Explain how you would design and implement a custom class structure in Python to manage a real-time order book, ensuring $O(1)$ lookup and update times.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Continuous holding spells from a daily position panel

Hard · gaps-and-islands · window-functions · calendar · panel-data

position_daily holds one row per (business_date, account_id, strategy_id, instrument_id) with signed quantity, weight_pct_nav, realized_pnl_base_day and unrealized_pnl_base_day. Rows exist only for business dates, and a flat day may appear as quantity = 0 or not at all. For one account_id, return every continuous holding spell — consecutive business dates with non-zero quantity and unchanged sign — as one row: instrument_id, strategy_id, side, start date, end date, length in trading days, peak absolute weight and total P&L. Weekends and exchange holidays must not split a spell.

Approach

Build a trading-day index before anything else: DENSE_RANK() OVER (ORDER BY business_date) across the distinct business dates in the panel, or against a calendar table if the panel itself can have gaps. Differencing raw dates counts calendar days and breaks every spell at a weekend; differencing the index counts sessions, which is the definition in play.
Restrict to quantity <> 0 and derive SIGN(quantity). A long that flips to short is two spells, so sign belongs in the island's partition key, not in the aggregate — otherwise a reversal is reported as one continuous position with a misleading average.
Apply the islands construction on the index: day_idx - ROW_NUMBER() OVER (PARTITION BY account_id, strategy_id, instrument_id, qty_sign ORDER BY day_idx) is constant within a run of consecutive sessions, so grouping by it collapses each run to one row.
Aggregate per island: MIN and MAX of business_date, COUNT(*) as the session length, MAX(ABS(weight_pct_nav)) for peak size, and SUM(realized_pnl_base_day + unrealized_pnl_base_day) for spell P&L.
Fix the gap policy before writing the predicate and state it: a single flat session that splits a spell into two is a different answer from one that is tolerated. If tolerated, merge adjacent islands separated by one session in a second pass, rather than loosening the non-zero filter, which would also swallow genuine exits.

Worked solution (40 min)

Create the session index CTE with DENSE_RANK over business_date across the account's panel.
Filter to non-zero quantity, add SIGN(quantity), and compute the island key with ROW_NUMBER.
Group by (strategy_id, instrument_id, qty_sign, island key) and aggregate the five output measures.
Check for overlap: self-join spells of one (instrument, sign) and assert no pair has intersecting date ranges.
Reconcile: summed spell lengths per (strategy, instrument) must equal that key's non-zero row count.

Expected result: One row per continuous spell; spells for a given (strategy_id, instrument_id, sign) are disjoint and ordered; lengths are in sessions, not calendar days; a position held across a long weekend produces one spell.

Verify it yourself

Summed spell length per (strategy_id, instrument_id) equals the count of non-zero rows for that key.
No two spells for one (instrument_id, sign) have overlapping [start, end] ranges.
A name known to be held continuously through the sample returns exactly one spell.
Total spell P&L equals the panel sum of realized_pnl_base_day + unrealized_pnl_base_day over non-zero rows.

Follow-up

A corporate action changes instrument_id mid-spell. How do you keep the spell whole, and which table establishes that the two ids are the same economic position?
Some rows inside a spell carry recon_status = 'break_quantity'. Include or exclude them, and what does each choice do to the length distribution?
How would you extend this to report the P&L contribution of the first five sessions of each spell separately from the rest?

Implement a Monte Carlo simulation in Python to estimate the price of a path-dependent financial option. How would you optimize this simulation using vectorization?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you optimize matrix multiplication for extremely large, sparse datasets? Discuss the underlying hardware and memory considerations.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Solve a classic LeetCode Hard string manipulation or graph traversal problem under live observation, explaining your memory allocation and complexity trade-offs.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Cut variance in an execution test without touching the effect

Medium · cuped · variance reduction · stratification

The order-level shortfall test you sized needs nine months and the desk will not wait. For every parent order you can compute, before assignment: the impact model's predicted cost in bps from participation_rate_target, adv_20d and trailing volatility; the instrument's gics_sector_code; and the strategy's trailing sixty-day mean shortfall. Predicted and realised cost correlate at 0.55 historically. Propose a variance-reduction plan, state the reduction you expect and the resulting runtime, and draw the line between admissible and inadmissible covariates.

Approach

Apply CUPED with predicted cost as the covariate: Y_adj = Y - theta * (X - mean(X)), with theta = Cov(Y, X) / Var(X) estimated on the pooled experiment sample. The adjusted estimator targets the same estimand without bias because X is independent of assignment, and its variance falls by the factor 1 - rho^2.
Put a number on it. At rho = 0.55 the reduction is 1 - 0.3025, so about 30 percent of the variance and therefore 30 percent of the sample size. State that this is a ceiling: rho estimated in-sample overstates the reduction you will realise, so quote the out-of-sample correlation if you have one.
Draw the admissibility line sharply. A covariate is admissible only if it is fixed before randomisation. Predicted cost, adv_20d, participation_rate_target, order_qty, decision_mid_px, gics_sector_code and the strategy's trailing shortfall all qualify. Realised participation rate, realised fill count, interval_vwap_px, terminal_ts and anything read from execution_fill do not, because the treatment changes them, and conditioning on a post-treatment variable introduces a bias that no amount of additional sample removes.
Add stratified assignment on the same information, bucketed as participation quintile by adv_20d quintile, so balance is enforced rather than hoped for. Stratification and CUPED compete for the same variance; running both buys a little extra, mostly as insurance against a bad draw inside a thin stratum rather than as a further cut to the standard error.
Pre-register how theta is estimated and what the strata are. Choosing theta after seeing the treatment effect turns CUPED into a specification search with a respectable name attached to it.

Worked solution (30 min)

Build a covariate table keyed on order_id containing only pre-assignment fields: predicted cost, adv_20d, participation_rate_target, order_qty, gics_sector_code and trailing sixty-day strategy shortfall.
On a historical sample that precedes the experiment, regress realised shortfall on predicted cost, confirm rho is near 0.55 out of sample, and record theta-hat = Cov / Var.
Compute the variance reduction 1 - rho^2 and the revised per-arm sample size and runtime.
Once the test is running, verify that mean predicted cost is statistically indistinguishable across arms. If it is not, the randomisation is broken and CUPED will quietly paper over it.
Report the adjusted and unadjusted effects side by side, along with the realised variance reduction.

Expected result: A 30 percent variance reduction at rho = 0.55. Per-arm sample size falls from about 8,550 to about 5,970 and runtime from roughly nine months to a little over six. Adjusted and unadjusted point estimates should differ only by noise.

Verify it yourself

Adjusted and unadjusted point estimates should sit within roughly one standard error of each other. A large divergence means the covariate is post-treatment or the arms were not balanced on it.
The variance reduction realised in the experiment should be close to 1 - rho^2 measured out of sample. Materially more means rho was fitted on the same data it is being credited for.
Theta estimated on the control arm alone and on the pooled sample should be statistically indistinguishable; a difference indicates the treatment is altering the covariate-outcome relationship.

Follow-up

What goes wrong if theta is estimated separately within each arm rather than pooled?
Shortfall is heavy tailed. Does winsorising before CUPED help or hurt, and what does it do to the estimand you are reporting?
Which is worth more here: CUPED at rho = 0.55, or moving the decision from the equal-weighted to the notional-weighted mean?

TCA traded notional fell while book turnover held

Medium · pipeline failure · timezone · segment cuts · reconciliation

The daily TCA report's traded notional dropped about 12 percent starting on a Tuesday, while turnover computed from position_daily is unchanged. No orders were rejected and parent_order.status shows the usual filled share. You have parent_order (order_id, decision_ts, arrival_ts, terminal_ts, order_qty, filled_qty, broker_code, algo_name) and execution_fill (fill_id, order_id, exec_ts TIMESTAMPTZ(6), received_ts, venue_mic, fill_qty, fill_px, is_correction). Find the missing notional and name the mechanism. Give the ordered checklist, including the cut you would make first and the cut you would deliberately not make first.

Approach

Reconcile at the order level first. parent_order.filled_qty is maintained by the order management system independently of the fill table, so orders where filled_qty exceeds SUM(fill_qty) over their non-correction fills enumerate exactly which fills the report lost. That turns a 12 percent aggregate into a concrete row set you can cut.
Cut those rows by exec_ts hour in UTC and by venue_mic before cutting by broker_code. The hour cut tests a time-boundary mechanism. The broker cut will also light up, because whichever broker routes the affected region's flow looks responsible, and that correlation is what sends people down the wrong path.
Read the report's date predicate. exec_ts is TIMESTAMPTZ(6) in UTC; if the job buckets fills by a local calendar date while bounding parent orders by a different zone's date, every fill that crosses midnight in the mismatched zone falls outside the join window and vanishes without raising an error.
Establish whether rows were dropped or misfiled. Sum the report over two days instead of one. If the two-day total reconciles, the rows landed in the adjacent bucket; if it does not, they are gone. These look identical in a daily total and demand different fixes.
Date the change against deploys and configuration rather than against market events, and re-run the prior week's data through the current job to confirm the code moved and the data did not.

Follow-up

The rows turn out to be misfiled rather than dropped. Does that change your severity assessment, and what does it do to the month-to-date figure?
Write the predicate you would use so the same report is correct for venues in every timezone, including one whose session spans the UTC date boundary.
received_ts and exec_ts differ by up to several seconds. Which belongs in the bucketing predicate and which belongs in latency monitoring?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Scoping is our execution any good into one answerable question

Medium · scoping · tca · requirements

A chief operating officer stops you in the hallway and asks whether our execution is any good. You have parent_order and execution_fill for eighteen months, roughly 400,000 parent orders across four algos and six brokers. Nobody has defined good, no deadline is set, and two people have already produced conflicting answers. Before writing any SQL, produce a one-page scoping memo: the single question you will answer, the metric defined at field level, the slices, the exclusions, and what you are explicitly not answering.

Approach

What is probed: whether you convert a request into a decision. The deliverable of scoping is not a work plan, it is the sentence describing what the requester will be able to decide once you are done.
Choose the benchmark explicitly and state what each one charges. Arrival mid charges spread, impact and the price drift between the desk receiving the order and completing it, including the unfilled remainder. Interval VWAP charges almost none of that and is partly determined by the trader's own volume participation. Answer against arrival and report interval VWAP alongside so nobody believes you hid the flattering number.
Write the metric to the field level so two analysts cannot compute it differently: side_sign times (avg_fill_px minus arrival_mid_px) times filled_qty, plus commission_amt plus exchange_fee_amt minus rebate_amt, plus opportunity cost on order_qty minus filled_qty at the terminal mid, divided by order_qty times arrival_mid_px, times 10000, aggregated by weighting each order by order_qty times arrival_mid_px.
Fix the slices in advance, because choosing them after seeing results is how a scoping memo becomes a fishing expedition: algo_name, broker_code, side, and a participation bucket defined as order_qty divided by adv_20d. Set a minimum cell size so a 12 bps difference on 40 orders never reaches a slide.
State the exclusions and the known data problems in the same memo: corrections net against corrects_fill_id rather than being counted twice, orders with a null arrival_mid_px are reported as a coverage percentage rather than dropped quietly, and clock skew between exec_ts and received_ts is a data-quality item and not an execution-quality finding.
Close with the questions you are not answering, each with a rough cost: whether the strategy should trade less, whether broker relationships are priced correctly, and whether the algos' internal logic is sound.

Follow-up

The COO says 12 bps sounds fine. What do you compare it against, and where does that comparison come from?
The two existing answers disagree. How do you determine whether they used different benchmarks or different populations?
What changes if this has to become a monthly production number rather than a one-off study?

Answering whether a six-week-old signal is working yet

Easy · uncertainty · executive communication · statistical power

A new signal has been live six weeks: 30 trading days of realized cross-sectional IC against a 5-day forward return, mean 0.030, standard deviation across days 0.12. An executive with no statistics background asks in a Monday meeting whether it is working and wants a yes or a no. You have the daily IC series and nothing else. Give an answer in three sentences plus one number the executive can hold onto, and say when the question becomes answerable.

Approach

What is probed: whether you can be honest about statistical power without hiding behind the word significant and without giving a yes that gets quoted back at you in three months.
Compute the interval before you speak. The standard error of the mean daily IC is 0.12 divided by the square root of 30, which is 0.022, so a mean of 0.030 sits about 1.4 standard errors from zero. That is the optimistic bound and it is already not a yes.
Adjust for overlap and say that you did. A 5-day forward return sampled every day shares four of five days with its neighbour, so the honest standard error uses a Newey-West estimator with at least 4 lags and lands materially above 0.022. Presenting the naive figure without that caveat is the same error as the signal's own author would make.
Convert power into a date rather than a verdict. Detecting a true mean IC of 0.03 at two standard errors needs roughly (2 x 0.12 / 0.03)^2 = 64 independent days, and with the overlap inflation of a 5-day horizon that is on the order of 300 trading days, so the question becomes answerable around fifteen months in, not six weeks.
Give one number and one decision, because wait is useless on its own. Offer a tripwire that makes waiting active: a pre-committed stop if the trailing 60-day mean IC turns negative, and a named review date.

Follow-up

Another desk called their signal working after four weeks. What do you say when the executive raises that?
What single observation before the review date would make you stop the signal early?
The six-week mean is minus 0.03 instead. Does your answer change in substance or only in sign?

Correcting a published TCA note that dropped cancelled orders

Medium · error disclosure · tca · selection bias

Three months ago you published a monthly transaction cost note concluding that algo A beat algo B by 12 bps of implementation shortfall, and the desk moved flow to A. You have since found that your query filtered parent_order to status = 'filled', so cancelled and expired orders never entered the numerator or the denominator. A cancels more often than B. Write the correction: what you tell the desk, in what order, and what changes so this class of error is caught rather than found.

Approach

What is probed: whether you report your own error at the speed and specificity you would demand from someone else. The clock starts when you know, not when the rework is finished.
Size the error before describing it. Recompute both algos with cancelled and expired parent orders included and opportunity cost charged on order_qty minus filled_qty at the terminal mid, as the shortfall definition requires. The gap may shrink, vanish or reverse, and I do not yet know the sign is a legitimate first message only if it arrives within hours.
Name the mechanism that makes the bias directional rather than noisy. Orders get cancelled disproportionately when price runs away from the decision, so excluding them removes the worst outcomes, and it removes more of them from the algo that cancels more. That is why the filter flattered A specifically.
Tell the desk head in person before the corrected note circulates, and lead with the operational consequence rather than the methodology: flow moved on a wrong number, and here is what to do with it today.
Make the fix structural rather than a promise to be careful. Add an assertion to the TCA job that the count and notional of parent orders in the report equal the count and notional in parent_order over the window, grouped by status, so a silent population drop fails the job instead of shipping.

Follow-up

The corrected numbers still favour A, by 3 bps. Does the desk need to hear from you at all, and why?
Who else built on that note, and how do you find out rather than guess?
Your review process passed a status filter. What specifically in it was supposed to catch a population change?