Interview guides · ASRC Federal Holding

ASRC Federal Holding Data Scientist Interview Questions & Guide 2026

Modeling drills for a ASRC Federal Holding data scientist interview: baselines, regression interpretation, class imbalance, model defense.

The 60-second brief

As a Data Scientist at ASRC Federal Holding, you operate at the intersection of complex data ecosystems and high-stakes government contracting. This role is pivotal for transforming vast, often fragmented datasets into actionable insights that support mission-critical decisions. You aren't just building models; you are solving structural problems that impact the efficiency and effectiveness of large-scale operations.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

Separate obligations, outlays and ceilings correctly
Design outputs that survive public disclosure
Estimate eligible populations outside administrative data

Overview

The work is intellectually demanding and requires a high degree of technical autonomy. You will be expected to translate ambiguous requirements into rigorous analytical frameworks, ensuring that every data product you deliver is both technically sound and strategically aligned with organizational goals. Success in this role requires a unique blend of curiosity, technical precision, and the ability to communicate complex findings to stakeholders who may not have a technical background.

The ASRC Federal Holding interview process

ASRC Federal Holding candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Structured Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Final Assessment

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Computing coverage or take-up with an administrative denominator, for example approved applications divided by all applications

That ratio measures throughput among people who already found the system, not delivery to the people entitled to the service. It gets better when outreach is cut, because the marginal applicant is the one most likely to be denied or to abandon, and it gets worse when a new access channel brings in harder cases. The correct denominator is a modelled eligible population from survey microdata run through the eligibility rules, and the gap between it and the applicant count is usually the finding.

Treating event_at in the case event log as when the event actually happened

Staff backdate to the true date of an action and enter it later, and batch loads write many rows at once, so event_at and recorded_at diverge systematically and the divergence is largest at fiscal-period and reporting-deadline boundaries. An interrupted time series keyed on event_at will read a queue flush before a deadline as a level shift caused by the policy change, and a dashboard keyed on recorded_at will show activity on days when nothing happened. Reconcile both columns, model the entry lag distribution, and state which clock each metric uses.

Comparing periods without accounting for seasonality or day-of-week

Compare whole weeks against whole weeks and check whether the same swing appeared in prior cycles or prior years before attributing it to anything you changed. Weekday and weekend populations often differ enough that a Tuesday-to-Saturday comparison is meaningless.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

What are the trade-offs between false positives and false negatives in a predictive model?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Resolve service-request reopen chains and score the 30-day rate

Hard · graph traversal · sessionisation · reopen rate

requests has service_request_id, reported_at, closed_at, closure_code, request_type_code and reopen_of_request_id, a nullable self reference to the earlier request this one repeats. Resolve the chains into episodes: give every row a root id, a depth and an ordinal, without recursion and without looping forever on a cycle. Then compute the 30-day reopen rate, where the denominator is requests closed in the window with closure_code 'resolved' and the numerator is those with a child reported within 30 days of closed_at. Report dangling parent references and any cycle you find.

Approach

Separate genuine roots (reopen_of_request_id is null) from dangling references where the parent id is absent from the table. Rooting a dangling child silently welds two episodes together and hides an extract that was cut mid-chain.
Walk with iterative pointer jumping: each pass remaps every row to its parent's parent, so the remaining distance to the root halves per pass and about 20 passes cover a chain a million deep. Point every root at itself so the jump is a no-op once a row has arrived, and stop when no pointer changes rather than at a fixed count.
Carry a distance vector alongside the pointer or the depth column is wrong. Under doubling a row at true depth d stops moving after about ceil(log2 d) passes, so the pass number is log2(depth) and not depth: a chain 8 deep stabilises on pass 3. Hold dist with the invariant that dist[i] is the number of edges from i to ptr[i], initialised to 1 for every child and 0 for every root, and update dist[i] += dist[ptr[i]] and ptr[i] = ptr[ptr[i]] together from the previous pass's arrays. At the fixed point ptr[i] is the root and dist[i] is the true depth.
Detect cycles by capping iterations at ceil(log2(n)) + 1 and flagging rows whose pointer is still moving. A cycle means a request was recorded as a reopen of its own descendant, which is a data error and must be quarantined rather than reported as an episode.
Assign ordinal within root by reported_at, then compare that ordering against the foreign-key chain order. Disagreement means backdated reported_at values and belongs in the data-quality report, not silently inside the metric.
For the metric, join each qualifying closed request to its children and keep those with 0 <= reported_at - closed_at <= 30 days. A child reported before its parent closed is an error, not a reopen, and counting it inflates the numerator.
Apply the exclusion of duplicate, invalid and closed_on_age to both numerator and denominator, then report the rate both ways so the size of the flattery is on the page rather than in a footnote.

Follow-up

Two crews close the same kind of work differently, one as 'resolved' and one as 'referred_external'. What does that do to this metric, and how would you detect it from this table alone?
A chain crosses a request_type_code retirement, so parent and child carry different codes. How do you attribute the episode, and what do you report by type?
The rate is 12 percent overall and 3 percent in one district. Give two explanations that are not about work quality.

Reconcile obligations, ceilings and outlays across snapshot dates

Easy · snapshots · aggregation · data quality

obligations holds one row per funding action: obligation_action_id, award_id, action_type, action_at, fiscal_year, obligated_delta_cents (signed), ceiling_amount_cents, outlay_to_date_cents, program_code, snapshot_date and is_current_snapshot. The table is a stack of as-of snapshots, and outlay_to_date_cents is an award-level cumulative figure repeated on every action row inside a snapshot. Produce a per-award frame with net obligated value, ceiling in force and outlay; a program by fiscal-year obligation total; and an exception report for awards where net obligated exceeds the ceiling or outlay exceeds net obligated.

Approach

Filter to is_current_snapshot and assert that exactly one snapshot_date survives. If several do, the flag is maintained per award rather than per snapshot and every total below counts some awards twice.
Net obligated per award is the sum of signed obligated_delta_cents. Deobligations and terminations are negative and must be summed in; filtering to action_type = 'new_award' reports gross intent, not money committed.
Ceiling is the value carried on the latest action by action_at, because a modification can raise it. Summing ceiling_amount_cents across actions is a category error: it is a maximum the award may reach, not cash.
Outlay is award-level and repeated, so reduce it with first after asserting nunique() == 1 per award. Summing it multiplies the figure by the action count, and the result still looks plausible, which is what makes it dangerous.
Build fiscal totals by grouping deltas on the action's fiscal year, and cross-check the stored fiscal_year against the value derived from action_at. Disagreements are an assignment bug worth listing, not rounding.
Exception frame: net > ceiling, outlay > net, earliest action not of type new_award, and net < 0. Report counts and cents rather than percentages of a denominator you have not defended.

Worked solution (25 min)

Filter on is_current_snapshot, then assert obligations['snapshot_date'].nunique() == 1 and stop if it fails.
Group by award_id with a different reducer per column: sum for obligated_delta_cents, the value at max action_at for ceiling_amount_cents, and first for outlay_to_date_cents guarded by an nunique check.
Group by program_code and fiscal_year over the signed deltas for the fiscal totals, and recompute fiscal_year from action_at to list mismatched rows.
Build the exception frame by stacking the four rule violations with a rule column, so one award can appear more than once.
Return the three frames with cents intact; convert to dollars only at presentation.

Expected result: A per-award frame (award_id, net_obligated_cents, ceiling_in_force_cents, outlay_cents, outlay_ratio), a program by fiscal-year total built only from signed deltas, and an exception frame with one row per violated rule. Awards carrying terminations show net below the sum of their positive actions, and a low outlay_ratio on a recent award is expected rather than an exception.

Verify it yourself

The sum of per-award net equals the sum of obligated_delta_cents over the filtered rows, to the cent.
Every award has exactly one distinct outlay_to_date_cents and one distinct snapshot_date after filtering.
Recomputing fiscal_year from action_at reproduces the stored column for every row, or the mismatched rows are listed rather than silently overwritten.
No award in the per-award frame has ceiling_in_force_cents below the ceiling on its earliest action unless a modification lowered it, which should be visible as a matching action row.

Follow-up

Outlay sits at 40 percent of obligations for a program this fiscal year. Name three innocent explanations before you write the word underspending.
Leadership asks how much money was committed this year for an award that spans three years with option exercises. What number do you give and what do you refuse to give?
The snapshot flag turns out to be per award. What breaks in your aggregation and how do you rebuild it from snapshot_date alone?

Explain how you would optimize a slow-running query on a multi-terabyte dataset.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Write a query using SQL window functions to calculate a rolling average of user activity over the last 30 days.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Deduplicate replayed decision rows before counting denials

Medium · dedup · row_number · fan-out

Batch reloads wrote decision rows into fact_case_event more than once for the same case: identical application_id and event_at, different case_event_id and recorded_at, actor_type = 'system_batch'. Using fact_case_event (case_event_id, application_id, event_type, event_at, recorded_at, actor_type) and fact_application (application_id, status, denial_reason_code, decision_at), return denial counts by denial_reason_code for one month of decision events, plus the number of applications carrying more than one decision row. Treat the earliest recorded_at as the canonical decision for each application.

Approach

Rank the decision events with ROW_NUMBER() OVER (PARTITION BY application_id ORDER BY recorded_at, case_event_id) and keep rn = 1, using case_event_id as a deterministic tiebreaker in case two replays share a recorded_at.
In the same window pass, compute COUNT(*) OVER (PARTITION BY application_id) AS dup_count so the duplicate count falls out of the query that fixes the duplicates, rather than requiring a second scan.
Keep two different quantities apart: the number of applications with dup_count > 1 (what the prompt asks for) and the number of surplus rows, SUM(dup_count - 1). A group replayed three times contributes 1 to the first and 2 to the second, so the two agree only in the special case where every replayed application was written exactly twice.
Join the deduplicated event set to fact_application on application_id only after the dedup; joining the raw event table multiplies each denial by its replay factor, and because the factor varies by load batch the inflation is not a constant you can divide out.
Filter to status = 'denied' before grouping by denial_reason_code, otherwise approved and withdrawn rows contribute a NULL reason bucket that reads as an unlabelled denial category.
Report procedural and substantive reason codes as separate subtotals, since evidence_not_returned and over_income describe different failures and pooling them hides which one is moving.

Worked solution (25 min)

Count rows in fact_case_event with event_type = 'decision' for the month, and COUNT(DISTINCT application_id) over the same set. Their difference is the number of surplus rows, SUM(dup_count - 1) — not the number of affected applications, which is smaller whenever any application was replayed more than twice.
Build a CTE with ROW_NUMBER and COUNT(*) OVER the same partition, then select rn = 1.
Aggregate the duplicate measures over the rn = 1 rows: COUNT() FILTER (WHERE dup_count > 1) is the answer the prompt wants, SUM(dup_count - 1) reconciles to step 1, and COUNT() FILTER (WHERE dup_count > 2) tells you whether the two can be quoted interchangeably.
Join the rn = 1 set to fact_application, filter status = 'denied', group by denial_reason_code.
Run the same aggregation against the raw event table and record how much larger the denial counts come out.

Expected result: One row per denial_reason_code with counts that sum exactly to COUNT(*) of fact_application rows with status = 'denied' and decision_at inside the month, plus a single scalar for applications with more than one decision row. That scalar is smaller than the surplus-row count whenever any application was replayed three or more times. The un-deduplicated version returns strictly larger counts, and the inflation factor differs by reason code because batches did not cover reason codes evenly.

Verify it yourself

SUM of the per-reason counts equals COUNT(*) of denied applications decided in the month, computed independently from fact_application alone.
COUNT(DISTINCT application_id) after dedup equals COUNT(*) after dedup.
No NULL denial_reason_code bucket appears in the output.
SUM(dup_count - 1) over the rn = 1 rows equals raw decision rows minus distinct applications for the month — that is the identity that reconciles to step 1. The reported scalar, COUNT() FILTER (WHERE dup_count > 1), is greater than zero and no larger than that difference, and equals it only when COUNT() FILTER (WHERE dup_count > 2) is zero. Check that filter before describing either number as 'the duplicates'.

Follow-up

Denials counted from fact_case_event exceed the count of applications with status = 'denied'. Which number do you trust, and what does the gap tell you?
Would you dedup on earliest or latest recorded_at, and what evidence would settle it?
The replay dropped a few decision rows entirely rather than duplicating them. How would you detect that from these two tables?

If a key performance metric suddenly drops by 10%, what is your step-by-step process for diagnosing the root cause?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize which product features to build based on user engagement data?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design a metric to measure the success of a new internal data portal?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What are the risks of performing multiple comparisons in an A/B test, and how do you mitigate them?

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

Explain the difference between correlation and causation in the context of a controlled experiment.

Medium · a/b testing and experimentation

Approach

Name the randomisation unit first; it decides the variance and what the test can detect.
Decide the analysis before seeing data, including how long it runs and when you look.
Say whether units interfere with each other, and switch design if they do.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Define success and a clearance date for a growing adjudication backlog

Hard · queueing · littles law · censoring

An adjudication backlog is growing and leadership wants one number for whether it is clearing, plus a date. Available: fact_application (application_id, submitted_at, decision_at, sla_due_at, status, adjudicator_unit_id), fact_case_event (application_id, event_type, event_at, recorded_at, queue_id) and a capacity table giving each unit's headcount, shift pattern and decisions per staff-day. Define the primary metric and the projection, state the stability condition under which the projection means anything, and list the assumptions that make the date wrong.

Approach

Separate the two questions. The health metric is the on-time share anchored on sla_due_at, where an undecided case past its due date counts as a failure. The clearance projection is a capacity arithmetic problem and must not be presented as a trend extrapolation of the backlog line.
Measure duration correctly before projecting anything. Report a Kaplan-Meier estimate of days to decision over the submission cohort with still-open cases entered as right-censored at extract_date minus submitted_at, and treat withdrawal and administrative closure as competing risks with a cumulative incidence function rather than pooling them with substantive decisions. Any statistic taken over decided cases alone is conditioned on having finished before the extract, so it drops the slowest cases and sits below the corresponding full-cohort figure. State which statistic you are comparing: the censoring effect shows up as KM median above decided-only median, both medians.
Derive capacity from the capacity table rather than from recent output, because recent output is bounded by demand in some units and by staffing in others. Compute the service rate as headcount times shift days times decisions per staff-day, then discount for leave, training and supervisory time with the discount stated as a number.
State the stability condition explicitly: the queue drains only while the arrival rate is below the effective service rate, and the clearance date is backlog divided by the difference of two large and separately uncertain numbers. State that Little's Law relating work in progress, arrival rate and time in system is a steady-state identity, so it describes a stable queue and cannot be applied to a growing one to produce a wait time.
Publish the projection as a sensitivity band rather than a date. Show how the clearance horizon moves under a small capacity shortfall, because that is the honest headline: when the drain rate is a small difference between two large numbers, a five percent capacity error moves the date by years.

Worked solution (45 min)

Compute the on-time share anchored on sla_due_at, with undecided past-due cases in the denominator only.
Fit a Kaplan-Meier curve on the submission cohort with right-censoring at extract, and report the cumulative incidence of decision, withdrawal and administrative closure separately.
Derive the effective weekly service rate from the capacity table with the leave and training discount stated.
Compute the net drain rate and the clearance horizon, then repeat under a small capacity shortfall to produce the sensitivity band.
Write the assumption list, naming the stability condition, the constant-arrivals assumption, the constant-capacity assumption and the treatment of reopened cases as new arrivals.

Expected result: With arrivals of 900 per week and capacity of 4 units times 12 adjudicators times 5 days times 4 decisions per staff-day = 960 decisions per week, utilisation is 900 / 960 = 0.94, which satisfies the stability condition. A backlog of 11,000 drains at 60 per week, giving roughly 183 weeks, about 3.5 years. At a 5 percent capacity discount for leave and training, capacity is 912, the drain rate is 12 per week and the horizon is roughly 917 weeks. The deliverable reports the horizon as that band, not as a date, and states that the sensitivity is the finding.

Verify it yourself

Confirm the Kaplan-Meier median sits at or above the median over decided cases only, which is the signature of the censoring: the decided-only sample omits every case still open at extract. Do not compare the KM median against the decided-only mean, because days to decision is right-skewed and that mean routinely exceeds the KM median with no censoring effect present at all.
Confirm utilisation is below 1 before quoting any clearance horizon, and say what you would report instead if it were not.
Confirm reopened cases are counted as arrivals in the drain arithmetic, since a reopen puts work back into the queue.

Follow-up

Arrivals are strongly seasonal and capacity is not. Which parts of your projection break, and what would you compute instead of a single clearance date?
Two units show identical on-time shares but very different censored medians. What is happening, and which metric would you trust for a staffing decision?

Overturn rate appears to fall in recent denial cohorts

Hard · right truncation · development triangle · backdating

Appeal overturn rate on denials trends down across the last five monthly denial cohorts. The metric allows 180 days of follow-up and is computed at a fixed extract date. Using fact_application (status, decision_at, denial_reason_code, appeal_filed_at, appeal_outcome) and fact_case_event (event_type, event_at, recorded_at, is_backdated), decide whether decision quality improved, and give a defensible projection of the mature overturn rate for the newest cohort with the assumption it rests on.

Approach

Build a development triangle rather than a trend line: rows are denial cohort months, columns are months of lag since decision_at, cells are cumulative overturns per 100 denials in that cohort. The recent cohorts are observable only at short lags, so any comparison across cohorts must be read down a column at matched lag.
Read the matched-lag columns. If the lag-3 column is flat across all five cohorts, the apparent decline is right truncation and nothing about decision quality has changed; the reporting error is that immature cohorts were published as if complete.
If a decline survives at matched lag, project the newest cohort with age-to-age development factors estimated from the mature cohorts, and state the assumption explicitly: that the lag distribution is stable across cohorts. Then test it, because hearing capacity and scheduling changes move the lag distribution directly and would make the projection fabricate a number.
Split procedural from substantive denial_reason_code throughout. They have different overturn rates and different appeal lags, so a shift toward procedural denials moves the pooled rate with no change in any decision, and the triangle must be built per stratum.
Pin the clock. Appeal outcomes are backdated, so a cohort's cell grows after the fact; recompute the triangle against the prior extract and report the restatement size as a property of the metric, naming which column each published number used.

Follow-up

The lag distribution did shift because hearings were rescheduled. What do you publish for the newest cohort instead of a projection?
How would you present this so that a reader who only looks at the headline cannot mistake an immature cohort for a low one?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Bring numbers, including the ones you are unsure of

An answer without a quantity is hard to interrogate, so interviewers keep probing until they find one. Come with the baseline, the change, the window it was measured over, and how confident you were. If the effect never got measured, say so and say what you would have measured. Fabricated precision is worse than an honest gap.

Describe a situation where you disagreed with a teammate on an analytical approach; how did you resolve it?

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Correct a published duration figure you got wrong

Medium · censoring · error disclosure · survivorship

Six weeks ago you published median days to decision as the median over applications that had a decision_at. A colleague points out that applications still in status 'submitted' or 'pending_evidence' were dropped rather than censored; the Kaplan-Meier median on the same cohort is 41 days against the 22 you published. The figure sits on a public page and was quoted in a briefing. Set out what you do in the first twenty-four hours, and draft the correction note in under 150 words.

Approach

Quantify before you talk. Recompute with still-open applications entered as right-censored at extract_date minus submitted_at, confirm the direction and size, and check whether any decision that cited the figure would have gone differently at 41 days.
Establish who acted on it. Tell the owner of the affected decision and the owner of the publication first, in that order, rather than broadcasting to the widest audience or waiting until you have a full remediation plan.
Write the correction so the corrected number is the first thing a reader finds: the corrected figure with its method named, the mechanism in one clause (decided-only samples exclude the slow cases that are still open, so they are length-biased), and the exact period affected.
Fix the pipeline rather than the cell. Put the censoring rule into the metric definition, and add a test that fails if the published duration is computed on a decided-only subset.
Say what the incident teaches about the review step that missed it, and be specific: a reviewer checking the numerator and denominator counts would have seen the cohort shrink.

Follow-up

If the corrected figure is worse for the agency, does any part of your process change?
What single automated check goes into the recurring job, and what does it compare?

Explain why a ranked district table should not be published

Easy · small-area estimation · uncertainty · executive communication

You have twelve small areas ranked by service requests per 1,000 residents, computed from fact_service_request counts over dim_geography.population_estimate. The estimates carry population_estimate_moe published at 90 percent confidence, and for the smallest areas the margin is close to a quarter of the estimate. An executive wants the ranked list on a slide tomorrow as "the twelve worst districts". You have five minutes with her and cannot put an equation on the slide. Say what you show instead, and what you tell her.

Approach

Convert each published margin to a standard error first: at 90 percent confidence the standard error is moe divided by 1.645. Compute the relative standard error of the denominator for every area in the table.
Show where the uncertainty lives. Treating the administrative count as fixed and independent of the survey estimate, the rate's relative standard error equals the denominator's, so an area with a 24 percent relative standard error on population has a 24 percent relative standard error on its rate no matter how clean fact_service_request is.
Demonstrate the instability rather than asserting it: resample each denominator from its published standard error, recompute the ranking a few thousand times, and report how often each area actually lands in the worst twelve. Areas that appear in only a third of draws are not findings.
Replace the rank with something that survives the uncertainty: aggregate to a geo_level where the relative standard error clears a threshold you state out loud, or group areas into tiers whose intervals do not overlap, and name the threshold as a choice you made rather than a standard.
Give the executive one sentence she can repeat without you in the room, for example that the data supports naming a group of high-demand areas but not ordering them.

Follow-up

Which relative standard error threshold do you use, and why is it a judgement rather than a rule?
Two of the areas you aggregated sit across a boundary redraw. What breaks in the join, and how would you notice?