American Enterprise Institute Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at the American Enterprise Institute (AEI), you are stepping into a unique intersection of advanced analytics, machine learning, and public policy. Unlike traditional tech roles, your work here directly informs high-stakes economic, political, and social science research. You will be tasked with transforming massive, complex datasets into clear, actionable insights that empower scholars, policymakers, and the public.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

• Analyse at the account grain, cluster errors
• Separate contracted seats from actively used seats
• Power experiments for heavy-tailed account revenue

Overview

Your impact in this role is profound. By applying rigorous statistical methods and modern AI techniques to policy questions, you help shape national conversations. Whether you are modeling economic trends, analyzing demographic shifts, or building predictive tools to assess policy outcomes, your technical expertise acts as the engine for AEI’s intellectual output. You are not just building models in a vacuum; you are translating data into narratives that drive strategic influence.

Expect an environment that demands both academic rigor and industry-standard technical execution. The problems you will tackle are structurally complex and often highly ambiguous, requiring you to navigate large-scale survey data, government databases, and unstructured text. If you thrive on intellectual challenges and want your machine learning expertise to have a tangible impact on real-world policy debates, this role offers an unparalleled platform.

The American Enterprise Institute interview process

American Enterprise Institute candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

• Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
• Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
• Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

• Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
• Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
• Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Team Interviews

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Final Presentation

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

• Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
• Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
• Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
• Whether you ask what decision is actually on the table before explaining anything

How to prepare

• Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
• For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
• Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Reaching for a model before the target metric exists

Before naming an algorithm, write down the label, the prediction time, and the action that changes when the score crosses a threshold. If you cannot say what decision the output drives, any modelling choice is guesswork dressed up as method.

Questions & practice

How do you determine if a statistically significant result is actually practically significant?

Medium · statistics and probability

Approach

1. Say what the estimate is of, and over what population it generalises.
2. Quantify uncertainty explicitly rather than reporting a point estimate alone.
3. Translate the result into the decision it informs, in one plain sentence.

Follow-up

• What sample size would you need to detect an effect half this size?
• Which assumption here is most likely to be violated in practice?

How would you structure an analysis to determine the causal effect of a specific policy change?

Medium · statistics and probability

Approach

1. Sanity-check the answer against a simple bound or a simulated case.
2. Say what the estimate is of, and over what population it generalises.
3. Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

• Which assumption here is most likely to be violated in practice?
• How would you explain this result to someone who does not know statistics?

Walk me through the most complex machine learning model you have deployed.

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

Explain the assumptions of linear regression and what happens if they are violated.

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Check what information would not exist at prediction time, and exclude it.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Implement seven-day activation rate from its written definition

Easy · metrics · pandas · activation

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

Approach

1. Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
2. Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
3. Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
4. Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
5. Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Worked solution (20 min)

1. cohort = accounts.loc[~accounts.is_internal & accounts.created_at.between(week_start, week_start + pd.Timedelta(days=7), inclusive='left'), ['account_id','created_at']].drop_duplicates('account_id')
2. ok = requests[(requests.http_status < 400) & requests.api_key_id.notna() & (requests.traffic_class != 'synthetic_monitor')]
3. first_ok = ok.groupby('account_id', as_index=False).request_at.min(); m = cohort.merge(first_ok, on='account_id', how='left'); assert len(m) == len(cohort)
4. activated = (m.request_at - m.created_at) <= pd.Timedelta(hours=168); return {'rate': float(activated.mean()), 'numerator': int(activated.sum()), 'denominator': len(m)}

Expected result: A dict whose denominator is the count of distinct non-internal accounts created in that ISO week, numerator <= denominator, and rate = numerator / denominator; accounts with no qualifying request sit in the denominator with NaT and evaluate to False.

Verify it yourself

• len(m) == cohort.account_id.nunique(); the merge must not add rows.
• Dropping the synthetic_monitor filter can only raise the numerator, never lower it.
• Widening the window from 168h to 336h is monotone: the numerator must not decrease.
• An account whose only qualifying request falls at exactly created_at + 168h counts as activated, since the comparison is inclusive.

Follow-up

• Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
• How would you decide whether 168 hours is the right window rather than 72 or 336?
• An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Consecutive qualifying weeks before renewal, as a ranked worklist

Hard · gaps and islands · window functions · date boundaries

fct_api_request carries account_id, workspace_id, environment, request_at (timestamptz), http_status and traffic_class. fct_subscription_period carries account_id, term_end_date, auto_renew and is_current. A qualifying week for an account is an ISO week with at least 50 successful production requests in traffic_class ('interactive','batch'). Over the last 52 whole ISO weeks, and for accounts whose current term ends within 90 days, return the current qualifying-week streak length, the week it began, and the longest earlier streak. An account whose streak has broken must appear with a current length of zero.

Approach

1. Bucket weeks as date_trunc('week', request_at AT TIME ZONE 'UTC'). request_at is a timestamptz, so an unpinned date_trunc silently uses the session time zone, weeks start at a local midnight, and Monday-morning traffic lands in the previous week for part of the fleet. Pinning UTC also makes the seven-day arithmetic below exact across daylight-saving transitions.
2. Apply the exclusions before counting: environment = 'production', http_status < 400, traffic_class IN ('interactive','batch'). Then apply the volume floor and drop the current partial week, which can never meet a floor calibrated on whole weeks.
3. Build islands with the row-number anchor: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) as rn, then week_start - rn * interval '7 days' is constant inside a run of consecutive weeks. Group by that anchor to get each streak's start, end and length.
4. The current streak is the island whose end equals the last whole week; if none does, the account's current streak is zero and that is the interesting case. The longest earlier streak is the maximum length among the remaining islands.
5. Join the renewal filter from the current subscription row and LEFT JOIN the streak summary so an account with no qualifying week at all still appears, rather than vanishing from the risk list precisely because it went quiet.
6. Finish with an operating point. The list is worked by a team with finite capacity, so order it and cut it at that capacity, and say what happens to the accounts below the line.

Follow-up

• The floor of 50 requests was picked for this exercise. How would you calibrate it from data, and what would force a recalibration?
• A regional holiday week drops several accounts below the floor at once. How do you keep that out of the risk list?
• How would you evaluate whether contacting these accounts actually changed renewal, given that coverage is assigned deliberately?

Seven-day activation rate by signup cohort week

Easy · funnel · cohorts · censoring

dim_account carries account_id, created_at, is_internal and is_current; fct_api_request carries account_id, request_at, http_status, api_key_id and traffic_class. Build a signup cohort by the ISO week of created_at over accounts with is_internal = false. An account counts as activated when it issues a request with http_status < 400, a non-null api_key_id and traffic_class <> 'synthetic_monitor' within 168 hours of its own created_at. Return cohort accounts, activated accounts and the rate per week, and exclude any week that has not yet fully elapsed its 168-hour window.

Approach

1. Collapse dim_account to one row per account_id before anything else. It is a type 2 dimension, so a plan or status change gives the same account several rows; filtering to is_current = true is the cheapest correct choice here because created_at does not change across versions.
2. Express the window as interval arithmetic on the timestamptz column: request_at >= created_at AND request_at < created_at + interval '168 hours'. A date-difference of 7 days is a different and wrong condition for accounts created mid-day.
3. Test activation with EXISTS rather than a join to MIN(request_at). EXISTS short-circuits, keeps the cohort at one row per account, and cannot fan out.
4. Aggregate by date_trunc('week', created_at AT TIME ZONE 'UTC'), counting accounts and activated accounts, and divide as a ratio of counts.
5. Drop unreportable weeks: the last account in a cohort week is created just under week_start + 7 days, so the week is only complete once now() >= week_start + interval '14 days'. Without that filter the newest week always looks like a regression.

Worked solution (20 min)

1. Write the cohort CTE: SELECT account_id, created_at FROM dim_account WHERE is_current AND NOT is_internal AND created_at >= <window start>.
2. Add the activation predicate as a correlated EXISTS over fct_api_request on account_id with the four conditions: http_status < 400, api_key_id IS NOT NULL, traffic_class <> 'synthetic_monitor', and the 168-hour bracket.
3. Group by date_trunc('week', created_at AT TIME ZONE 'UTC'); select count() AS cohort_accounts, count() FILTER (WHERE activated) AS activated_accounts, and the ratio cast to numeric.
4. Add HAVING or an outer WHERE that keeps only weeks where week_start + interval '14 days' <= now().
5. Spot-check one account that activated on hour 167 and one that activated on hour 169 to confirm the boundary is exclusive at the top.

Expected result: One row per ISO cohort week with cohort_accounts, activated_accounts and activation_rate between 0 and 1; the newest row is at least 14 days old, and accounts with no rows at all in fct_api_request appear in the denominator with activated = 0 rather than disappearing.

Verify it yourself

• Summing cohort_accounts across weeks equals the count of non-internal current-version accounts created in the same window.
• Re-running the query a day later leaves every previously reported week byte-identical; any movement means the censoring filter is wrong.
• activated_accounts is never greater than cohort_accounts, and the count of distinct account_id in the activated set matches a separate COUNT DISTINCT over the fact table.

Follow-up

• The median time-to-first-successful-call is more informative than a fixed-window rate. Why can you not compute it from this query, and what estimator does it need?
• How would you separate accounts that never called from accounts that called and got only 4xx responses, and which of those is a product problem?

What were the biggest limitations or blind spots in the dataset you used for this project?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would the conclusions of your presentation change if [Specific Variable] was altered?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

If you had an extra month to work on this project, what would you improve?

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

How do you prioritize tasks when supporting multiple research teams with competing deadlines?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Set the operating point for a capacity-bound renewal-risk worklist

Medium · precision at k · operating point · leakage · selection bias

A customer-success team of six can contact at most 40 accounts a week. You have an account-level renewal-risk score and the tables behind it: fct_subscription_period (account_id, term_end_date, arr_cents, amendment_type, booked_at, is_current), fct_support_ticket (account_id, severity, reopened_count, breached_sla, linked_incident_id, opened_at) and fct_api_request. Define the metric that grades this worklist, set the operating point, and state what happens to accounts below the line. The team will not use a score they cannot explain, so say how that constrains what you build.

Approach

1. Grade the list the way it is used. The team works 40 accounts a week, so the metric is precision at k = 40, not AUC. A global AUC over thousands of accounts rewards correct ordering among accounts nobody will ever call, which is ordering that cannot change an outcome.
2. Restrict the candidate pool before ranking anything: accounts whose term_end_date falls within the next 90 days. An account eleven months from renewal cannot churn in the measurement window, so it can occupy a slot in the list while being structurally incapable of producing the outcome.
3. Report two numbers because they disagree. Precision at 40 counts accounts; ARR captured at 40 counts money. A list that catches many small churns and misses one large one scores well on the first and badly on the second, so state which the team is optimising before the ranking is built.
4. Audit leakage by timestamp, feature by feature. Every feature must carry a timestamp proving it existed before the scoring date: a downgrade amendment's booked_at, a ticket's opened_at, a churn reason code written at close. Drop anything you cannot date, and be specific that reason codes and cancellation amendments are written after the renewal conversation started.
5. Attach a policy to the threshold, because a threshold without one is not a decision. Accounts ranked 41 and below get no contact this week; that is an accepted cost, and the weekly readout must include the churn that landed below the line so the operating point can be revisited rather than assumed. Favour a small inspectable model, because a stronger model the team ignores has an effect of zero.

Worked solution (30 min)

1. Build the weekly candidate set: accounts from the current fct_subscription_period version whose term_end_date falls within 90 days of the scoring date.
2. Assemble features only from rows timestamped before the scoring date: trailing 28-day 5xx rate and successful production request volume from fct_api_request, sev1 and sev2 ticket counts plus reopened_count from fct_support_ticket filtered on opened_at, and seat utilisation from the contract version current at that date.
3. Label the outcome as the absence of a successor fct_subscription_period row with amendment_type in ('renewal','upsell','downgrade') within 45 days of term_end_date.
4. Backtest week by week: take the top 40, and report precision at 40, ARR captured at 40, and the churned ARR that landed below rank 40.

Expected result: A weekly backtest table carrying precision at 40, ARR captured at 40, and churned ARR missed below the line, over a candidate pool restricted to accounts renewing within 90 days. The answer ties the operating point k = 40 to team capacity, states the leakage rule that every feature must carry a timestamp earlier than the scoring date, and accepts explicitly that accounts below rank 40 receive no contact.

Verify it yourself

• No feature may derive from a row timestamped at or after the scoring date; re-run the backtest dropping each feature in turn to confirm none carries post-hoc information.
• A 90-day renewal window on an annual-contract base should hold roughly a quarter of paying accounts; a pool the size of the whole base means the term_end_date restriction is not being applied.
• Churn must be dated at term_end_date, not at the date the record was updated, or losses shift into whichever week the operations team did its paperwork.

Follow-up

• Coverage is assigned by this list, so contacted and uncontacted accounts differ systematically in exactly the risk signal that drives the outcome. How would you ever measure whether the outreach worked?
• Annual contracts produce few churn events per quarter. At what point is the honest answer that there is not enough signal to model this, and what would you deliver instead?

Billable units per account jumped while nothing shipped

Medium · retries · idempotency · guardrail metrics

Billable units per paying account rose 22 percent month over month with no pricing or packaging change. From fct_api_request (account_id, endpoint, http_status, is_retry, idempotency_key, traffic_class, billable_units, request_at) and fct_usage_daily (account_id, sku_code, usage_date, billable_quantity), determine how much of the rise is delivered value and how much is duplicated work. Deliverable: the decomposed figure, the accounts it concentrates in, and a recommendation on whether to report the 22 percent at all.

Approach

1. Check the guardrail before the headline: compute the customer-visible server error rate per account for both months, numerator http_status >= 500 and denominator excluding synthetic_monitor and load_test. Metered volume rising alongside an error rate is the known failure mode in this domain.
2. Deduplicate logical work by counting billable_units once per (account_id, idempotency_key) at the first successful request rather than once per row. Rows with a null idempotency_key cannot be deduplicated, so report their share as an explicit uncertainty band instead of assuming they are all unique.
3. Split by traffic_class before interpreting anything, because one change to a continuous-integration configuration can multiply request volume overnight without a human deciding anything about the product.
4. Test concentration: compute the per-account distribution of the increase and its top-decile share. A rise carried by a few accounts scaling one batch job is a different finding from a broad shift and gets a different recommendation.
5. Reconcile against fct_usage_daily for the same accounts and dates, and be ready to explain the expected gap in two sentences: request rows include retries and failures carrying zero billable_units, and the usage table restates after first write.

Follow-up

• A client retrying a request the server already completed produces duplicate billed work. What protocol or product change removes that, and what would you measure to confirm it worked?
• If the duplicated volume was genuinely invoiced, what should finance do, and how does that change what belongs in the metric?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Why are you interested in applying data science to public policy and research at AEI?

Medium · behavioural and stakeholder questions

Approach

1. Close with what you would do differently, concretely.
2. State the situation in two sentences and spend the rest on your reasoning.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• What would you do differently if you ran that project again?
• What did you decide not to do, and why?

Allocate one analyst week across three competing requests

Easy · prioritisation · stakeholders · scoping

Three requests arrive the same morning and you have one week. Finance wants per-account gross margin from fct_usage_daily for a pricing review in three weeks. Sales wants a renewal-risk list for accounts with term_end_date inside 60 days. A product manager wants an experiment readout for a decision being taken on Thursday. Produce your allocation with hours attached, what you say to whoever receives less, and one thing you refuse to do this week, with the reason each decision is defensible to the person it costs.

Approach

1. The interviewer is probing whether you prioritise on decision timing and reversibility or on who asked most forcefully. Sort by the date each decision is actually taken and by what the default outcome is if nothing arrives.
2. Apply that sort concretely. The Thursday readout has a hard irreversible deadline and no value afterwards. The pricing review has three weeks of slack. The renewal list has a rolling deadline set by term_end_date, so part of it is urgent this week and the rest is not, which means it can be split rather than deferred whole.
3. Find the cheapest sufficient version of each request rather than the full version. The readout goes in full. The renewal list ships as a filtered query over renewal-eligible accounts ranked by two inspectable signals rather than as a model. The margin work is scoped to the accounts that dominate the pricing decision, since revenue is heavily skewed and the tail will not change the conclusion.
4. Make the trade visible in one written note to all three at once, with dates. Telling each person separately that they are the priority is how an allocation becomes a credibility problem.
5. Refuse something explicitly and say why. The model version of the renewal list is the usual candidate, because it cannot be evaluated without a holdout nobody has agreed to yet, and building it this week forecloses that.
6. Leave slack. A plan with none is a plan to miss the one deadline that cannot move.

Follow-up

• The sales leader escalates to your manager. What did you already do that makes that a short conversation?
• Which of the three deadlines would you push back on, and what exactly would you ask for?
• What would you change about how these requests reach you so next week is not the same?

Walk through an analysis you later discovered was wrong

Easy · data quality · error ownership · metering

Six weeks ago you reported that consumption fell 9 percent in the last week of the month, and a team spent a sprint investigating the cause. The fall was an artefact: rows in fct_usage_daily land late and are restated in place, and you queried before the tail had settled. Describe how you found the error, what you told the people who acted on it, and the control you put in place so this class of mistake cannot reach a dashboard again. Be specific about how the settling window was measured.

Approach

1. The interviewer is probing whether you self-report errors before someone else finds them, and whether your fix is structural rather than a promise to be more careful. Say plainly that the number was wrong and that a sprint was spent on it, before describing any diagnosis.
2. Establish the artefact quantitatively instead of asserting that data lands late. For each usage_date, compare the total as of first_written_at against the settled total and read the settling time off that curve, for example 97 percent of final by day three and 99.5 percent by day five.
3. Correct the record the same day, in the channel the original number went out in, to the same audience. The cost of the wasted sprint belongs in the correction, not in a footnote.
4. Make the fix structural: exclude a trailing lag window from every reportable figure, and make the reporting view return no rows inside that window rather than returning partial ones. A dashboard that shades unsettled days still gets read as a decline.
5. State what generalises. Any fact table restated in place has this failure mode, so the guard belongs at the source rather than on the one dashboard that embarrassed you. A strong answer ends with the class of error closed; a generic one ends with a lesson learned.

Follow-up

• How did you choose the completeness threshold behind the lag window, and what would make you recalibrate it?
• What did you say to the team that lost the sprint, and what did they say back?
• Is there a legitimate case for showing the unsettled tail at all, and to whom?