Interview guides · ABC Education

ABC Education Data Scientist Interview Questions & Guide 2026

Causal inference drills for ABC Education data scientist prep: diff-in-diff, matching, instruments, and the limits of observed data.

The 60-second brief

The role of a Data Scientist at ABC Education is pivotal in driving data-driven decision-making and enhancing educational products through analytical insights. As a Data Scientist, you will leverage statistical methods, machine learning algorithms, and data visualization techniques to transform raw data into actionable insights that can influence product development and strategy. This position is central to understanding user behavior, optimizing educational pathways, and ultimately improving learning outcomes for students.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Align cohorts to term weeks, not calendar weeks
Model seat activation as the renewal leading indicator
Separate mastery signals from raw usage exposure

Overview

In your role, you will engage closely with product teams to analyze user data and identify trends that can inform product enhancements or new feature developments. The complexity of the educational landscape, combined with the scale at which ABC Education operates, presents unique challenges and opportunities. You will be at the forefront of innovation, using data to shape the educational experiences of thousands of users, thereby making a significant impact on the company's mission to provide accessible, high-quality education.

The ABC Education interview process

ABC Education candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Resume Screening

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Online Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Technical Interviews

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Final Round

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Consent and age rules silently truncate the data, not just the joins.

Where age_gated is TRUE, behavioural logging and cross-system joins are restricted, so those learners are missing from the very tables used to compute engagement. An analysis that simply inner-joins will produce a population skewed to older learners and self-serve accounts while appearing complete. Check the age_gated share of every population you report on, and state it.

Adaptive item selection holds observed accuracy flat by construction.

A selector targeting a fixed success probability, say 70%, will keep measured first-attempt accuracy near 70% whether learners are improving or not, because as theta rises the engine simply serves harder items. Reporting 'accuracy improved 3 points' on adaptive content therefore usually means the selector got more conservative, not that anyone learned more. Accuracy is only interpretable on a fixed form with unchanged item versions, which is why the first-attempt accuracy metric above carries both restrictions.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

Write a Python function to calculate the Pearson correlation coefficient.

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Say what the estimate is of, and over what population it generalises.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

Can you discuss a recent project where you applied machine learning?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Describe the process of building a machine learning model from start to finish.

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Simulate the gain a bottom-quartile selection manufactures

Medium · regression to the mean · simulation · measurement error · selection bias

Observed quiz scores are a true score plus independent noise, both normal, with reliability r equal to true-score variance over observed-score variance. Simulate 200,000 learners, standardise the pretest to mean 0 and SD 1, select the bottom quartile on the pretest, and measure the mean pretest-to-posttest change with no intervention at all. Report the manufactured gain for r in {0.5, 0.7, 0.9, 1.0}, and give the closed form your simulation should reproduce. Use numpy only, no statistical packages.

Approach

Parameterise by r directly: draw the true score with variance r and each noise term with variance 1 - r, so the observed score has unit variance and reliability exactly r by construction. Draw one true score and two independent noise terms per learner.
Select on the pretest and only the pretest. Selecting on the true score, or on the average of the two measurements, removes the correlation between selection and the pretest's own noise, and the artefact disappears. That sensitivity is the lesson.
Compute the mean of posttest minus pretest over the selected group. The true score cancels in the difference, so whatever remains is entirely the difference of two noise draws conditioned on the first being low.
Check against the closed form. For jointly normal standardised scores, E[posttest | pretest] = r * pretest, so the expected gain is (1 - r) * |E[pretest | bottom quartile]|, and E[pretest | bottom quartile] = -phi(z_0.25) / 0.25 = -1.2711.
Extend the script with a control group selected by the identical rule from an untreated population and show the difference of differences returns to zero. That is the design fix you would actually propose, not a caveat in a footnote.

Worked solution (25 min)

For each r: t = rng.normal(0, sqrt(r), n); e1, e2 = rng.normal(0, sqrt(1 - r), n) twice; pre = t + e1; post = t + e2.
sel = pre <= np.quantile(pre, 0.25); gain = (post[sel] - pre[sel]).mean().
Compare gain to (1 - r) * 1.2711 and print the absolute difference.
Repeat the whole loop over the four r values and tabulate simulated against closed form.
Add the untreated control arm selected by the same rule and report the difference of differences.

Expected result: Mean manufactured gain close to (1 - r) * 1.2711 standard deviations: about 0.636 at r = 0.5, 0.381 at r = 0.7, 0.127 at r = 0.9, and 0 at r = 1.0, each within roughly 0.01 at 200,000 draws. The difference of differences against an identically selected control is zero within Monte Carlo error at every r.

Verify it yourself

r = 1.0 must give a gain indistinguishable from zero; any systematic positive value means noise leaked into the true-score draw.
Selecting the top quartile instead gives a mirror-image loss of the same magnitude.
The Monte Carlo standard error on the selected group of about 50,000 is roughly 0.0045 at r = 0.5, so a discrepancy above about 0.015 is a bug, not sampling noise.

Follow-up

A published case study reports large gains specifically for learners who started in the bottom quartile. What do you ask for before believing any of it?
The posttest is twice as long as the pretest, so its reliability is higher. Does the manufactured gain grow or shrink, and why?
Give a design that measures a real effect on exactly this selected population without a randomised control.

Explain how you would use SQL to aggregate user data by month.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

What breaks if events arrive late or out of order?
How does the query change if the join becomes one-to-many?

How would you optimize a SQL query that is running slowly?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Pace adherence at term-week six from activity history

Hard · window functions · running aggregates · term alignment

Compute the share of active enrollments at or ahead of scheduled pace at the end of term-week 6, from fct_enrollment(enrollment_id, learner_id, course_id, section_id, term_id, enrolled_at, scheduled_start_date, due_date, units_total, units_completed, status), fct_lesson_activity(enrollment_id, content_item_id, completion_status, started_at, activity_date) and dim_term(term_id, term_start_date, term_end_date, term_length_weeks). units_completed is maintained incrementally and reflects now, not week 6. Rebuild the as-of count from activity, align each enrollment to its own term calendar, and handle NULL scheduled_start_date and NULL term_id explicitly.

Approach

Refuse the stored column for this question. units_completed is a current snapshot, so using it scores every enrollment on today's progress and labels the result week 6. That single substitution is the difference between a metric and a number that drifts upward every time you re-run it.
Define a completed unit from activity: take MIN(started_at) per (enrollment_id, content_item_id) over rows with completion_status = 'completed', so a learner re-opening finished content does not add a unit. A COUNT over that deduplicated set ordered by first-completion time gives units completed as of any instant.
Set the boundary per term, not per calendar: week 6 ends at term_start_date + 42 days for that enrollment's term_id. Enrollments with NULL term_id cannot be week-aligned at all; exclude them and report the count rather than defaulting them into somebody else's calendar.
For NULL scheduled_start_date, choose between the term start and the learner's first activity, state the choice, and measure how many enrollments it touches. If the share is large, the pace metric is being defined by a data gap rather than by pacing.
Apply the threshold as rebuilt_units >= units_total * (6.0 / term_length_weeks), guarding units_total > 0 and excluding terms shorter than 6 weeks instead of letting the ratio exceed 1.
Be honest about the denominator: fct_enrollment.status has no effective dating, so active at the end of week 6 is not reconstructible from this schema. Exclude enrollments whose enrolled_at is after the boundary, use current status as the stated approximation, and say which direction the survivor bias runs.

Worked solution (40 min)

Build the first-completion CTE with MIN(started_at) per (enrollment_id, content_item_id) and assert one row per pair.
Join dim_term to get each enrollment's week-6 boundary timestamp, and count rows lost to NULL term_id.
Count completions at or before the boundary per enrollment, then apply the pace threshold.
Aggregate to (term_id, course_id) and attach the three exclusion counts.
Re-run the query with a different CURRENT_DATE to prove the answer does not move.

Expected result: One row per (term_id, course_id) with active_enrollments, on_pace_enrollments, pace_rate, plus separate counts of enrollments excluded for NULL term_id, units_total = 0, term_length_weeks < 6, and enrolled_at after the boundary.

Verify it yourself

Rebuilt units at week 6 is at or below stored units_completed for every enrollment. A row above it means the stored column is stale or the completion dedup is admitting re-opens.
Running the query again a week later returns identical numbers. Any movement means a dependency on now survives somewhere in the logic.
COUNT(DISTINCT (enrollment_id, content_item_id)) in the completion CTE equals its row count.
The exclusion counts plus the reported denominator sum to the raw enrollment count for the term.

Follow-up

status has no history. What would you add to the model to make active at the end of week 6 reconstructible, and until then does the current answer over- or under-state pace?
Two institutions on different calendars now compare their pace rates. What is still not comparable after term-week alignment?
A holiday lands in week 4 for one institution and week 5 for another. Does your denominator punish either, and how would an instructional-day calendar change the threshold?

Given a dataset with user engagement metrics, how would you determine which features to prioritize for improvement?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you approach analyzing the effectiveness of a new educational feature?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize competing tasks when working on multiple projects?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

What are some common techniques for feature selection?

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Pick randomisation units for two features with different carryover

Medium · interference · switchback · cluster randomisation

Two features need tests. Feature A shows each learner the peer explanations other learners in the same class section submitted. Feature B changes the turnaround target for rubric_model scoring from four hours to fifteen minutes, served from one shared scoring queue. Product wants both learner-randomised. Using fct_lesson_activity.is_assigned, fct_assessment_response.scoring_mode and fct_enrollment.section_id, propose a randomisation unit for each, name the interference each one suffers, and state the condition that makes a switchback design valid for one of them and invalid for the other.

Approach

Feature A leaks by construction. A treated learner's explanation renders to control learners in the same section, so the control arm is partly treated and the estimate is attenuated toward zero by an amount you cannot recover. Randomise at section_id, or at the school when sections share an instructor or a device cart.
Price the cluster design before agreeing to it. At section size 25 and intraclass correlation 0.15 the design effect is 1 + 24 x 0.15 = 4.6, so budget roughly 4.6 times the learner count a learner-randomised test would have needed.
Feature B's interference runs through the shared resource, not the classroom. Serving a fifteen-minute target to half the traffic changes queue depth for the other half, so unit-level randomisation is contaminated at the queue regardless of who the units are.
Apply the switchback condition: the treatment effect must not persist past the switch. Feature B acts on scoring latency, which resets the moment the config flips, so a time-blocked switchback over the whole queue is valid. Randomise the sequence and analyse at the block level, targeting at least 30 blocks per arm because blocks, not learners, are the unit.
Feature A fails that condition. Seeing a peer explanation changes what a learner knows, and knowledge does not revert when the flag does, so every post-switch control block is permanently contaminated. Switchback is the wrong tool for anything that changes mastery state.
Write a washout into Feature B's plan: discard the first block after each switch while the queue drains, and confirm the drain time from the observed scored_at minus submitted_at distribution rather than assuming it.

Worked solution (30 min)

For each feature, write the exposure mapping explicitly: whose outcome changes when one unit is treated.
For Feature A, set the unit to section_id and compute the sample multiplier 1 + (m - 1) x rho = 4.6 at m = 25, rho = 0.15.
For Feature B, fix a six-hour block, count the blocks available in the test window, and confirm at least 30 blocks land in each arm.
Specify analysis up front: section-clustered standard errors for A, block-level means with a randomisation-inference p-value for B.
Add the washout rule for B and a spillover diagnostic for A comparing control-arm outcomes in sections with high versus low treated share.

Expected result: Feature A: cluster-randomise at section_id, budget about 4.6 times the learner count, analyse with section-clustered errors. Feature B: switchback the whole shared queue on six-hour blocks with a randomised sequence, at least 30 blocks per arm, discarding the first block after each switch. Switchback is valid for B because latency resets at the flip and invalid for A because learning does not.

Verify it yourself

Check global application per block, not per learner: every fct_assessment_response row scored inside one block must carry a single scoring_mode. Two modes in one block means the flip did not reach the whole queue, or the block boundary is offset from the switch instant, and that block is unusable. Learners are not a diagnostic here. A learner whose submissions all fall inside one six-hour block legitimately appears in exactly one arm, so single-arm learners are expected under a correct switchback and are not evidence of anything.
For Feature A, the treated share inside a control section must be zero. A non-zero share means the flag is leaking and the design has already failed.
The design-effect multiplier must appear in the sizing document before launch, not in the post-mortem after the test reads null.

Follow-up

For Feature B, how do you choose the switch interval, and what does shortening it cost you?
Feature A is section-randomised, but two sections in the study share an instructor who teaches both. What do you do about that pair?
How would you measure the size of Feature A's spillover rather than only designing around it?

Verified mastery rate fell six points with usage flat

Hard · mastery · censoring · decomposition · clustering

Verified mastery rate fell from 0.72 to 0.66 month over month while active learners and submissions per learner held flat. Tables: fct_assessment_response (learner_id, objective_id, content_item_id, content_version_no, attempt_no, submitted_at, is_correct, mastery_state_after, mastery_prob_after) and dim_content_item (content_item_id, version_no, irt_b, irt_a, calibration_n). The metric carries a 30-day reporting lag and requires a delayed retention check at least 7 days after the mastery crossing. Establish whether learning regressed. Deliverable: a ranked list of causes, each eliminated by a named check.

Approach

Split the rate into its two counts and chart them separately: crossings with an eligible delayed check available, which is the denominator, and crossings that passed it, which is the numerator. A rate that moves while its denominator is also moving is a composition question before it is a learning question.
Test censoring next, because it is cheap and it invalidates everything downstream. Recompute the prior month's value as it stood at the same maturity as the current month. If the prior month at equal maturity also read 0.66, there is no regression, only an early read against a fully matured comparison.
Test the decision rule. mastery_prob_after is a posterior from the knowledge-tracing model, so compare its distribution at the crossing response across the two months rather than only counting crossings. A retrained model or a moved threshold changes which learners enter the numerator at all and shows up as a shifted posterior, not as a usage change.
Test the instrument. Compute the response-weighted mean irt_b of items used in delayed checks in each month, restricted to items with calibration_n at or above 200, and compare irt_a alongside it. A harder or differently discriminating retention form lowers the pass rate with no change in what learners know.
Segment only once the artefacts are excluded, and segment cluster-aware: cut by grade_band, locale and is_assigned, then check how concentrated the move is across section_id. Outcomes correlate within sections through a shared instructor, schedule and device fleet, so a handful of sections can carry a headline move.
If a residual survives, size it with the section as the unit of analysis and report an interval that carries the design effect, roughly 1 + (m - 1) * rho for near-equal sections, rather than a learner-level interval that will overstate certainty.

Follow-up

If the drop survives every check, how would you design the follow-up study to establish cause, and at what unit would you randomise?
The delayed retention check is itself served by the adaptive engine. What does that do to the verified mastery definition, and how would you fix it?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Getting an analysis acted on

Work that nobody used is a common and unflattering pattern in data careers, and interviewers probe for it. Have a story about an analysis that changed a decision, and be specific about how you got it in front of the person who could act. Also have one about work that went nowhere, with your reading of why.

Describe a situation where you had to work with a difficult team member. How did you handle it?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

State the impact of your last project without inflating it

Hard · impact measurement · attribution · causal reasoning

Pick one project from the past year and account for its impact as if the listener could audit every number. Cover the metric you claim to have moved and its exact definition, how much of the movement is attributable to your work, what else was changing at the same time, what the counterfactual was and where it came from, and what you would have needed to measure at the start to make the claim clean. If the effect is not separable, say so and say what you would do differently. Probed: attribution discipline applied to your own work.

Approach

Name the metric by its definition rather than its label. 'Verified mastery rate' means nothing without its numerator, denominator, window, and the reporting lag the delayed retention check forces.
Separate the three claims usually merged into one: the metric moved, your work moved it, and the movement was worth what it cost.
State the counterfactual explicitly and say what produced it: a holdout, matched sections, or a pre-period trend. If it was a before-and-after with no control, say that in those words.
List the co-occurring changes with dates: a term boundary, a roster sync at scale, a pricing change, another team shipping into the same surface. In this domain the academic calendar alone routinely dwarfs a product effect.
Convert the shortfall into a specific design artefact: the holdout you would have reserved, the instrumentation you would have shipped before the change, the analysis plan you would have written first.

Follow-up

What fraction of the observed movement would you defend under cross-examination, and on what evidence?
Your project shipped in term-week one. How does that change what you can claim?
If the effect is genuinely not separable, was the project worth doing?

Scope a vague request to investigate falling engagement

Easy · scoping · metric definition · stakeholder management

A curriculum director messages you: engagement is down in middle school, can you dig into it. You have dim_learner, fct_lesson_activity, fct_assessment_response, fct_enrollment and dim_term. Before writing any query you get one fifteen-minute conversation. Deliverable: the questions you ask in that conversation, the written scope you send afterwards, and the first three things you compute, with the reason each comes before the others. Probed: whether you convert an ambiguous request into a decision and a single metric before touching data, rather than after.

Approach

Find the decision behind the request first: what the director would do differently depending on the answer, and by when. That sets how much of this analysis is worth doing at all.
Pin the four ambiguities that make 'engagement is down' uncomputable: which population (grade_band = '6_8', which orgs), which metric (active_seconds, activity days, scored submissions), which comparison window, and against which baseline.
Ask where the belief came from. 'Down' is almost always a chart someone saw, and reproducing that exact chart is faster than building a new one and then arguing about why the two differ.
Check the calendar before believing the shape at all: a decline spanning a term boundary is the default explanation in this domain, so term-week alignment through dim_term is the first computation, not a later refinement.
Send the scope back in writing with an explicit out-of-scope list and a time box, so follow-on questions arrive as new requests instead of as scope creep.

Follow-up

The director cannot name the decision this feeds. Does that change whether you do the work, and how much of it?
Your term-aligned version shows no decline at all. What do you send, and to whom first?