Interview guides · Eab (Education Advisory Board)

Eab (Education Advisory Board) Data Scientist Interview Questions & Guide 2026

SQL practice for Eab (Education Advisory Board) data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

The role of a Data Scientist at Eab is pivotal in harnessing data to drive insights that enhance educational outcomes. You will be at the forefront of transforming raw data into actionable strategies that inform decision-making across various educational institutions. Your work will directly impact the development of innovative products and solutions aimed at improving student engagement and institutional effectiveness.

Allocate prep to your weakest link rather than your favourite topic. Of the three things that usually gate the outcome (SQL that is correct under messy joins, sound reasoning about experiments, and structured framing of an open-ended problem), candidates tend to over-invest in modelling theory and under-invest in framing.

Focus areas

Separate mastery signals from raw usage exposure
Read item difficulty and discrimination before trusting accuracy
Align cohorts to term weeks, not calendar weeks

Overview

As a Data Scientist, you will collaborate with cross-functional teams, including product management, engineering, and client services, to analyze large datasets, develop predictive models, and contribute to research initiatives. This role requires not only technical prowess in statistical analysis and machine learning but also a deep understanding of the educational landscape, enabling you to effectively address complex challenges faced by educational institutions. Expect to engage in high-stakes projects that shape the future of education, making your contributions critical to the mission of Eab.

The Eab (Education Advisory Board) interview process

Eab (Education Advisory Board) candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What to demonstrate

Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Technical Assessments

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Evaluations

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
The counterfactual: what would have been done without your analysis, and why that default was worse
How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Where candidates lose points

The academic calendar creates structural breaks that look like product effects.

Term start, exam weeks, holidays, and summer each shift usage by amounts far larger than any feature change. A launch timed to week one of a term will show a large lift that is entirely calendar, and a launch in the last week of term will show a collapse. Comparisons must be term-week aligned through dim_term, and any pre/post analysis over a term boundary needs a comparison group living on the same calendar.

Pre/post gain studies that select on low pretest scores manufacture improvement.

Any measure with reliability below 1 produces regression to the mean, so a group chosen for scoring in the bottom quartile will score higher on retest with no intervention at all. The apparent gain scales with measurement error, which for a short quiz is large. The fix is a control group selected by the identical rule, or a design that models the pretest as a covariate rather than as a selection filter.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Optimising accuracy on a heavily imbalanced target

State the base rate first, then choose the metric from the relative cost of a false positive against a false negative: precision and recall at the operating threshold, PR-AUC, or expected cost. At a 1 percent positive rate, predicting the majority class for everyone scores 99 percent accuracy and is worthless.

Questions & practice

Describe a machine learning algorithm you have implemented and the results you achieved.

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Implement a binary search algorithm in your preferred programming language.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Permutation test with sections as the randomisation unit

Hard · permutation test · clustering · randomisation inference · design effect

You have outcomes with learner_id, section_id, arm ('treatment' or 'control'), and mastery_events, an integer count over the study window. Assignment was made at section level, so every learner in a section shares an arm. There are 48 sections of roughly 25 learners. Write a permutation test from scratch in numpy that respects the clustering, using the difference in learner-weighted arm means as the statistic. Report the observed statistic, a two-sided p-value, and the number of permutations. No t-test, no statistics package.

Approach

Collapse to a section table first: section_id, arm, n_learners, sum_mastery. The learner-weighted difference in means is a function of those two columns only, and computing it from the section table makes the permutation unit structurally obvious rather than a comment in the code.
Fix the number of treated sections at the observed count and permute arm labels over the 48 sections. Permuting learner labels destroys the within-section correlation and reproduces exactly the false precision that section-level assignment exists to avoid.
Vectorise: build a (B, 48) boolean assignment matrix, then get treated numerators and denominators with two matrix products. That runs 20,000 permutations in well under a second, which matters because you want enough replicates for the p-value to be stable in its third digit.
Compute the statistic as sum of treated sums over sum of treated n, minus the control equivalent. This is a ratio of sums, not a mean of section means; the two differ whenever sections vary in size, and you should say which one the study was powered on.
Two-sided p-value is (1 + count of |T_perm| >= |T_obs|) / (1 + B). Both the numerator and denominator offsets are required for validity at finite B; without them a p-value of exactly zero is reportable, and it is wrong.
Diagnose the clustering rather than assume it: estimate the intraclass correlation from the section table and check that the permutation null SD exceeds the naive learner-level standard error by roughly sqrt(1 + (m - 1) * rho). At m = 25 and rho = 0.15 that factor is about 2.1 on the standard error, or 4.6 on variance.

Worked solution (35 min)

sec = outcomes.groupby('section_id').agg(arm=('arm','first'), n=('learner_id','size'), s=('mastery_events','sum')).
k = (sec.arm == 'treatment').sum(); T_obs = s[treat].sum()/n[treat].sum() - s[ctrl].sum()/n[ctrl].sum().
Build assign = np.zeros((B, 48), bool); for each row set k randomly chosen columns True via rng.permuted on a tiled index array.
num_t = assign @ s; den_t = assign @ n; T_perm = num_t/den_t - (s.sum() - num_t)/(n.sum() - den_t).
p = (1 + (np.abs(T_perm) >= abs(T_obs)).sum()) / (1 + B); report T_obs, p, B, T_perm.std(), and the estimated rho.

Expected result: An observed learner-weighted difference in arm means, a two-sided p-value computed as (1 + exceedances) / (1 + B) with B at least 10,000, and a permutation null whose standard deviation is materially larger than a naive learner-level standard error whenever the estimated intraclass correlation is above zero.

Verify it yourself

Replace mastery_events with pure noise and rerun over many simulated datasets: the p-values must be roughly uniform on [0, 1].
Add a constant to every learner in BOTH arms: T_obs, every permuted statistic and the p-value are identical to the unshifted run, because the statistic is a difference of arm means. Shifting only the treated arm is not an invariance check: T_obs does rise by exactly that constant, but the loop relabels the same shifted data, so the shift travels with the originally-treated sections into whichever arm each permutation hands them and the null widens too. Two sections, k = 1, both outcomes 0 give a null of {0, 0}; add 5 to the treated section and the null becomes {+5, -5} while T_obs is +5, so the p-value is unmoved.
Permuting learners instead of sections must give a visibly narrower null. If it does not, rho is near zero for this outcome and you should report that rather than assume clustering.

Follow-up

48 sections, 24 per arm, baseline and variance as observed. What effect can this design actually detect at 80 percent power, and would you have run it?
Sections range from 8 to 60 learners. Does that change which statistic you should permute?
The institution insists on assigning by school rather than by section. What changes in the test, and what changes in the power?

Write a function to find the longest common prefix string amongst an array of strings.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Given two lists, write a function to return the intersection of the lists.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Find enrollments with no scored work using anti-joins

Medium · null semantics · anti-join · correctness

You have fct_enrollment(enrollment_id, learner_id, course_id, section_id, org_id, status, enrolled_at, due_date) and fct_assessment_response(response_id, learner_id, enrollment_id, submitted_at, scored_at), where fct_assessment_response.enrollment_id is NULL for work done outside any enrollment. Return every active enrollment with a due_date in the next 14 days that has no scored response attached, for an instructor at-risk list. Write it as an anti-join. A colleague's NOT IN version returns zero rows against production but the correct count against their small test extract. Explain exactly why, and fix it.

Approach

Name the mechanism precisely: NOT IN expands to a conjunction of inequality comparisons, and x <> NULL is unknown rather than false, so once a single NULL enters the subquery list the predicate can never evaluate to TRUE and the result is empty. The test extract contained no work outside an enrollment, so it contained no NULLs, so it passed.
Fix with NOT EXISTS and a correlated equality on enrollment_id. It tests row existence rather than list membership and is unaffected by NULLs elsewhere in the column. LEFT JOIN with an IS NULL filter is equivalent and usually plans the same; NOT IN with an added IS NOT NULL guard also works but leaves the landmine armed for the next reader.
Decide what no scored work means before writing it. No response at all and responses that exist but are unscored are different at-risk lists, and rubric lag puts already-submitted work in the second. Return both counts rather than collapsing them.
Filter status = 'active' and due_date BETWEEN CURRENT_DATE AND CURRENT_DATE + 14, comparing DATE to DATE. Casting a DATE against a timestamp midnight quietly drops the final day.
Guard the complement in the other direction: joining enrollments to responses and counting enrollments fans out one enrollment with 40 responses into 40 rows, so the has-scored-work list needs COUNT(DISTINCT enrollment_id).

Worked solution (20 min)

Run SELECT COUNT(*) FROM fct_assessment_response WHERE enrollment_id IS NULL. That one number is the whole explanation.
Write both versions side by side over the same window and show the row counts differ.
Write the NOT EXISTS version with the response_count and scored_response_count columns attached.
Verify the partition of the due-date window: at-risk count plus distinct enrollments with scored work equals total active enrollments in the window.

Expected result: One row per at-risk enrollment_id with learner_id, section_id, due_date, response_count and scored_response_count, where NOT EXISTS returns a non-zero set and NOT IN returns zero rows on data containing NULL enrollment_id.

Verify it yourself

COUNT(*) of responses with NULL enrollment_id is positive, which is the direct cause of the colleague's empty result.
At-risk count plus COUNT(DISTINCT enrollment_id) of enrollments with at least one scored response equals total active enrollments in the due-date window.
Swapping NOT EXISTS for LEFT JOIN with IS NULL returns an identical row set.

Follow-up

Produce the complement list, the enrollments that do have scored work, without double counting, and state the grain of your join.
An enrollment whose learner submitted work with enrollment_id NULL has done the work but is not attached to it. How would you attribute those rows, and what does attributing by learner and course risk?
fct_assessment_response is partitioned by date. How would you bound the scan without changing the answer?

How do you prioritize your tasks when managing multiple projects?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

If tasked with improving the performance of an educational product, what data would you need to collect?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Describe how you would optimize a function that processes a large dataset.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you approach analyzing student retention rates for a university?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you explain a complex statistical concept to a non-technical audience?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer would contain.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Apply CUPED and stratification to an under-powered clustered test

Medium · cuped · stratification · variance reduction

A section-randomised test is under-powered and the team cannot get more sections. Eight term-weeks of pre-period fct_assessment_response and fct_lesson_activity exist before assignment. The learner-level correlation between pre-period and in-test weekly mastery events is 0.55; at section-mean level it is 0.72. Propose a variance-reduction plan. Compute the MDE multiplier CUPED buys at each level, say which level you apply it at and why, and handle the learners with no pre-period because roster_sync created their accounts after assignment.

Approach

State the estimator: Y_adj = Y - theta x (X - mean(X)), where X is the pre-period covariate and the binding precondition is that X is measured strictly before assignment. Fit theta = Cov(Y, X) / Var(X) on the in-test data pooled across both arms. That is standard CUPED and it is the only option here: Y is the in-test outcome, so theta cannot be computed from pre-period data alone. Pooling is what keeps it honest, because a pre-assignment X has the same distribution in both arms and the adjustment therefore cannot absorb the treatment effect. Var(Y_adj) = (1 - r^2) x Var(Y), so the MDE scales by sqrt(1 - r^2).
Apply it at the unit of analysis. Under section randomisation that is the section mean, so the 0.72 section-level correlation is the one that matters. It exceeds the learner-level 0.55 because averaging removes idiosyncratic learner noise while preserving the section's persistent level.
Stratify the randomisation itself rather than only adjusting afterwards: block sections on pre-period mastery-event quartile crossed with grade_band, then randomise within block. With 300 sections per arm, blocking also protects against a bad draw, which post-hoc CUPED cannot undo.
Handle the no-pre-period learners without imputing a value that would drag theta. Place them in their own stratum with X set to that stratum's mean, which leaves them with zero adjustment and no bias, and report their share alongside the result.
Hold the two rules that actually protect the estimate: the covariate must be measured strictly before assignment, and theta must be fitted once on the pooled arms rather than separately within each arm. A post-assignment covariate can itself be moved by treatment, so adjusting on it subtracts part of the effect you are measuring. Arm-specific thetas make the adjustment differ by arm, which reintroduces the same bias by another route. Fitting theta on the data it adjusts carries a finite-sample bias of order 1 / n_clusters; at 300 sections per arm that is negligible, and if you want it gone entirely, estimate theta from an earlier pre-period regression on data outside the test window and accept a slightly lower r.

Worked solution (30 min)

Learner level: sqrt(1 - 0.55^2) = sqrt(0.6975) = 0.835, a 16.5% smaller MDE.
Section level: sqrt(1 - 0.72^2) = sqrt(0.4816) = 0.694, a 30.6% smaller MDE.
Convert to sample equivalence: 1 / 0.4816 = 2.08, so section-level CUPED is worth roughly doubling the number of sections.
Build the blocks as four pre-period quartiles crossed with grade_band, randomise within block, and persist the block id for the analysis.
Fit theta once on the pooled section-level in-test data, form Y_adj, then estimate the effect at section level with block fixed effects and compare the adjusted and unadjusted estimates and intervals.

Expected result: Apply CUPED at the section mean: multiplier 0.694, a 30.6% MDE reduction, equivalent to about 2.08 times the sections. Learner-level CUPED delivers only 0.835 and is the wrong level under section randomisation. Block on pre-period quartile crossed with grade_band before randomising rather than relying on adjustment alone.

Verify it yourself

CUPED should narrow the interval and move the point estimate only by about as much as the pre-period imbalance justifies. A large shift means the covariate is imbalanced across arms or was not measured strictly before assignment; it does not mean theta was fitted on in-test data, which is where theta is supposed to be fitted.
Mean pre-period covariate should be close across arms before adjustment. A large gap is itself a randomisation finding and needs explaining before the adjusted number is trusted.
The share of learners with no pre-period must be reported. Above roughly 10%, a meaningful part of the result is carrying no adjustment at all.

Follow-up

Your pre-period window spans a holiday for half the sections. What does that do to the correlation and to theta?
Can you stack CUPED on top of stratification, and does the second one still buy anything once the first has run?
The section-level correlation is 0.72 overall but 0.45 for sections whose instructor changed between periods. How do you handle that subgroup?

Pace adherence at term-week six rose eleven points

Medium · definition-change · survivorship · pacing

Pace adherence at term-week 6 rose from 48 percent to 59 percent between two consecutive terms, and the dashboard query changed owners last month. Tables: fct_enrollment (enrollment_id, learner_id, course_id, units_completed, units_total, status, scheduled_start_date, due_date, enrolled_at) and dim_term (term_id, term_start, term_length_weeks). Nothing shipped to pacing in that period. Decide whether learners are pacing better, and reconcile the old and new numbers to the point.

Approach

Diff the definitions before touching the data. Recover the prior query and compare it line by line against the current one on four axes: the denominator population, the status filter, how term-week 6 is derived, and what happens to enrollments with NULL scheduled_start_date.
Recompute both definitions on the same term and the same enrollment set. If the difference does not account for the full eleven points there is a second cause, and knowing that before you start hunting saves the hunt.
Test the survivorship mechanism directly: count enrollments with status in ('dropped','expired') at the end of term-week 6 and compute their units_completed over units_total. If that ratio sits well below the active median, removing them from the denominator lifts the rate mechanically and the lift is arithmetic, not behaviour.
Verify the numerator independently, because units_completed is maintained incrementally rather than derived on read. Recount it from lesson activity for a sample of a couple of hundred enrollments and compare against the stored column to rule out a backfill as a second mover.
Confirm term-week 6 maps to the intended dates in both terms through dim_term rather than calendar week, since an edited term_start silently shifts the evaluation window and can move the number on its own.

Follow-up

The old definition penalises a course for enrollments that dropped in week two. Is that the behaviour you actually want from a pacing metric, and who is the metric for?
How do you treat enrollments with NULL scheduled_start_date, and what does each choice do to the reported number?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Stories where someone disagreed with you

Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.

What motivates you to work in the education sector?

Medium · behavioural and stakeholder questions

Approach

Close with what you would do differently, concretely.
Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Account for an analysis you later discovered was wrong

Hard · postmortem · analytical rigor · ownership

Describe a case where you reported a result and later found it was wrong. Choose one where the error was yours and the number had already been used for a decision. Cover: the claim, the actual defect at the data or design level, how you found it, how long it had been live, who you told and in what order, and what you changed so the same class of defect cannot recur. The interviewer will push on the mechanism, not the apology. Probed: whether you understand your own failure modes well enough to have engineered around them.

Approach

Pick an error with a nameable mechanism, such as a grain mistake, a survivorship filter, or a join that silently dropped age_gated learners. 'I misread a chart' has nothing testable in it.
State the blast radius in decisions rather than in dashboards: what was decided, what was spent or promised, and to whom.
Describe detection honestly, including whether someone else caught it and how long it ran undetected. A slow external catch is a usable answer; a vague one is not.
Give the disclosure order and the reason for it: the decision owner first, with the corrected number and its direction in the opening sentence, then the wider audience.
Close on the systemic fix and say whether it has fired since. A control that has never triggered on anything is a resolution, not a fix, and saying so is worth more than claiming otherwise.

Follow-up

What did the corrected number change about the decision? If nothing changed, why did you escalate at all?
Has the check you added caught anything since, and on what?
What class of error would still get past you today?

Explain a risk score to a non-technical sales executive

Easy · communicating uncertainty · renewal risk · stakeholder communication

Your renewal-risk score joins fct_subscription_period to seat activation from dim_learner.first_activity_at and to pace adherence from fct_enrollment. For one institution with 900 seats_provisioned and 31% activation at term-week 6, it outputs a 0.62 probability of non-renewal. A sales VP asks whether you are losing the account or not. You have two minutes and no slides. Deliverable: the spoken answer, plus the one line you would put in the weekly account review so the number is not later quoted as a certainty. Probed: whether you can carry uncertainty and still give a decision.

Approach

Answer the question that was asked before qualifying it. 'Treat this as at-risk and act this week' is the answer; the probability is the support for it.
Translate the score into a frequency on a reference class the VP already trusts, such as how many of the last fifty accounts scored in the 0.55-0.70 band did not renew, instead of explaining calibration in the abstract.
Name the driver that is still movable before period_end: 31% seat activation at term-week 6 can be changed this term, whereas last term's usage cannot, so that is where the conversation goes.
State what would move the score and by when, so the number arrives attached to an action rather than as a verdict.
For the written line, use a band and a review date rather than a bare decimal, because a single number in a document will be re-quoted without its interval.

Follow-up

The VP asks for every account scored above 0.50. What do you tell them about where that threshold came from?
How would you check whether 0.62 is calibrated at all, and on what sample?