Interview guides · Harvard University

Harvard University Data Scientist Interview Questions & Guide 2026

Data scientist prep for Harvard University: practice defining a education technology metric, picking its guardrails and defending the tradeoff.

The 60-second brief

As a Data Scientist at the HBS AI Institute, you serve as a bridge between cutting-edge research and real-world business application. You are not merely building models; you are collaborating with Harvard Business School (HBS) faculty to investigate the societal and economic implications of artificial intelligence. Your work directly influences research-driven insights that shape global business strategy and digital transformation.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

Cluster experiments at class or school level
Align cohorts to term weeks, not calendar weeks
Read item difficulty and discrimination before trusting accuracy

Overview

The role demands a rare combination of rigorous statistical expertise and the ability to act as an expert guide for non-technical stakeholders. You will be expected to transition fluidly between complex experimental design, data cleaning, and the development of ML-based tools for the broader research community. Because the institute operates at the intersection of academia and industry, your success depends on your ability to communicate complex methodologies to faculty and managers while maintaining the highest standards of research integrity.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Pre/post gain studies that select on low pretest scores manufacture improvement.

Any measure with reliability below 1 produces regression to the mean, so a group chosen for scoring in the bottom quartile will score higher on retest with no intervention at all. The apparent gain scales with measurement error, which for a short quiz is large. The fix is a control group selected by the identical rule, or a design that models the pretest as a covariate rather than as a selection filter.

The academic calendar creates structural breaks that look like product effects.

Term start, exam weeks, holidays, and summer each shift usage by amounts far larger than any feature change. A launch timed to week one of a term will show a large lift that is entirely calendar, and a launch in the last week of term will show a collapse. Comparisons must be term-week aligned through dim_term, and any pre/post analysis over a term boundary needs a comparison group living on the same calendar.

Extrapolating a first-week lift inflated by novelty effects

Plot the treatment effect by days since first exposure instead of quoting one pooled average. A lift that decays toward zero across the test window is behaviour that will not persist, and annualising it produces a forecast that misses by an order of magnitude.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

How do you ensure your code is production-ready and maintainable for a team of researchers?

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

Explain how you would address data sparsity when working with large-scale retrieval datasets.

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Translate the result into the decision it informs, in one plain sentence.
Write down the assumption the method needs before you use the method.

Follow-up

How would you explain this result to someone who does not know statistics?
What sample size would you need to detect an effect half this size?

What are the trade-offs between using a pre-trained LLM API versus training a custom model for specific research tasks?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

How do you determine the appropriate statistical model for causal analysis in an experimental research setting?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Permutation test with sections as the randomisation unit

Hard · permutation test · clustering · randomisation inference · design effect

You have outcomes with learner_id, section_id, arm ('treatment' or 'control'), and mastery_events, an integer count over the study window. Assignment was made at section level, so every learner in a section shares an arm. There are 48 sections of roughly 25 learners. Write a permutation test from scratch in numpy that respects the clustering, using the difference in learner-weighted arm means as the statistic. Report the observed statistic, a two-sided p-value, and the number of permutations. No t-test, no statistics package.

Approach

Collapse to a section table first: section_id, arm, n_learners, sum_mastery. The learner-weighted difference in means is a function of those two columns only, and computing it from the section table makes the permutation unit structurally obvious rather than a comment in the code.
Fix the number of treated sections at the observed count and permute arm labels over the 48 sections. Permuting learner labels destroys the within-section correlation and reproduces exactly the false precision that section-level assignment exists to avoid.
Vectorise: build a (B, 48) boolean assignment matrix, then get treated numerators and denominators with two matrix products. That runs 20,000 permutations in well under a second, which matters because you want enough replicates for the p-value to be stable in its third digit.
Compute the statistic as sum of treated sums over sum of treated n, minus the control equivalent. This is a ratio of sums, not a mean of section means; the two differ whenever sections vary in size, and you should say which one the study was powered on.
Two-sided p-value is (1 + count of |T_perm| >= |T_obs|) / (1 + B). Both the numerator and denominator offsets are required for validity at finite B; without them a p-value of exactly zero is reportable, and it is wrong.
Diagnose the clustering rather than assume it: estimate the intraclass correlation from the section table and check that the permutation null SD exceeds the naive learner-level standard error by roughly sqrt(1 + (m - 1) * rho). At m = 25 and rho = 0.15 that factor is about 2.1 on the standard error, or 4.6 on variance.

Worked solution (35 min)

sec = outcomes.groupby('section_id').agg(arm=('arm','first'), n=('learner_id','size'), s=('mastery_events','sum')).
k = (sec.arm == 'treatment').sum(); T_obs = s[treat].sum()/n[treat].sum() - s[ctrl].sum()/n[ctrl].sum().
Build assign = np.zeros((B, 48), bool); for each row set k randomly chosen columns True via rng.permuted on a tiled index array.
num_t = assign @ s; den_t = assign @ n; T_perm = num_t/den_t - (s.sum() - num_t)/(n.sum() - den_t).
p = (1 + (np.abs(T_perm) >= abs(T_obs)).sum()) / (1 + B); report T_obs, p, B, T_perm.std(), and the estimated rho.

Expected result: An observed learner-weighted difference in arm means, a two-sided p-value computed as (1 + exceedances) / (1 + B) with B at least 10,000, and a permutation null whose standard deviation is materially larger than a naive learner-level standard error whenever the estimated intraclass correlation is above zero.

Verify it yourself

Replace mastery_events with pure noise and rerun over many simulated datasets: the p-values must be roughly uniform on [0, 1].
Add a constant to every learner in BOTH arms: T_obs, every permuted statistic and the p-value are identical to the unshifted run, because the statistic is a difference of arm means. Shifting only the treated arm is not an invariance check: T_obs does rise by exactly that constant, but the loop relabels the same shifted data, so the shift travels with the originally-treated sections into whichever arm each permutation hands them and the null widens too. Two sections, k = 1, both outcomes 0 give a null of {0, 0}; add 5 to the treated section and the null becomes {+5, -5} while T_obs is +5, so the p-value is unmoved.
Permuting learners instead of sections must give a visibly narrower null. If it does not, rho is near zero for this outcome and you should report that rather than assume clustering.

Follow-up

48 sections, 24 per arm, baseline and variance as observed. What effect can this design actually detect at 80 percent power, and would you have run it?
Sections range from 8 to 60 learners. Does that change which statistic you should permute?
The institution insists on assigning by school rather than by section. What changes in the test, and what changes in the power?

Consecutive instructional-week streaks and term-over-term return

Hard · gaps and islands · window functions · retention

Using fct_lesson_activity(learner_id, enrollment_id, activity_date, active_seconds) and a calendar dim_term_week(term_id, term_week_no, week_start_date, week_end_date, is_instructional), where holiday weeks carry is_instructional = FALSE, compute for each learner and term the longest run of consecutive instructional weeks containing at least one activity day. A non-instructional week must not break a run, and activity inside one neither extends nor starts a run. Then compute term-over-term return rate into term T+1 with a floor of five activity days in term T. Return learner_id, term_id, activity_days counted over the whole term including non-instructional weeks, longest_streak_weeks and the return flag.

Approach

Collapse activity to (learner_id, term_id, term_week_no) with COUNT(DISTINCT activity_date). fct_lesson_activity is one row per item per session, so a learner with twelve rows on one afternoon is one activity day and counting rows inflates everything downstream.
Re-index before differencing. Apply DENSE_RANK() OVER (PARTITION BY term_id ORDER BY term_week_no) across instructional weeks only, producing a gap-free sequence where a holiday week has simply been removed. Differencing raw term_week_no instead splits a run at every holiday, which is exactly the artefact the task forbids.
Accept what dropping those weeks costs. Activity in a non-instructional week is removed from the streak calculation entirely, so a learner whose only activity lands in a holiday week has activity_days >= 1 and longest_streak_weeks = 0. That is what a streak over instructional weeks means, but it is why the two columns can disagree and why the checks must expect a zero streak alongside non-zero activity rather than assert a floor of 1.
Apply the island trick on the re-indexed sequence: instructional_index - ROW_NUMBER() OVER (PARTITION BY learner_id, term_id ORDER BY instructional_index) is constant within a run. Group on that constant, count rows per group, take the max per learner and term.
Build activity_days from the unfiltered term aggregate, then LEFT JOIN the streak result onto it and COALESCE longest_streak_weeks to 0. An inner join here deletes the holiday-only learners from the output and from the return-rate denominator, which is the same bug as the streak-floor assertion wearing different clothes.
Set the activity floor on COUNT(DISTINCT activity_date) >= 5 across term T, applied to the return-rate denominator only. The floor exists to remove provisioned-but-unused accounts, so leaking it into the numerator silently redefines the metric as active in both terms. Note that a learner can clear the floor on holiday-week activity alone and enter the denominator with a zero streak.
Define the return flag as EXISTS any activity row in term T+1 for that learner, pairing terms by an explicit term ordering rather than date arithmetic, since term lengths differ. Learners whose org has no term T+1 loaded must be excluded and counted, not treated as non-returners.

Follow-up

This streak rewards a learner doing one minute a week over one doing four hours in a single week. Which of those is the product claim, and what would you pair the streak with to catch the difference?
Roster sync deactivates accounts in bulk at term end. How does that interact with the five-day floor and with the return denominator?
Show how return rate moves as the floor goes from one day to five, and argue which floor is the honest one to publish.

Find enrollments with no scored work using anti-joins

Medium · null semantics · anti-join · correctness

You have fct_enrollment(enrollment_id, learner_id, course_id, section_id, org_id, status, enrolled_at, due_date) and fct_assessment_response(response_id, learner_id, enrollment_id, submitted_at, scored_at), where fct_assessment_response.enrollment_id is NULL for work done outside any enrollment. Return every active enrollment with a due_date in the next 14 days that has no scored response attached, for an instructor at-risk list. Write it as an anti-join. A colleague's NOT IN version returns zero rows against production but the correct count against their small test extract. Explain exactly why, and fix it.

Approach

Name the mechanism precisely: NOT IN expands to a conjunction of inequality comparisons, and x <> NULL is unknown rather than false, so once a single NULL enters the subquery list the predicate can never evaluate to TRUE and the result is empty. The test extract contained no work outside an enrollment, so it contained no NULLs, so it passed.
Fix with NOT EXISTS and a correlated equality on enrollment_id. It tests row existence rather than list membership and is unaffected by NULLs elsewhere in the column. LEFT JOIN with an IS NULL filter is equivalent and usually plans the same; NOT IN with an added IS NOT NULL guard also works but leaves the landmine armed for the next reader.
Decide what no scored work means before writing it. No response at all and responses that exist but are unscored are different at-risk lists, and rubric lag puts already-submitted work in the second. Return both counts rather than collapsing them.
Filter status = 'active' and due_date BETWEEN CURRENT_DATE AND CURRENT_DATE + 14, comparing DATE to DATE. Casting a DATE against a timestamp midnight quietly drops the final day.
Guard the complement in the other direction: joining enrollments to responses and counting enrollments fans out one enrollment with 40 responses into 40 rows, so the has-scored-work list needs COUNT(DISTINCT enrollment_id).

Worked solution (20 min)

Run SELECT COUNT(*) FROM fct_assessment_response WHERE enrollment_id IS NULL. That one number is the whole explanation.
Write both versions side by side over the same window and show the row counts differ.
Write the NOT EXISTS version with the response_count and scored_response_count columns attached.
Verify the partition of the due-date window: at-risk count plus distinct enrollments with scored work equals total active enrollments in the window.

Expected result: One row per at-risk enrollment_id with learner_id, section_id, due_date, response_count and scored_response_count, where NOT EXISTS returns a non-zero set and NOT IN returns zero rows on data containing NULL enrollment_id.

Verify it yourself

COUNT(*) of responses with NULL enrollment_id is positive, which is the direct cause of the colleague's empty result.
At-risk count plus COUNT(DISTINCT enrollment_id) of enrollments with at least one scored response equals total active enrollments in the due-date window.
Swapping NOT EXISTS for LEFT JOIN with IS NULL returns an identical row set.

Follow-up

Produce the complement list, the enrollments that do have scored work, without double counting, and state the grain of your join.
An enrollment whose learner submitted work with enrollment_id NULL has done the work but is not attached to it. How would you attribute those rows, and what does attributing by learner and course risk?
fct_assessment_response is partitioned by date. How would you bound the scan without changing the answer?

Describe a time you had to optimize a model to meet a specific performance metric (e.g., MAP@K).

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Metrics that survive an adaptive item selector

Medium · adaptive selection · irt · measurement design

Content selection targets a 70% success probability per learner. First-attempt accuracy reads 69.8% before a model change and 70.3% after. Using dim_content_item (irt_a, irt_b, calibration_n, version_no, status) and fct_assessment_response (attempt_no, is_correct, objective_id), propose a measurement design that can detect a real learning change under this selector. Specify the fixed-form probe set, its size and calibration requirement, the statistic you report, and say why the adaptive accuracy series carries no information about learning.

Approach

Show why the series is pinned. Under a 2PL item, P(correct) = 1 / (1 + exp(-a(theta - b))). A selector targeting 0.70 solves for b given theta: a(theta - b) = ln(0.7/0.3) = 0.847, so b = theta - 0.847/a. As theta rises the selector raises b with it and measured accuracy stays at the target.
Demote accuracy accordingly: it measures how well the selector hit its own target, so keep it as a selector-calibration guardrail and never as an outcome.
Build a fixed-form probe: 10 to 15 items with frozen (content_item_id, version_no), status='published', calibration_n at least 200, spanning irt_b roughly -1.5 to +1.5, served on a fixed schedule identically to every arm and never chosen adaptively.
Report a theta estimate or mean probe score. For 2PL, test information is I(theta) = sum of a_i^2 P_i Q_i, so 12 items with a = 1.0 measured near their peak give I = 3 and SE = 1/sqrt(3) = 0.58 logits per learner. Individual estimates are noisy, but a 400-learner group mean has SE near 0.029 logits, which is where the comparison lives.
Protect the probe: exclude its items from the adaptive pool so practice cannot leak into the measurement, and freeze versions, because editing an item changes the meaning of every historical response to it.
Add the difficulty audit as a second guardrail: mean irt_b of the items backing mastery decisions, which exposes a selector that simply got easier.

Worked solution (30 min)

Derive the selector identity b = theta - ln(0.7/0.3)/a = theta - 0.847/a, and evaluate it at a = 1.2, where the served item sits 0.71 logits below the learner's theta.
Specify the probe: 12 frozen item versions, calibration_n >= 200 each, irt_b spread across -1.5 to +1.5, administered every four weeks to all arms.
Compute probe precision: I = sum a_i^2 P_i Q_i = 12 x 1.0 x 0.25 = 3, SE = 0.58 logits per learner, SE of a 400-learner mean = 0.58/20 = 0.029 logits.
Write the reporting rule: probe mean as the outcome, adaptive accuracy and mean irt_b of mastery-backing items as guardrails.

Expected result: A 10 to 15 item fixed-form probe on frozen versions with calibration_n >= 200, excluded from the adaptive pool, reporting a group mean theta or probe score with SE derived from test information; adaptive first-attempt accuracy demoted to a selector-calibration guardrail alongside mean irt_b, with 69.8% versus 70.3% declared uninformative about learning.

Verify it yourself

b = theta - ln(0.7/0.3)/a = theta - 0.847/a, and at a = 1.2 the offset is 0.71 logits.
Probe items are frozen at one version_no and removed from adaptive selection.
Per-learner SE from 12 items at a = 1.0 is about 0.58 logits, and you did not present individual theta as precise.

Follow-up

How many probe items would you need before reporting an individual learner's theta rather than a group mean?
What happens to the probe if the selector begins serving items that share a template with probe items?
The probe costs learner time every cycle. How do you justify that cost to a curriculum owner?

Overall seat activation fell while every grade band rose

Medium · simpsons-paradox · mix-shift · segmentation

Seat activation, computed at learner grain as the share of provisioned accounts with first_activity_at within 30 days of term start, fell from 0.61 to 0.54 term over term. Computed inside each value of dim_learner.grade_band it rose in all seven bands. You have dim_learner (learner_id, org_id, grade_band, created_at, first_activity_at, acquisition_channel, age_gated) and fct_subscription_period (org_id, plan_tier, seats_purchased, seats_provisioned, period_start). Quantify how many of the seven points are mix and how many are within-segment, and say which number belongs in the board deck.

Approach

Confirm the arithmetic before reaching for a story: compute segment weights (share of provisioned accounts) and segment activation rates for both terms at grade_band grain, and verify the weighted segment rates reproduce each term's headline exactly.
Run the decomposition with both terms written out: total delta equals sum of w_pre * (r_post - r_pre) for within-segment plus sum of (w_post - w_pre) * r_pre for mix, assigning the interaction term to one side explicitly so the pieces sum without residual. Report each in points, not percentages of each other.
Trace the weight shift to its source. Cut newly created learners by acquisition_channel and by created_at hour; several thousand accounts created inside a single hour under district_deploy or roster_sync is a provisioning job, and provisioning is not learner behaviour.
Check the denominator for over-provisioning: seats_provisioned can exceed seats_purchased, and an org that creates accounts it never intends to use drags its own activation down with no learner acting differently. Compare provisioned against purchased by org and quantify the affected share.
Report the raw rate alongside a mix-adjusted rate holding prior-term band weights, and state the age_gated share of the population, since consent rules restrict behavioural logging for those learners and can bias which accounts appear active at all.

Follow-up

Which of the two numbers goes on the institutional health score, and what does the other one still tell you that the first hides?
How would you detect this kind of mix shift automatically next time, rather than after someone noticed the aggregate was wrong?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Scope a vague request to investigate falling engagement

Easy · scoping · metric definition · stakeholder management

A curriculum director messages you: engagement is down in middle school, can you dig into it. You have dim_learner, fct_lesson_activity, fct_assessment_response, fct_enrollment and dim_term. Before writing any query you get one fifteen-minute conversation. Deliverable: the questions you ask in that conversation, the written scope you send afterwards, and the first three things you compute, with the reason each comes before the others. Probed: whether you convert an ambiguous request into a decision and a single metric before touching data, rather than after.

Approach

Find the decision behind the request first: what the director would do differently depending on the answer, and by when. That sets how much of this analysis is worth doing at all.
Pin the four ambiguities that make 'engagement is down' uncomputable: which population (grade_band = '6_8', which orgs), which metric (active_seconds, activity days, scored submissions), which comparison window, and against which baseline.
Ask where the belief came from. 'Down' is almost always a chart someone saw, and reproducing that exact chart is faster than building a new one and then arguing about why the two differ.
Check the calendar before believing the shape at all: a decline spanning a term boundary is the default explanation in this domain, so term-week alignment through dim_term is the first computation, not a later refinement.
Send the scope back in writing with an explicit out-of-scope list and a time box, so follow-on questions arrive as new requests instead of as scope creep.

Follow-up

The director cannot name the decision this feeds. Does that change whether you do the work, and how much of it?
Your term-aligned version shows no decline at all. What do you send, and to whom first?

State the impact of your last project without inflating it

Hard · impact measurement · attribution · causal reasoning

Pick one project from the past year and account for its impact as if the listener could audit every number. Cover the metric you claim to have moved and its exact definition, how much of the movement is attributable to your work, what else was changing at the same time, what the counterfactual was and where it came from, and what you would have needed to measure at the start to make the claim clean. If the effect is not separable, say so and say what you would do differently. Probed: attribution discipline applied to your own work.

Approach

Name the metric by its definition rather than its label. 'Verified mastery rate' means nothing without its numerator, denominator, window, and the reporting lag the delayed retention check forces.
Separate the three claims usually merged into one: the metric moved, your work moved it, and the movement was worth what it cost.
State the counterfactual explicitly and say what produced it: a holdout, matched sections, or a pre-period trend. If it was a before-and-after with no control, say that in those words.
List the co-occurring changes with dates: a term boundary, a roster sync at scale, a pricing change, another team shipping into the same surface. In this domain the academic calendar alone routinely dwarfs a product effect.
Convert the shortfall into a specific design artefact: the holdout you would have reserved, the instrumentation you would have shipped before the change, the analysis plan you would have written first.

Follow-up

What fraction of the observed movement would you defend under cross-examination, and on what evidence?
Your project shipped in term-week one. How does that change what you can claim?
If the effect is genuinely not separable, was the project worth doing?

Defend a null result against an already-announced launch

Medium · null results · experiment design · stakeholder communication

A section-randomised test of a new practice-sequencing feature ran across 62 class sections in 11 orgs. Primary metric: verified mastery events per active learner. With section-clustered standard errors the effect is +1.8% (95% CI -4.1% to +7.9%). Leadership has already told two district administrators that the feature improves outcomes. You have ten minutes in a review. Deliverable: what you say, what you put on one slide, and what you propose next. Probed: whether you can hold a null under commercial pressure without either caving or lecturing the room on statistics.

Approach

Lead with the decision, not the p-value: state the interval in the unit the audience already uses (mastery events per active learner per week) and say plainly what the study could and could not have detected.
Separate the interval's half-width from the minimum detectable effect before you put either on a slide. The stated 95% CI of -4.1 to +7.9 has a half-width of 6.0 points, so the clustered standard error is 6.0 / 1.96 = 3.06 points; the MDE at 80% power and two-sided alpha = 0.05 is (1.96 + 0.8416) * 3.06 = about 8.6 points, roughly 1.43 times the half-width. Quote 8.6, and show that it follows from 62 sections at the section size and intraclass correlation you assumed via the design effect 1 + (m - 1) * rho, so that 'no effect found' is visibly separated from 'too small to see'.
Check the pre-registered guardrails before conceding anything: median active_seconds per mastery event and the calibrated irt_b distribution of the items backing mastery decisions. If both are flat, the null is about learning, not about a broken pipeline.
Handle the promise made to the two districts as its own item with a named owner, rather than leaving the correction implicit in a statistics discussion.
Close with the cheapest decision-grade next step: extend to enough additional already-instrumented sections to reach the effect size leadership actually cares about, with the stopping rule fixed before the extension starts.

Follow-up

If the interval had been -0.5% to +8.0%, would you support launch, and what exactly changes in your reasoning?
Leadership asks you to re-run with learner-level standard errors since it is the same data. What do you say?
What would you have set up at design time so that this meeting was easier?