Carnegie Mellon University Data Scientist Interview Questions & Guide 2026

The 60-second brief

The Data Scientist role at Carnegie Mellon University is a unique, high-impact position situated at the intersection of advanced research and real-world public policy. Unlike traditional corporate roles, this position is embedded within an interdisciplinary group spanning the School of Computer Science and the Heinz College of Public Policy. Your work directly contributes to social good, ranging from improving public health outcomes and criminal justice reform to supporting economic development through data-driven intervention.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

• Align cohorts to term weeks, not calendar weeks
• Separate mastery signals from raw usage exposure
• Model seat activation as the renewal leading indicator

Overview

You will act as both a technical lead and a bridge between complex algorithmic solutions and non-technical stakeholders in government and non-profit sectors. Success in this role requires a deep commitment to ethical AI, as your primary focus is on developing systems that are not only accurate but also equitable, transparent, and interpretable. You will be responsible for the full lifecycle of data-intensive systems, from scoping and formulation to deployment and field evaluation, while mentoring the next generation of researchers and students.

The Carnegie Mellon University interview process

Carnegie Mellon University candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Technical Screening

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

• Whether the query you write matches the plan you just described
• What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
• Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

• Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
• Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
• Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Intensive Rounds

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

• Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
• Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
• Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

• For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
• Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
• For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Behavioral and Case-Study Rounds

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

• Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
• Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
• Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

• For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
• Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
• Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Where candidates lose points

Roster sync creates and deactivates accounts in bulk, and it is not behaviour.

An automated roster import at term start can create thousands of accounts in one hour and deactivate thousands more at term end. Signup, activation, and churn series computed without filtering enrollment_source IN ('roster_sync','admin_bulk') will show spikes and cliffs driven entirely by an integration job. It also breaks cohort retention: a roster-created account that never activates is a provisioning artefact, not a churned learner.

Pre/post gain studies that select on low pretest scores manufacture improvement.

Any measure with reliability below 1 produces regression to the mean, so a group chosen for scoring in the bottom quartile will score higher on retest with no intervention at all. The apparent gain scales with measurement error, which for a short quiz is large. The fix is a control group selected by the identical rule, or a design that models the pretest as a covariate rather than as a selection filter.

Sizing estimates built on unnamed, unrevisable assumptions

Write each assumption as a named number you can change, then show the arithmetic so the interviewer can challenge one input instead of the whole answer. Finish by saying which assumption the result is most sensitive to, which matters more than the point estimate.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

If you notice a sudden drop in the model's precision for a public health intervention, what steps would you take to diagnose the root cause?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Frame the prediction: the label, the moment of prediction, and the action it triggers.
3. Say how the offline result would be validated online before it is trusted.

Follow-up

• Where could label leakage enter this setup?
• What would you monitor after launch to know the model is still valid?

Given a messy dataset from a government agency, what approach do you take to validate data integrity before building a model?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Sessionise a raw heartbeat stream into activity rows

Hard · sessionisation · event streams · groupby · time series

You get a DataFrame heartbeats with columns learner_id, content_item_id, content_version_no, and ts (UTC timestamps, unsorted). Produce one row per learner per session per content item with started_at, ended_at, wall_seconds, and active_seconds. A session closes after 30 minutes with no heartbeat from that learner. active_seconds sums consecutive gaps inside a session, capping each gap at 120 seconds. Attribute each gap to the content item of the earlier heartbeat. State your rule for a gap of exactly 1800 seconds and for a session with a single heartbeat.

Approach

1. Sort by learner_id then ts, and compute the gap with a per-learner diff so the first heartbeat of each learner has no gap. A global diff lets one learner's last heartbeat set the next learner's first gap.
2. Flag a session break where gap > 1800 seconds, then cumsum that boolean within the learner to get a session index. Say whether exactly 1800 opens a new session; either choice is defensible but it must be stated because it moves the session count.
3. Each gap covers the interval before the current heartbeat, so attribute min(gap, 120) to the content_item_id of the previous row via shift, not the current row.
4. Group by (learner_id, session_index, content_item_id): sum the attributed capped gaps for active_seconds, take min and max ts for started_at and ended_at, and derive wall_seconds from those two.
5. Handle the degenerate group explicitly. A single heartbeat gives active_seconds 0 and wall_seconds 0. Dropping those rows quietly removes real opens and biases any completion or abandonment rate computed later.
6. Validate that within every session the summed active_seconds never exceeds the session's wall_seconds, which catches attribution and cap errors in one assertion.

Worked solution (35 min)

1. Sort by (learner_id, ts) and compute gap = ts.diff().dt.total_seconds(), then null the gap on the first row of each learner using a learner-change mask.
2. new_session = gap.isna() | (gap > 1800); session_index = new_session.groupby(learner_id).cumsum().
3. contribution = gap.clip(upper=120).fillna(0), attributed to the previous row's content_item_id using shift within the learner and session.
4. Aggregate by (learner_id, session_index, content_item_id) with sum for active_seconds and min and max of ts for the timestamps.
5. Assert per session that sum(active_seconds) <= (max(ts) - min(ts)) in seconds, and print the count of single-heartbeat groups.

Expected result: A frame keyed by (learner_id, session_index, content_item_id) where active_seconds equals the sum of min(gap, 120) over consecutive heartbeats with the first heartbeat contributing zero, wall_seconds equals ended_at minus started_at, and sum(active_seconds) <= wall_seconds holds for every session.

Verify it yourself

• Heartbeats at 0, 60, and 120 seconds on one item give one session with wall_seconds 120 and active_seconds 120.
• Heartbeats at 0 and 3600 seconds give two sessions, each with wall_seconds 0 and active_seconds 0.
• Inserting a 300-second gap inside a session adds exactly 120 to active_seconds, not 300, while wall_seconds rises by the full 300.

Follow-up

• The client emits a heartbeat every 60 seconds on desktop and every 15 seconds on a managed device fleet. What does the 120-second cap do to the comparison between those two populations?
• Your reconstructed active_seconds disagrees with the stored column by 4 percent overall. How do you localise the disagreement rather than argue about the total?
• A learner has two tabs open on different items at once. What does your output claim, and what should it claim?

Using SQL window functions, how would you identify the first and last transaction dates for a specific cohort of users?

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. State the window function and its partition and ordering out loud before writing it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Pace adherence at term-week six from activity history

Hard · window functions · running aggregates · term alignment

Compute the share of active enrollments at or ahead of scheduled pace at the end of term-week 6, from fct_enrollment(enrollment_id, learner_id, course_id, section_id, term_id, enrolled_at, scheduled_start_date, due_date, units_total, units_completed, status), fct_lesson_activity(enrollment_id, content_item_id, completion_status, started_at, activity_date) and dim_term(term_id, term_start_date, term_end_date, term_length_weeks). units_completed is maintained incrementally and reflects now, not week 6. Rebuild the as-of count from activity, align each enrollment to its own term calendar, and handle NULL scheduled_start_date and NULL term_id explicitly.

Approach

1. Refuse the stored column for this question. units_completed is a current snapshot, so using it scores every enrollment on today's progress and labels the result week 6. That single substitution is the difference between a metric and a number that drifts upward every time you re-run it.
2. Define a completed unit from activity: take MIN(started_at) per (enrollment_id, content_item_id) over rows with completion_status = 'completed', so a learner re-opening finished content does not add a unit. A COUNT over that deduplicated set ordered by first-completion time gives units completed as of any instant.
3. Set the boundary per term, not per calendar: week 6 ends at term_start_date + 42 days for that enrollment's term_id. Enrollments with NULL term_id cannot be week-aligned at all; exclude them and report the count rather than defaulting them into somebody else's calendar.
4. For NULL scheduled_start_date, choose between the term start and the learner's first activity, state the choice, and measure how many enrollments it touches. If the share is large, the pace metric is being defined by a data gap rather than by pacing.
5. Apply the threshold as rebuilt_units >= units_total * (6.0 / term_length_weeks), guarding units_total > 0 and excluding terms shorter than 6 weeks instead of letting the ratio exceed 1.
6. Be honest about the denominator: fct_enrollment.status has no effective dating, so active at the end of week 6 is not reconstructible from this schema. Exclude enrollments whose enrolled_at is after the boundary, use current status as the stated approximation, and say which direction the survivor bias runs.

Worked solution (40 min)

1. Build the first-completion CTE with MIN(started_at) per (enrollment_id, content_item_id) and assert one row per pair.
2. Join dim_term to get each enrollment's week-6 boundary timestamp, and count rows lost to NULL term_id.
3. Count completions at or before the boundary per enrollment, then apply the pace threshold.
4. Aggregate to (term_id, course_id) and attach the three exclusion counts.
5. Re-run the query with a different CURRENT_DATE to prove the answer does not move.

Expected result: One row per (term_id, course_id) with active_enrollments, on_pace_enrollments, pace_rate, plus separate counts of enrollments excluded for NULL term_id, units_total = 0, term_length_weeks < 6, and enrolled_at after the boundary.

Verify it yourself

• Rebuilt units at week 6 is at or below stored units_completed for every enrollment. A row above it means the stored column is stale or the completion dedup is admitting re-opens.
• Running the query again a week later returns identical numbers. Any movement means a dependency on now survives somewhere in the logic.
• COUNT(DISTINCT (enrollment_id, content_item_id)) in the completion CTE equals its row count.
• The exclusion counts plus the reported denominator sum to the raw enrollment count for the term.

Follow-up

• status has no history. What would you add to the model to make active at the end of week 6 reconstructible, and until then does the current answer over- or under-state pace?
• Two institutions on different calendars now compare their pace rates. What is still not comparable after term-week alignment?
• A holiday lands in week 4 for one institution and week 5 for another. Does your denominator punish either, and how would an instructional-day calendar change the threshold?

How would you structure a query to calculate a rolling average of engagement metrics over the last 30 days?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

How would you design a metric to measure the success of an outreach campaign aimed at enrolling eligible families in SNAP programs?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

What are the most common experimentation pitfalls you have encountered when working with limited sample sizes in public policy?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. Say whether units interfere with each other, and switch design if they do.
3. State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

• How would you handle interference between treated and control units?
• What would you conclude if the result is positive but the test is underpowered?

If you cannot run a traditional A/B test due to ethical constraints, what alternative evaluation strategies would you propose?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Decide the analysis before seeing data, including how long it runs and when you look.
3. Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

How do you evaluate the effectiveness of an intervention in a non-randomized field study?

Medium · a/b testing and experimentation

Approach

1. Name the randomisation unit first; it decides the variance and what the test can detect.
2. Say whether units interfere with each other, and switch design if they do.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

Apply CUPED and stratification to an under-powered clustered test

Medium · cuped · stratification · variance reduction

A section-randomised test is under-powered and the team cannot get more sections. Eight term-weeks of pre-period fct_assessment_response and fct_lesson_activity exist before assignment. The learner-level correlation between pre-period and in-test weekly mastery events is 0.55; at section-mean level it is 0.72. Propose a variance-reduction plan. Compute the MDE multiplier CUPED buys at each level, say which level you apply it at and why, and handle the learners with no pre-period because roster_sync created their accounts after assignment.

Approach

1. State the estimator: Y_adj = Y - theta x (X - mean(X)), where X is the pre-period covariate and the binding precondition is that X is measured strictly before assignment. Fit theta = Cov(Y, X) / Var(X) on the in-test data pooled across both arms. That is standard CUPED and it is the only option here: Y is the in-test outcome, so theta cannot be computed from pre-period data alone. Pooling is what keeps it honest, because a pre-assignment X has the same distribution in both arms and the adjustment therefore cannot absorb the treatment effect. Var(Y_adj) = (1 - r^2) x Var(Y), so the MDE scales by sqrt(1 - r^2).
2. Apply it at the unit of analysis. Under section randomisation that is the section mean, so the 0.72 section-level correlation is the one that matters. It exceeds the learner-level 0.55 because averaging removes idiosyncratic learner noise while preserving the section's persistent level.
3. Stratify the randomisation itself rather than only adjusting afterwards: block sections on pre-period mastery-event quartile crossed with grade_band, then randomise within block. With 300 sections per arm, blocking also protects against a bad draw, which post-hoc CUPED cannot undo.
4. Handle the no-pre-period learners without imputing a value that would drag theta. Place them in their own stratum with X set to that stratum's mean, which leaves them with zero adjustment and no bias, and report their share alongside the result.
5. Hold the two rules that actually protect the estimate: the covariate must be measured strictly before assignment, and theta must be fitted once on the pooled arms rather than separately within each arm. A post-assignment covariate can itself be moved by treatment, so adjusting on it subtracts part of the effect you are measuring. Arm-specific thetas make the adjustment differ by arm, which reintroduces the same bias by another route. Fitting theta on the data it adjusts carries a finite-sample bias of order 1 / n_clusters; at 300 sections per arm that is negligible, and if you want it gone entirely, estimate theta from an earlier pre-period regression on data outside the test window and accept a slightly lower r.

Worked solution (30 min)

1. Learner level: sqrt(1 - 0.55^2) = sqrt(0.6975) = 0.835, a 16.5% smaller MDE.
2. Section level: sqrt(1 - 0.72^2) = sqrt(0.4816) = 0.694, a 30.6% smaller MDE.
3. Convert to sample equivalence: 1 / 0.4816 = 2.08, so section-level CUPED is worth roughly doubling the number of sections.
4. Build the blocks as four pre-period quartiles crossed with grade_band, randomise within block, and persist the block id for the analysis.
5. Fit theta once on the pooled section-level in-test data, form Y_adj, then estimate the effect at section level with block fixed effects and compare the adjusted and unadjusted estimates and intervals.

Expected result: Apply CUPED at the section mean: multiplier 0.694, a 30.6% MDE reduction, equivalent to about 2.08 times the sections. Learner-level CUPED delivers only 0.835 and is the wrong level under section randomisation. Block on pre-period quartile crossed with grade_band before randomising rather than relying on adjustment alone.

Verify it yourself

• CUPED should narrow the interval and move the point estimate only by about as much as the pre-period imbalance justifies. A large shift means the covariate is imbalanced across arms or was not measured strictly before assignment; it does not mean theta was fitted on in-test data, which is where theta is supposed to be fitted.
• Mean pre-period covariate should be close across arms before adjustment. A large gap is itself a randomisation finding and needs explaining before the adjusted number is trusted.
• The share of learners with no pre-period must be reported. Above roughly 10%, a meaningful part of the result is carrying no adjustment at all.

Follow-up

• Your pre-period window spans a holiday for half the sections. What does that do to the correlation and to theta?
• Can you stack CUPED on top of stratification, and does the second one still buy anything once the first has run?
• The section-level correlation is 0.72 overall but 0.45 for sections whose instructor changed between periods. How do you handle that subgroup?

Overall seat activation fell while every grade band rose

Medium · simpsons-paradox · mix-shift · segmentation

Seat activation, computed at learner grain as the share of provisioned accounts with first_activity_at within 30 days of term start, fell from 0.61 to 0.54 term over term. Computed inside each value of dim_learner.grade_band it rose in all seven bands. You have dim_learner (learner_id, org_id, grade_band, created_at, first_activity_at, acquisition_channel, age_gated) and fct_subscription_period (org_id, plan_tier, seats_purchased, seats_provisioned, period_start). Quantify how many of the seven points are mix and how many are within-segment, and say which number belongs in the board deck.

Approach

1. Confirm the arithmetic before reaching for a story: compute segment weights (share of provisioned accounts) and segment activation rates for both terms at grade_band grain, and verify the weighted segment rates reproduce each term's headline exactly.
2. Run the decomposition with both terms written out: total delta equals sum of w_pre * (r_post - r_pre) for within-segment plus sum of (w_post - w_pre) * r_pre for mix, assigning the interaction term to one side explicitly so the pieces sum without residual. Report each in points, not percentages of each other.
3. Trace the weight shift to its source. Cut newly created learners by acquisition_channel and by created_at hour; several thousand accounts created inside a single hour under district_deploy or roster_sync is a provisioning job, and provisioning is not learner behaviour.
4. Check the denominator for over-provisioning: seats_provisioned can exceed seats_purchased, and an org that creates accounts it never intends to use drags its own activation down with no learner acting differently. Compare provisioned against purchased by org and quantify the affected share.
5. Report the raw rate alongside a mix-adjusted rate holding prior-term band weights, and state the age_gated share of the population, since consent rules restrict behavioural logging for those learners and can bias which accounts appear active at all.

Follow-up

• Which of the two numbers goes on the institutional health score, and what does the other one still tell you that the first hides?
• How would you detect this kind of mix shift automatically next time, rather than after someone noticed the aggregate was wrong?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Explaining a method to someone who will not follow the maths

Half of this section is about translation. Be ready to describe how you explained a result to someone who did not want the method, only the implication, and what you did when the simplified version started being repeated in a way that overstated it. Correcting your own simplification is a strong beat.

How do you approach mentoring junior researchers when they are struggling with a technical roadblock?

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Pick a story where you drove the decision, not one where you observed it.
3. Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

• How did you know the outcome was caused by your change?
• What would you do differently if you ran that project again?

Tell an executive the usage report they publish is wrong

Medium · data quality · metric definition · stakeholder communication

A quarterly usage report an executive sends to district administrators counts an active learner as anyone with a fct_lesson_activity row in the quarter. You find two defects: accounts created by roster_sync that opened a single item during an automated import are counted in the numerator, and learners with age_gated = TRUE never reach the source table, so they are absent from the denominator. The report has already gone out twice. Deliverable: what you send, to whom, in what order, and what the definition should become. Probed: whether you can deliver an unwelcome data-quality finding without detonating it.

Approach

1. Size the error before raising it. Recompute the metric both ways per org: 'the definition is wrong' and 'the number is overstated by 18 points on the largest account' get very different responses.
2. Keep the two defects separate, because they push in opposite directions. Roster artefacts inflate the numerator while age-gated truncation shrinks the denominator, so the net error varies by org and a single average hides that.
3. Go to the report owner first with the recomputation and a proposed replacement, not to their audience. The person who published it needs the chance to correct it themselves.
4. Propose a definition that is defensible and computable today: distinct learners with at least one scored submission in the quarter, with enrollment_source in ('roster_sync','admin_bulk') excluded from activation cuts, and the age_gated share stated as a footnote rather than silently dropped.
5. Say what should happen to the two reports already sent, with a recommendation and an owner. The restatement is the uncomfortable part, and leaving it for someone else to raise means the work is unfinished.

Follow-up

• The executive asks you to change the definition quietly from next quarter with no restatement. What do you do?
• How would you stop this class of defect reaching a published report again?

Account for an analysis you later discovered was wrong

Hard · postmortem · analytical rigor · ownership

Describe a case where you reported a result and later found it was wrong. Choose one where the error was yours and the number had already been used for a decision. Cover: the claim, the actual defect at the data or design level, how you found it, how long it had been live, who you told and in what order, and what you changed so the same class of defect cannot recur. The interviewer will push on the mechanism, not the apology. Probed: whether you understand your own failure modes well enough to have engineered around them.

Approach

1. Pick an error with a nameable mechanism, such as a grain mistake, a survivorship filter, or a join that silently dropped age_gated learners. 'I misread a chart' has nothing testable in it.
2. State the blast radius in decisions rather than in dashboards: what was decided, what was spent or promised, and to whom.
3. Describe detection honestly, including whether someone else caught it and how long it ran undetected. A slow external catch is a usable answer; a vague one is not.
4. Give the disclosure order and the reason for it: the decision owner first, with the corrected number and its direction in the opening sentence, then the wider audience.
5. Close on the systemic fix and say whether it has fired since. A control that has never triggered on anything is a resolution, not a fix, and saying so is worth more than claiming otherwise.

Follow-up

• What did the corrected number change about the decision? If nothing changed, why did you escalate at all?
• Has the check you added caught anything since, and on what?
• What class of error would still get past you today?