Interview guides · The Sparks Foundation

The Sparks Foundation Data Scientist Interview Questions & Guide 2026

Case practice for The Sparks Foundation data scientist candidates: scope a vague prompt into a measurable question before you analyze it.

The 60-second brief

The Data Scientist role at The Sparks Foundation is pivotal in harnessing data to drive impactful insights and solutions. As a Data Scientist, you will analyze complex data sets to support decision-making processes that enhance educational initiatives and resources. This position is essential for understanding user behavior, improving program effectiveness, and ultimately contributing to the foundation's mission of empowering individuals through education.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

Size an experiment before anyone launches it
Separate novelty effects from durable behaviour change
Turn a vague request into a measurable question

Overview

You will work closely with cross-functional teams, including engineering and product management, to deliver data-driven solutions that address real-world problems. The role involves not only data analysis but also the application of machine learning techniques, statistical modeling, and data visualization to present findings in a compelling way. With the growing emphasis on data in decision-making, your contributions will have a direct impact on the effectiveness of the foundation's initiatives, making this role both critical and rewarding.

The The Sparks Foundation interview process

The Sparks Foundation candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Mentor Interaction

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Project Assessments

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Where candidates lose points

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Reading a pooled rate that moved because the mix moved, not because any behaviour changed

A pooled conversion rate is a weighted average, and a shift in the weights can move it in the opposite direction to every one of its parts. A paid campaign that brings low-converting traffic drops overall signup conversion even if desktop, mobile web and app conversion each rose that week, which is Simpson's paradox and it is the single most common cause of an inexplicable dashboard move. The discipline is to decompose before explaining: recompute the rate holding last period's segment weights fixed, and compare that counterfactual to the actual, so the mix effect and the rate effect are separated numerically rather than argued about. Segment on the dimensions that actually reweight, which in this domain are almost always device_type, referrer_channel, country and new versus returning.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

Write a function that calculates the mean and standard deviation of a list of numbers.

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Write down the assumption the method needs before you use the method.
Say what the estimate is of, and over what population it generalises.

Follow-up

What sample size would you need to detect an effect half this size?
How would you explain this result to someone who does not know statistics?

Can you explain the concept of overfitting and how to prevent it?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.
Say how the offline result would be validated online before it is trusted.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

How would you implement a decision tree algorithm from scratch?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.
Check what information would not exist at prediction time, and exclude it.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
Set ref = df['received_at_utc'].max() and build the five masks against it.
Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Attribute account MRR to first-touch channel without seat fan-out

Medium · fan-out · joins · revenue · attribution

dim_user holds user_id, account_id, first_touch_channel, account_created_at_utc. dim_account holds account_id, lifecycle_status. fct_subscription_period holds account_id, period_start_utc, period_end_utc, period_status, mrr_cents_constant_fx. For a given as-of date, report active MRR by the first-touch channel of the account's earliest-created user. An account has many users and many periods. The channel totals must sum exactly to total active MRR on that date, and accounts whose label is NULL must appear as their own bucket.

Approach

Reduce the label side to one row per account before any join: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY account_created_at_utc, user_id) = 1 over dim_user, carrying first_touch_channel. Joining dim_user to the period table directly repeats each revenue row once per seat.
Reduce the money side to one row per account too: the period where period_start_utc <= :as_of AND period_end_utc > :as_of AND period_status IN ('active','past_due'). If more than one row survives that filter the account holds overlapping subscriptions, which must be deduped or deliberately summed, and either way the choice is stated rather than left to the join.
Join the two one-row-per-account CTEs and SUM(mrr_cents_constant_fx). Use the constant-FX column, since the same query run in two quarters otherwise reports currency moves as channel performance.
Bucket NULL first_touch_channel as 'unattributed' with COALESCE instead of letting it drop. Roughly the share of users with no observed touch is not small, and dropping it understates the total and inflates every named channel's share.
Reconcile: the SUM over the joined result must equal the SUM over the period CTE alone. That single equality check catches fan-out immediately, which is why it is worth writing before presenting the number.

Follow-up

Which user should own the account's label: earliest created, the 'owner' role, or the one who converted? What does each choice systematically favour?
What does this table entitle you to say about the value of a channel, and what does it not?
Some accounts have a parent_account_id. How does rolling those up change both the numerator and the channel mix?

Weekly visit-to-signup conversion split by acquisition channel

Easy · funnel · conversion · aggregation

From fct_session (session_id, visitor_id, started_at_utc, referrer_channel, is_bot_flagged, consent_state) and fct_event (visitor_id, occurred_at_utc, event_name), compute visit-to-signup conversion for one ISO week, split by channel. session_id is the unique key of fct_session. Denominator: distinct visitor_id with a session starting in the week, is_bot_flagged = FALSE and consent_state <> 'denied'. Numerator: those visitors with a 'signup_completed' event in the same week. Label each visitor with the referrer_channel of their first session in the window. Return channel, visitors, signups and rate, plus one all-channel total row.

Approach

Build a visitor spine that is one row per visitor: filter sessions to the week, drop is_bot_flagged and consent_state = 'denied', then take the first session per visitor with ROW_NUMBER() OVER (PARTITION BY visitor_id ORDER BY started_at_utc, session_id) = 1 to carry the channel label. Collapsing to one row here is what makes the channel buckets mutually exclusive and the totals additive.
session_id is the unique key, so that ordering is total and the label is reproducible. If the table carried no unique key you would have to write an explicit tie rule instead, because two sessions on different channels at the identical timestamp would otherwise label the visitor differently between runs.
Attach the outcome as a semi-join (EXISTS on a signup_completed event for that visitor inside the same week) rather than a join to the event table, so a visitor who fires the event twice does not count twice and inflate the numerator past the denominator.
Aggregate with COUNT() as visitors and COUNT() FILTER (WHERE signed_up) as signups, and compute the rate as signups::numeric / NULLIF(visitors, 0) so an empty channel returns NULL rather than a division error.
Produce the total with GROUP BY GROUPING SETS ((channel), ()), which re-sums numerator and denominator for the total row. Averaging the channel rates gives a different and wrong number whenever channel volumes differ, which they always do.
Verify the spine before trusting the output: COUNT(*) must equal COUNT(DISTINCT visitor_id), and the per-channel visitor counts must sum to the total row.

Worked solution (20 min)

Write the filtered session CTE and check its row count against an unfiltered count, so you know how much volume the bot and consent filters removed.
Add the ROW_NUMBER first-session pick and assert one row per visitor.
Add the EXISTS outcome flag and aggregate with FILTER.
Add GROUPING SETS for the total and format the rate to four decimal places.
Spot-check one channel by hand: pull its visitor list, count signups directly, compare.

Expected result: One row per referrer_channel plus a total row, with visitors, signups and a decimal rate. Channels with zero signups appear with rate 0, not missing. The total rate lies strictly between the minimum and maximum channel rate unless one channel holds all the volume.

Verify it yourself

Per-channel visitor counts sum exactly to the total row's visitor count.
COUNT(*) equals COUNT(DISTINCT visitor_id) on the spine CTE.
Replacing the EXISTS with a plain inner join changes the signup count only if duplicate signup_completed events exist; if it does change, say by how much.
Every rate is between 0 and 1.
Re-running the first-session pick with session_id removed from the ORDER BY returns the same labels; if it does not, visitors hold same-timestamp sessions on different channels and the tie rule is doing real work.

Follow-up

The denominator is distinct visitors. If a browser release shortens cookie lifetime, what happens to this rate, and how would you tell that apart from a genuine drop?
A visitor's first session is direct and their signup session is paid search. Your label says direct. When is that the wrong answer for the decision being made?
How do you roll four weeks into a month, and why is averaging the four weekly rates wrong?

If tasked with improving a current product feature, what data would you need to analyze?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How do you prioritize your tasks when working on multiple projects?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Given a dataset, how would you approach cleaning and preprocessing the data?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

What is your approach to feature selection?

Medium

Approach

Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer would contain.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Measure whether self-serve help actually answered the question

Medium · proxy metrics · bias · support

A help widget opens in-product. Leadership asks for the percentage of users who got their answer. You have fct_event (event_name, user_id, session_id, occurred_at_utc, is_core_action, properties JSONB carrying article_id), fct_session, and a support ticket table joined on user_id. Nothing records whether the answer was correct or whether the user was satisfied, and no such field is being added. Propose the metric you would publish, state the proxy plainly, name the direction and rough size of its bias, and say which decisions it can and cannot support. Deliverable: the definition plus the bias statement.

Approach

Say first that the target quantity is unobserved and will stay unobserved: nothing in the stream distinguishes a user who was helped from one who gave up, and both leave the same trace, so every candidate metric here is a proxy and the only question is which bias you prefer.
Build the proxy to remove the largest identifiable error: among widget-open sessions, the share with no ticket from that user within 72 hours and at least one is_core_action = TRUE event after the widget opened in the same session. The downstream action requirement strips out most of the silent abandonment that a no-ticket rule alone scores as success.
State the residual bias with a direction and a bound: it still over-counts, because a user who gave up and went elsewhere files no ticket and may still complete an unrelated core action later in the session; bound it using the observable population of widget-open sessions that end within two minutes with no further event.
Publish a directly-measured companion with its own weakness in the same sentence: article thumbs up/down reported with its response rate, and the note that a single-digit response rate missing non-randomly toward annoyed users is exactly why it cannot be the headline.
Write the use statement, because a proxy without one gets reused for the wrong decision: valid for ranking articles against each other and for detecting a week-on-week break, invalid as an absolute deflection rate or as an input to a cost-saved figure.

Worked solution (25 min)

Write the unobservable target in one sentence with the observable set beside it, so the gap is visible on the page rather than assumed away.
Define the published proxy with exact numerator and denominator, including the session scoping of the downstream core action.
Bound the bias from data you already have: compute the share of widget-open sessions ending within two minutes with no further event.
Publish the thumbs signal beside it with its response rate and the direction of its missingness.
Write the two-line use statement covering what the number supports and what it does not.

Expected result: A named proxy with numerator and denominator, a stated bias direction (over-counts success) with an observed magnitude bound, and an explicit list of the decisions the number does and does not support.

Verify it yourself

The proxy and the raw no-ticket rate should differ by roughly the abandonment share computed in step three. If they are nearly identical, the downstream-action requirement is not binding and adds nothing.
A change in support hours or ticket-form availability moves the no-ticket rate with no change in help quality, so the series must carry annotations for support operational changes.
Sessions with consent_state = 'denied' cannot be joined forward to a user_id and therefore cannot enter this metric at all; the denominator definition has to say so.

Follow-up

How would you validate this proxy once, and what would you do if the validation said it over-counts by 20 points?
Support asks for a dollars-saved number from this metric. What do you say, and what would you need before saying anything else?
Ticket volume drops the week support hours change. How do you keep that out of the series?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Tell me about a time you faced a challenge in a team project. How did you handle it?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Describe a situation where you had to persuade someone to embrace your data-driven recommendation.

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Turn an ambiguous onboarding question into a measurable metric

Easy · scoping · metric definition · stakeholder

Two days before a planning review, a director asks whether onboarding is working. You have dim_user (account_created_at_utc, signup_surface, is_internal), fct_event (is_core_action, flow_id, flow_instance_id, event_name, occurred_at_utc, received_at_utc) and fct_session. No further meeting with the director is possible before you start work. Deliver three clarifying questions you would send in writing, the metric you will compute in the meantime with its numerator, denominator, window and exclusions, and one sentence naming the question you are deliberately not answering.

Approach

Recognise what is being probed: whether you convert a goal into a computable predicate without stalling for requirements or guessing in silence. Listing clarifying questions is the generic answer; shipping a defensible default alongside them is the strong one, because the review is in two days and it will happen with or without you.
Infer the decision behind the request. A question about whether onboarding works, arriving before a planning cycle, usually means whether to staff it next quarter. That points at a rate with visible headroom over several cohorts, not at a descriptive dashboard.
Write the three questions so that each one changes the SQL. Which population, all signups or only self-serve from dim_user.signup_surface. What counts as working, reaching a core action or completing the onboarding flow_id. Against what bar, last quarter's cohorts or a stated target.
Propose the default explicitly: seven-day activation on weekly signup cohorts. Numerator, users with is_core_action = TRUE events on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). Denominator, the signup cohort with is_internal = FALSE. Publish with an eight-day lag, and state that the two-distinct-days threshold is a frozen choice rather than a discovery.
Name the exclusion in the same breath as the number. The series shows whether users activate; it does not establish that onboarding caused the level, which needs a staged rollout or an experiment.

Follow-up

The director replies that they meant the onboarding flow specifically, not activation. What changes in the query and in the caveats?
Your cohort metric needs an eight-day lag and the review is in two days. What do you present, and how do you label it?
Two of your three questions come back unanswered. Which one do you refuse to proceed without?