Interview guides · Covermymeds

Covermymeds Data Scientist Interview Questions & Guide 2026

SQL practice for Covermymeds data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

A Data Scientist at Covermymeds plays a pivotal role in transforming healthcare data into actionable insights that improve patient outcomes and streamline the medication prior authorization process. By leveraging advanced analytics, machine learning, and statistical modeling, you will help drive data-informed decision-making across the organization. This role is crucial not only for enhancing product offerings but also for ensuring that patients receive their medications promptly and efficiently.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

Size an experiment before anyone launches it
Decompose a metric move by segment and mix
Pick a randomisation unit that respects interference

Overview

The impact of this position resonates throughout various teams, including product development, user experience, and operations. As a Data Scientist, you will work on challenging problems that involve large datasets from diverse sources, contributing to innovative solutions like automated workflows and predictive modeling. The complexity and scale of the data you will handle are both demanding and rewarding, making this role an exciting opportunity for anyone passionate about data and its potential to transform healthcare.

The Covermymeds interview process

Covermymeds candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Behavioral Interviews

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Final Interviews

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Treating last-touch attribution as the causal value of a channel

The attribution label on dim_user is the output of a rule that assigns full credit to whichever touch happened to be recorded last inside a lookback window, and that rule systematically rewards channels that sit close to the conversion, especially branded search and retargeting, which largely intercept demand that already existed. Reallocating spend on those labels moves budget toward the channels that are best at being last, which is why attributed return on ad spend often improves while total signups do not. Nothing in the touchpoint data can settle this, because the counterfactual of not running the channel was never observed. The credible reads are a geo holdout or a scheduled pause, sized in advance on the total-signups metric rather than on the attributed one, and the honest framing in the meantime is that the label describes correlation with conversion and not incremental contribution.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

SQL that silently fans out on a one-to-many join

State the grain of each table and the grain you want in the result before writing the join. Pre-aggregate the many side to the join key, or use EXISTS or a window function, and verify with a row count against COUNT(DISTINCT id) rather than trusting that the numbers look plausible.

Extrapolating a first-week lift inflated by novelty effects

Plot the treatment effect by days since first exposure instead of quoting one pooled average. A lift that decays toward zero across the test window is behaviour that will not persist, and annualising it produces a forecast that misses by an order of magnitude.

Questions & practice

Write a function to calculate the mean and standard deviation of a list of numbers.

Medium · statistics and probability

Approach

Say what the estimate is of, and over what population it generalises.
Write down the assumption the method needs before you use the method.
Sanity-check the answer against a simple bound or a simulated case.

Follow-up

Which assumption here is most likely to be violated in practice?
What sample size would you need to detect an effect half this size?

Describe a machine learning algorithm you have implemented and its outcomes.

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Implement seven-day activation from its written definition

Medium · metric definition · cohorts · pandas · joins

Implement the seven-day activation rate. Inputs: dim_user with user_id, account_created_at_utc and is_internal; fct_event with user_id, occurred_at_utc and is_core_action. A user activates when core-action events carrying a non-NULL user_id fall on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). The denominator is every non-internal user whose account_created_at_utc lands in the cohort week, including users with no events at all. Return one row per cohort week with numerator, denominator and rate, publishing only weeks whose last signup is at least eight days old.

Approach

Build the denominator first, from dim_user alone, filtered on is_internal = False. Deriving it from the join is the standard way to lose every user who never fired an event, which is exactly the population the metric is about.
Join events to users on user_id with a left join from the user side, then apply the window as a half-open interval: occurred_at >= created AND occurred_at < created + 7 days. The right bound is exclusive, so an event at exactly created + 7 days does not count.
Count distinct UTC dates per user, not distinct events. Floor occurred_at_utc to date before the nunique, and do it in UTC rather than local time so the threshold does not move with the user's country.
Apply the >= 2 threshold, aggregate to cohort week, and compute the rate by re-summing numerator and denominator per week rather than averaging any per-user or per-day rate. Fix the week anchor explicitly: cohort_week is the Monday of the signup week in UTC, which is what Postgres DATE_TRUNC('week') returns and what any SQL version of this metric will produce. In pandas, subtract dt.weekday days from the floored timestamp. If you reach for periods instead, the anchor that matches is to_period('W') (equivalently 'W-SUN'), whose weeks end Sunday and therefore start Monday; to_period('W-MON') labels weeks that end on Monday, so it runs Tuesday through Monday and its start_time is a Tuesday. Mixing the two shifts every cohort label by one day and silently moves Mondays into the previous week.
Suppress immature weeks: drop any cohort week whose maximum account_created_at_utc is within 8 days of the data cut, and return them as absent rather than as a partial number.

Worked solution (30 min)

users = dim_user[~dim_user.is_internal].copy(); created = users['account_created_at_utc']; users['cohort_week'] = created.dt.floor('D') - pd.to_timedelta(created.dt.weekday, unit='D'), which is the Monday-start week. The period spelling that agrees with it is created.dt.to_period('W').dt.start_time; 'W-MON' does not agree and is off by a day.
ev = events[events.is_core_action & events.user_id.notna()]; merge onto users on user_id with how='inner' for the numerator side only.
Filter to the half-open window, add ev_date = occurred_at_utc.dt.date, group by user_id and count distinct dates, keep users with >= 2.
numer = users.merge(activated_user_ids, how='left', indicator=True) then group by cohort_week and sum the indicator; denom = users.groupby('cohort_week').size().
rate = numer / denom; drop weeks where users.groupby('cohort_week')['account_created_at_utc'].max() > data_max - 8 days.

Expected result: One row per mature cohort week with numerator, denominator and rate, where denominator equals the count of non-internal signups in that week regardless of activity, numerator is less than or equal to denominator, and the most recent 1 to 2 weeks are absent rather than reported low.

Verify it yourself

For a synthetic user with two core actions 90 minutes apart on the same UTC date, activation is False.
For a synthetic user with one core action on day 0 and one at exactly created + 7 days, activation is False; move the second to created + 6 days 23 hours and it becomes True.
Pin the week anchor with two users created at Sunday 23:59 UTC and the following Monday 00:01 UTC: they must land in different cohort weeks, and the Sunday user's cohort_week must be the Monday six days before them. to_period('W-MON') puts both in one period, which is the off-by-one this check exists to catch.
The denominator summed across all published weeks equals the count of non-internal users created in that date range.
Re-summing numerator and denominator to a monthly figure gives a different answer from averaging the weekly rates; the re-summed one is the metric.

Follow-up

The threshold is 2 distinct days. What changes in the reported history if someone moves it to 3, and how would you publish that change?
Invited seats and SSO-provisioned users have no pre-signup session. Should they be in this denominator at all, and what does including them do to the rate for sales-assisted accounts?
How would you produce the same metric at account grain, and which of the two would you put on the dashboard?

How would you optimize a slow SQL query?

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Can you explain the time complexity of your solution?

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Attribute account MRR to first-touch channel without seat fan-out

Medium · fan-out · joins · revenue · attribution

dim_user holds user_id, account_id, first_touch_channel, account_created_at_utc. dim_account holds account_id, lifecycle_status. fct_subscription_period holds account_id, period_start_utc, period_end_utc, period_status, mrr_cents_constant_fx. For a given as-of date, report active MRR by the first-touch channel of the account's earliest-created user. An account has many users and many periods. The channel totals must sum exactly to total active MRR on that date, and accounts whose label is NULL must appear as their own bucket.

Approach

Reduce the label side to one row per account before any join: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY account_created_at_utc, user_id) = 1 over dim_user, carrying first_touch_channel. Joining dim_user to the period table directly repeats each revenue row once per seat.
Reduce the money side to one row per account too: the period where period_start_utc <= :as_of AND period_end_utc > :as_of AND period_status IN ('active','past_due'). If more than one row survives that filter the account holds overlapping subscriptions, which must be deduped or deliberately summed, and either way the choice is stated rather than left to the join.
Join the two one-row-per-account CTEs and SUM(mrr_cents_constant_fx). Use the constant-FX column, since the same query run in two quarters otherwise reports currency moves as channel performance.
Bucket NULL first_touch_channel as 'unattributed' with COALESCE instead of letting it drop. Roughly the share of users with no observed touch is not small, and dropping it understates the total and inflates every named channel's share.
Reconcile: the SUM over the joined result must equal the SUM over the period CTE alone. That single equality check catches fan-out immediately, which is why it is worth writing before presenting the number.

Worked solution (30 min)

Compute total active MRR on the as-of date from the period table alone and write the number down as the target.
Build the label CTE and assert COUNT(*) = COUNT(DISTINCT account_id).
Build the as-of period CTE and assert the same, investigating any account with more than one surviving row.
Join, COALESCE the channel, aggregate, and compare the total to the target.
Deliberately run the naive dim_user join and record the inflation factor, which is the MRR-weighted mean seat count.

Expected result: One row per first_touch_channel plus an 'unattributed' bucket, whose MRR values sum exactly to the independently computed total active MRR for the as-of date.

Verify it yourself

Channel total equals the standalone total from the period CTE, to the cent.
Both CTEs satisfy COUNT(*) = COUNT(DISTINCT account_id).
The unattributed bucket is present and non-zero rather than silently absent.
The naive join version produces a strictly larger total, and you can state the ratio.

Follow-up

Which user should own the account's label: earliest created, the 'owner' role, or the one who converted? What does each choice systematically favour?
What does this table entitle you to say about the value of a channel, and what does it not?
Some accounts have a parent_account_id. How does rolling those up change both the numerator and the channel mix?

Can you walk us through a complex problem you solved in a previous project?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Given a dataset, how would you approach identifying trends and anomalies?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

How do you prioritize tasks when faced with multiple deadlines?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you design an experiment to test a new feature in a product?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Say whether units interfere with each other, and switch design if they do.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
How would you handle interference between treated and control units?

What is the difference between supervised and unsupervised learning?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.
Clarify what is being asked and what a complete answer would contain.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Define success for a rebuilt first-run onboarding checklist

Easy · activation · metric tree · cohorts

A rebuilt first-run checklist ships to all new signups next month. You have dim_user (user_id, account_created_at_utc, signup_surface, is_internal) and fct_event (user_id, occurred_at_utc, is_core_action, event_name, surface). Propose a primary metric with an explicit numerator, denominator, window and publication lag, plus two guardrails and one diagnostic you would refuse to treat as success. The team wants a number it can read weekly, and the constraint is that everything must be computable from those two tables alone. Deliver the metric tree from the north star down to the metric you chose.

Approach

Fix the grain and the cohort key first: the checklist is seen by users, so count on user_id with account_created_at_utc as the cohort key, and say out loud that the north star is account-grained so the tree crosses grains here deliberately rather than by accident.
Take seven-day activation as the primary — core action on at least 2 distinct UTC dates inside [account_created_at_utc, +7 days) — because the two-distinct-days predicate cannot be satisfied by the single checklist-completion click the feature itself produces.
Demote checklist completion rate to a diagnostic and give the reason: it is an output of the feature, so it is near-perfectly correlated with having shipped the feature and cannot fall when the feature is bad.
Pick guardrails by the failure each one catches, not by what is easy to query: week-4 signup-cohort retention catches an activation gain that does not persist, and p50 minutes to first core action catches a checklist that adds steps to a path users already completed.
State the operational rules explicitly: is_internal = FALSE, an 8-day publication lag, and signup_surface values 'invite' and 'sso_provisioned' split out because those users arrive through an administrator rather than a self-serve signup and may not be shown the checklist at all.

Worked solution (20 min)

Write the primary in full: numerator = cohort users with is_core_action = TRUE events on at least 2 distinct UTC dates in [account_created_at_utc, +7 days); denominator = all dim_user rows in the cohort week with is_internal = FALSE.
Draw the tree downward: weekly active accounts completing a core action, then seven-day activation rate multiplied by weekly signups, then checklist step completion and time-to-first-core-action labelled as diagnostics.
Attach the timing rules: 8-day lag, and a cohort with fewer than seven full elapsed days published blank rather than partial.
Attach each guardrail to its named failure mode in one line each, so the guardrail list reads as a list of specific risks rather than a list of metrics.
Write the exclusion rule and the surface split, and state what you would do if 'invite' users turn out to be a third of the cohort.

Expected result: A primary metric stated as numerator, denominator, window, lag and exclusions; two guardrails each tied to a named failure mode; and checklist completion explicitly demoted to a diagnostic with the reason given.

Verify it yourself

The denominator includes cohort users who never returned. If it is built from users with any event, the metric is conditioned on survival and will read several points too high.
Recomputing a cohort's activation rate three days after first publication must return the same number. If it moves, the right-hand window bound or the late-arrival rule is missing.
The per-surface cohort counts must sum to the unsplit cohort count, so confirm signup_surface IN ('invite','sso_provisioned') is broken out rather than dropped. Note the limit while you are there: neither dim_user nor fct_event carries a pre-signup identifier, so whether those users were ever shown the checklist is not answerable from these two tables — the surface split is the only handling available, and the metric definition has to say that rather than imply exposure was verified.

Follow-up

Activation rises three points but week-4 retention is flat. What do you tell the team, and what would you need to distinguish a real gain from pulled-forward activity?
The checklist ships on web only. What changes in the denominator, and what breaks if you leave every surface in?
Signup mix shifted toward 'invite' the same week. How would you show whether the activation move was mix or behaviour?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Describe a time when you had to communicate complex data findings to a non-technical audience.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Turn an ambiguous onboarding question into a measurable metric

Easy · scoping · metric definition · stakeholder

Two days before a planning review, a director asks whether onboarding is working. You have dim_user (account_created_at_utc, signup_surface, is_internal), fct_event (is_core_action, flow_id, flow_instance_id, event_name, occurred_at_utc, received_at_utc) and fct_session. No further meeting with the director is possible before you start work. Deliver three clarifying questions you would send in writing, the metric you will compute in the meantime with its numerator, denominator, window and exclusions, and one sentence naming the question you are deliberately not answering.

Approach

Recognise what is being probed: whether you convert a goal into a computable predicate without stalling for requirements or guessing in silence. Listing clarifying questions is the generic answer; shipping a defensible default alongside them is the strong one, because the review is in two days and it will happen with or without you.
Infer the decision behind the request. A question about whether onboarding works, arriving before a planning cycle, usually means whether to staff it next quarter. That points at a rate with visible headroom over several cohorts, not at a descriptive dashboard.
Write the three questions so that each one changes the SQL. Which population, all signups or only self-serve from dim_user.signup_surface. What counts as working, reaching a core action or completing the onboarding flow_id. Against what bar, last quarter's cohorts or a stated target.
Propose the default explicitly: seven-day activation on weekly signup cohorts. Numerator, users with is_core_action = TRUE events on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). Denominator, the signup cohort with is_internal = FALSE. Publish with an eight-day lag, and state that the two-distinct-days threshold is a frozen choice rather than a discovery.
Name the exclusion in the same breath as the number. The series shows whether users activate; it does not establish that onboarding caused the level, which needs a staged rollout or an experiment.

Follow-up

The director replies that they meant the onboarding flow specifically, not activation. What changes in the query and in the caveats?
Your cohort metric needs an eight-day lag and the review is in two days. What do you present, and how do you label it?
Two of your three questions come back unanswered. Which one do you refuse to proceed without?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

The executive says to give the midpoint and they will manage the risk. What do you do?
How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?