Hermes Corporate Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at Hermes Corporate, you are at the intersection of complex business operations and advanced analytical modeling. Your work is critical to driving data-informed decision-making, as you translate ambiguous business requirements into scalable, high-impact solutions. Whether you are building anomaly detection models for time-series sensor data or optimizing performance metrics, your role directly influences the efficiency and reliability of Hermes Corporate operations.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

• Define numerator, denominator and window precisely
• Pick a randomisation unit that respects interference
• Separate novelty effects from durable behaviour change

Overview

This position demands more than just technical proficiency; it requires a product-oriented mindset. You will be expected to own the end-to-end lifecycle of your models, from data pipeline construction to production deployment and monitoring. Because Hermes Corporate operates at significant scale, you will thrive here if you enjoy solving high-stakes problems, collaborating across cross-functional teams, and maintaining a high level of ownership over your deliverables.

The Hermes Corporate interview process

Hermes Corporate candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Technical Screens

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

• Whether the query you write matches the plan you just described
• What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
• Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

• Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
• Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
• Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Deep-Dive Problem Solving

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

• Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
• Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
• Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

• For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
• Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
• For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Behavioral Assessments

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

• Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
• Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
• Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

• For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
• Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
• Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Peer and Stakeholder Interaction

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Where candidates lose points

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

SQL that silently fans out on a one-to-many join

State the grain of each table and the grain you want in the result before writing the join. Pre-aggregate the many side to the join key, or use EXISTS or a window function, and verify with a row count against COUNT(DISTINCT id) rather than trusting that the numbers look plausible.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Questions & practice

Permutation test for a difference in conversion rates

Medium · permutation test · resampling · inference

Write a two-sided permutation test from scratch for a difference in conversion rates, using no scipy hypothesis function. Input: a DataFrame with unit_id, variant in {control, treatment} and converted in {0,1}, one row per randomisation unit. Compute the observed difference in proportions, then build the null distribution by reshuffling the variant labels while holding each arm's size fixed. Report the p-value as (1 + the count of permuted statistics at least as extreme in absolute value) / (B + 1) with B at least 10,000, and return the permutation distribution.

Approach

1. Name the null being tested: the sharp null that each unit's outcome is the same under either label. That is what licenses permuting labels, and it is stronger than the null of equal means, which matters when someone asks whether the test is valid under unequal variances.
2. Extract converted to a single numpy array of 0s and 1s and record n_treatment. Every permutation is then just a reshuffle of one array, and the treatment mean is the mean of the first n_treatment entries of the shuffled array.
3. Vectorise the B permutations with rng.permuted on a tiled 2-D array, or with argsort of a (B, n) random matrix. A Python loop calling np.random.shuffle B times is correct but roughly an order of magnitude slower and often runs past the time limit.
4. Use the +1 correction in both numerator and denominator. Without it a p-value of exactly 0 is reportable, which is false: the observed labelling is itself one of the permutations, so the smallest attainable p-value is 1/(B+1).
5. Compare the resulting p-value against a two-proportion z-test as a sanity check. At these sample sizes they should agree closely; a large divergence means the statistic or the shuffle is wrong, not that the permutation test found something subtle.

Follow-up

• The arms are 200 and 20,000 units. Does the permutation test stay valid, and what happens to its resolution at B = 10,000?
• Give a 95 percent confidence interval for the difference. Can you get it from this permutation distribution, and if not, what would you run instead?
• The randomisation unit is user_id but the outcome is per session. What breaks, and what is the fix?

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

1. State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
2. Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
3. The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
4. Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
5. Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
6. Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Follow-up

• The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
• Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
• Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

1. Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
2. Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
3. For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
4. Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
5. Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
6. Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

1. Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
2. Set ref = df['received_at_utc'].max() and build the five masks against it.
3. Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
4. Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
5. Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

• Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
• Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
• Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
• Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

• The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
• Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
• Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Using SQL window functions, how would you calculate a rolling average of user activity over the last 30 days?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Explain how you would optimize a slow-running query that joins multiple large tables.

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Attribute account MRR to first-touch channel without seat fan-out

Medium · fan-out · joins · revenue · attribution

dim_user holds user_id, account_id, first_touch_channel, account_created_at_utc. dim_account holds account_id, lifecycle_status. fct_subscription_period holds account_id, period_start_utc, period_end_utc, period_status, mrr_cents_constant_fx. For a given as-of date, report active MRR by the first-touch channel of the account's earliest-created user. An account has many users and many periods. The channel totals must sum exactly to total active MRR on that date, and accounts whose label is NULL must appear as their own bucket.

Approach

1. Reduce the label side to one row per account before any join: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY account_created_at_utc, user_id) = 1 over dim_user, carrying first_touch_channel. Joining dim_user to the period table directly repeats each revenue row once per seat.
2. Reduce the money side to one row per account too: the period where period_start_utc <= :as_of AND period_end_utc > :as_of AND period_status IN ('active','past_due'). If more than one row survives that filter the account holds overlapping subscriptions, which must be deduped or deliberately summed, and either way the choice is stated rather than left to the join.
3. Join the two one-row-per-account CTEs and SUM(mrr_cents_constant_fx). Use the constant-FX column, since the same query run in two quarters otherwise reports currency moves as channel performance.
4. Bucket NULL first_touch_channel as 'unattributed' with COALESCE instead of letting it drop. Roughly the share of users with no observed touch is not small, and dropping it understates the total and inflates every named channel's share.
5. Reconcile: the SUM over the joined result must equal the SUM over the period CTE alone. That single equality check catches fan-out immediately, which is why it is worth writing before presenting the number.

Worked solution (30 min)

1. Compute total active MRR on the as-of date from the period table alone and write the number down as the target.
2. Build the label CTE and assert COUNT(*) = COUNT(DISTINCT account_id).
3. Build the as-of period CTE and assert the same, investigating any account with more than one surviving row.
4. Join, COALESCE the channel, aggregate, and compare the total to the target.
5. Deliberately run the naive dim_user join and record the inflation factor, which is the MRR-weighted mean seat count.

Expected result: One row per first_touch_channel plus an 'unattributed' bucket, whose MRR values sum exactly to the independently computed total active MRR for the as-of date.

Verify it yourself

• Channel total equals the standalone total from the period CTE, to the cent.
• Both CTEs satisfy COUNT(*) = COUNT(DISTINCT account_id).
• The unattributed bucket is present and non-zero rather than silently absent.
• The naive join version produces a strictly larger total, and you can state the ratio.

Follow-up

• Which user should own the account's label: earliest created, the 'owner' role, or the one who converted? What does each choice systematically favour?
• What does this table entitle you to say about the value of a channel, and what does it not?
• Some accounts have a parent_account_id. How does rolling those up change both the numerator and the channel mix?

How do you balance short-term optimization with long-term product health?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• What would you do if the primary metric and the guardrail moved in opposite directions?

If we launch a new service, what leading indicators would you monitor to predict long-term engagement?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

A key product metric has suddenly dropped by 10%. How would you go about diagnosing the root cause?

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would you design a metric to track the success of a new feature?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

What are the most common experimentation pitfalls you have encountered when running A/B tests?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

If an experiment shows a positive result but the p-value is borderline, how do you proceed?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. State the primary metric and the minimum effect worth shipping, then size the test.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you do if you could not randomise at all?
• How would you handle interference between treated and control units?

Halve the runtime of a flat test with pre-period data

Medium · cuped · variance reduction · stratification

A test on 28-day core actions per user is powered for six weeks and the team will not wait. You have fct_event (user_id, occurred_at_utc, is_core_action) covering the period before each unit's first_exposed_at_utc in fct_experiment_exposure, and the pre-period count correlates with the in-experiment count at r = 0.6. Apply CUPED, state precisely how much runtime it buys, and handle the units with no pre-period because they signed up after the experiment started.

Approach

1. Define the adjusted outcome: Y_cuped = Y - theta (X - Xbar), with theta = Cov(Y, X) / Var(X) estimated on the pooled sample so theta itself carries no treatment effect. It is unbiased because X is pre-exposure and therefore has equal expectation in both arms.
2. Quantify the gain exactly: Var(Y_cuped) = Var(Y)(1 - r^2) = Var(Y) x 0.64, so 36% of the variance is removed. Required sample scales with variance, so six weeks becomes 3.8 weeks, and at the original sample the MDE shrinks by sqrt(0.64) = 0.8.
3. State the precondition that actually bites: X must be measured strictly before first_exposed_at_utc, not before assigned_at_utc and not before the calendar start of the experiment. Any post-exposure information in the covariate lets the treatment effect leak in and biases the estimate rather than merely failing to help.
4. Handle missing pre-periods as a stratum, never by dropping rows. Dropping changes the population; setting X to the pooled mean leaves those units' adjustment at zero, which is unbiased but buys them no reduction, so report the covered share and expect the blended reduction to fall short of 36%.
5. Note that stratification is the same mechanism with a categorical covariate. Post-stratifying on device_type and referrer_channel removes the variance those stratum means explain, and it composes with CUPED provided the strata are also fixed before exposure.

Worked solution (30 min)

1. Build the covariate: for each unit_id in fct_experiment_exposure, count fct_event rows with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc - 28 days, first_exposed_at_utc).
2. Estimate theta on the pooled sample as Cov(Y, X) / Var(X), and confirm Xbar is statistically indistinguishable between arms before using it.
3. Form Y_cuped = Y - theta (X - Xbar) and run exactly the test you would have run on Y.
4. For units with no pre-period, set X to the pooled mean so their adjustment is zero, and report what share of the sample that is.
5. Report unadjusted and adjusted estimates side by side, with the interval narrower on the adjusted one.

Expected result: 1 - 0.6^2 = 0.36 of the outcome variance is removed, so the required sample falls to 64% of the original and the six-week plan becomes about 3.8 weeks. At the original sample the MDE shrinks by a factor of 0.8. The blended reduction is smaller in proportion to the share of units with no pre-period, and that share is stated alongside the result.

Verify it yourself

• Adjusted and unadjusted point estimates should be close. CUPED reduces variance, it does not move the effect, so a large shift means X was imbalanced at baseline and needs investigating rather than celebrating.
• The realised standard error ratio should be near sqrt(1 - r^2) using the r observed in this experiment, not the historical r.
• Var(Y_cuped) must be no larger than Var(Y) for the fitted theta. A larger value means theta has the wrong sign or was estimated on the wrong window.

Follow-up

• Here the covariate is the same metric as the outcome. When would you deliberately pick a different one, and what is the risk if that covariate is itself affected by treatment?
• The point estimate moves by 0.4 standard errors when CUPED is applied. Is that reassuring or alarming, and what would you check?
• How does CUPED combine with cluster randomisation on account_id, and at which grain is the covariate built?

Separate a definition edit from a behaviour change

Easy · definition-change · metric-versioning · cohorts

Seven-day activation rate is documented as core actions on at least 2 distinct UTC dates inside the first seven days after account_created_at_utc, and every cohort published through last week was computed on that definition. The dashboard now reads 38% for the signup cohort that started Monday, against 31% for the cohort before it. Two things landed in that week: an onboarding change, and a commit to the metric's SQL that you have not read yet and that came with no documentation update. You have dim_user, fct_event and the metric's version history. Deliver two numbers: how many of the 7 points are behaviour and how many are the definition. Then say which published cohorts are no longer comparable.

Approach

1. Diff the metric SQL across the commit boundary and enumerate every changed predicate individually: the distinct-day threshold, the is_internal exclusion, the is_core_action source, the window bounds, and the identity key. A single commit routinely changes more than one, and each has its own sign.
2. Dual-run both definitions over the same 26 cohort weeks. The definition effect is the level shift between the two series computed on identical data, which isolates it from anything that happened in the world.
3. Read the behaviour effect only from the old-definition series, comparing the new cohort to its own trailing cohorts. The new-definition series cannot answer the behaviour question because it has no pre-period.
4. Explain the mechanism of the changed predicate rather than only its size. Lowering a distinct-day threshold from 2 to 1 admits every user who acted once in a single session, which is a large and low-intent population, so a several-point jump is the expected magnitude and not a surprise.
5. State the restatement plan: pick one definition, backfill the full history on it, and annotate the break date. A series carrying two definitions is not a time series.

Follow-up

• The onboarding team wants the 1-day threshold kept because it is easier to move. What is your argument, and what would change your mind?
• How would you make a definition change visible to every consumer of this metric without relying on people reading a changelog?

Suggested practice checklist

Instead of guessing where the week should go, day one measures it under a fixed rubric and allocates the remaining hours in proportion to the gaps. The method is deliberately rigid: the allocation is written down before any studying starts and is not renegotiated when a topic turns out to be unpleasant.

Day 1 — Diagnostic, scored before you study anything

• Sit a 100-minute timed diagnostic in four blocks: 30 minutes of SQL across three prompts, 25 minutes of short-answer statistics, 25 minutes on one modelling or case prompt, and 20 minutes delivering one behavioural story aloud.
• Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct, and 0 is stuck, grading the output rather than how the attempt felt.
• Allocate the hours for days two to five roughly in proportion to 3 minus the score in each block, write the allocation down, and commit to not revising it midweek.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

• Break the weakest area into five named sub-skills (for query work: grain control, window frames, date arithmetic, set logic with NULLs, and reading a query plan) and rate each one, so the rest of the week targets a sub-skill instead of a subject.
• Solve three problems chosen to sit just above where the rating drops off, and for each write the first move you failed to make.
• Re-solve one of them from memory four hours later, on paper, with nothing open.

Deliverable: A five-item sub-skill map with the two blocking sub-skills circled.

Day 3 — Largest gap: drill the blocking sub-skill

• Do eight short repetitions of the same shape rather than eight different problems, so what you practise is the pattern and not the puzzle.
• Write the rule you now hold in one sentence, then test it against a case built to break it: a ranking function over a column with ties, or a two-sample test on observations that are obviously dependent.
• Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on your strongest area

• Run the same sub-skill map and boundary protocol on the second-largest gap, compressed into half the day.
• Spend 25 timed minutes on your strongest area to stop it decaying, choosing the hardest problem you can still finish rather than an easy warm-up.
• Compare how the two areas fail: whether you lose time on recall, on setup, or on arithmetic, because the fix differs for each.

Deliverable: A second sub-skill map plus a one-line diagnosis of how each area fails you.

Day 5 — The gap that is not a skill

• Record yourself answering one technical and one behavioural prompt, then count two things in the playback: how many seconds before your first clarifying question, and how many sentences you started without knowing where they ended.
• Rewrite your three most-used stock phrases into shorter versions, and practise saying "I do not know, here is how I would find out" without softening it into a guess.
• Deliver one answer again with a hard 90-second limit to force structure before detail.

Deliverable: Two recordings with a counted improvement in time-to-first-question.

Day 6 — Retest under day-one conditions

• Sit the same 100-minute diagnostic structure with new prompts of comparable difficulty and score it on the identical rubric.
• Compare block by block, and for any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
• Write which single block you would still lose the offer on.

Deliverable: A second scored rubric placed next to the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

• Run a 60-minute mock covering the two blocks that moved least, with an interviewer instructed to interrupt and change direction.
• Write your recovery script for the moment you go blank: restate the question, state your assumption, name the first thing you would check.
• Reduce the week to the rule statements you wrote, each with its preconditions attached, then say every one of them out loud without reading it and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive being interrupted.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

How do you handle missing values or outliers when preparing a dataset for time-series analysis?

Medium · behavioural and stakeholder questions

Approach

1. Quantify the outcome, including what you would not claim credit for.
2. Name the disagreement or constraint, and how you resolved it with evidence.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• How did you know the outcome was caused by your change?
• What did you decide not to do, and why?

Handle a request for numbers supporting a decision already made

Hard · integrity · framing · stakeholder

A senior leader has already decided to sunset a plan tier and asks you for the analysis showing it is the right call. Accounts on that tier carry 6% of MRR at constant FX and have the highest licensed-seat utilisation in the book. The leader's support matters to your next review cycle, and the decision is being presented in four days. Deliver what you produce, what you decline to produce, and the exact sentence you will say in the meeting where the number appears on a slide.

Approach

1. Recognise what is being probed: whether you can find the legitimate request inside an illegitimate framing instead of either complying or refusing on principle. The generic answer promises to push back; the strong one produces something genuinely useful and states its limits in the room, without ambushing anybody.
2. Separate the decision from the justification. Sunsetting the tier may be correct for reasons the data does not hold, such as support cost, roadmap surface area or sales motion. What you decline is a one-sided document. What you produce is the case read both ways, which also happens to be more useful to the leader.
3. Build the symmetric analysis: MRR at risk at constant FX, the share of affected accounts with a plausible migration path given seats_licensed and billing_term, the recovery rate assumed for that migration and where it came from, and the downside case in which high-utilisation accounts treat the sunset as a reason to re-evaluate the vendor entirely.
4. Surface the inconvenient fact privately and early. The highest seat utilisation in the book is a retention signal, and the leader should hold it before the room does, so they can incorporate it rather than be caught by it.
5. Agree the meeting sentence in advance with the leader, so that nobody is surprised. Something to the effect that the tier is 6% of MRR and its accounts are the most heavily used in the book, and that the case for sunsetting rests on cost and focus rather than on revenue. That is true, it supports the decision on its real grounds, and it stops the deck claiming the numbers endorse it.
6. Decide your own line before you need it: what you will not put your name to, and that the route if asked anyway is your own manager rather than a confrontation in the meeting.

Follow-up

• The deck circulates with your analysis included and the downside case removed. What do you do, and by when?
• What changes if the honest analysis says the sunset is clearly the wrong call?
• How do you write the same memo when the leader is your skip-level and the meeting is tomorrow?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

1. Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
2. Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
3. Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
4. Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
5. Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

• The executive says to give the midpoint and they will manage the risk. What do you do?
• How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
• Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?