Interview guides · Earthcam

Earthcam Data Scientist Interview Questions & Guide 2026

Take-home style practice for data scientist candidates: a messy dataset, a fixed time budget, and an analysis you write up and defend.

The 60-second brief

The Data Scientist role at Earthcam is pivotal in transforming raw data into actionable insights that drive business decisions and enhance product offerings. As a Data Scientist, you will work with large datasets from Earthcam's innovative camera technologies, analyzing trends and patterns that inform both operational strategies and customer engagement. This position directly impacts the development of Earthcam's products, contributing to improved user experiences and operational efficiency.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

Strip CI, retry and synthetic traffic first
Separate contracted seats from actively used seats
Power experiments for heavy-tailed account revenue

Overview

Your work will involve collaboration with cross-functional teams, including engineering and product management, to develop models and algorithms that can predict user behavior and optimize service delivery. This role is critical not only for enhancing Earthcam's existing offerings but also for identifying new opportunities in the market, making it a dynamic and strategic position within the organization. Expect to engage with complex datasets and sophisticated analytical tools while driving the vision of Earthcam forward.

The Earthcam interview process

Earthcam candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Assessments

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Project Discussion

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Where candidates lose points

Randomising an experiment at the user level when users share an account

Two problems fire at once. Colleagues in one workspace see each other's work and talk to each other, so a treated user changes the behaviour of a control user in the same account, which violates the no-interference assumption and biases the estimate toward zero. Separately, outcomes within an account are strongly correlated, so the effective sample size is roughly n / (1 + (m - 1) * rho) for m users per account and intra-class correlation rho, not n. With rho around 0.3 and twenty users per account that is a design effect near 6.7, meaning a user-level confidence interval is about two and a half times narrower than it should be and results cross significance thresholds on noise alone. Randomise the account and cluster the standard errors.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Questions & practice

Discuss the architecture of a machine learning model you have built.

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Set a baseline first, so any model has something honest to beat.
Check what information would not exist at prediction time, and exclude it.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Can you explain the concept of overfitting and how to avoid it?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Audit daily usage rows for grain and arithmetic violations

Easy · data-quality · pandas · grain

You are handed fct_usage_daily as a pandas DataFrame with account_id, workspace_id, sku_code, usage_date, billable_quantity, included_quantity_applied, overage_quantity, list_amount_cents, discount_amount_cents, net_amount_cents, cogs_cents, is_restated, first_written_at and restated_at. The declared grain is one row per (account_id, workspace_id, sku_code, usage_date). Write audit(df) returning a DataFrame with one row per failing check: check name, failing row count, and one example key. Cover at minimum grain duplication, negative quantities or amounts, the identity net = list - discount, billable = included + overage, and rows where is_restated is true but restated_at is null.

Approach

Check the grain before anything else with df.duplicated(subset=key, keep=False), and count rows rather than groups so a key appearing twice contributes 2 — if the grain is broken every arithmetic count below it is uninterpretable.
Express each invariant as a boolean Series over the whole frame. The cent columns are integers and compare exactly, so use !=; the numeric(18,6) quantity columns need np.isclose with atol=1e-6 because included + overage is a decimal sum.
Handle null as its own failure mode. Comparisons against NaN return False, so a check written as rows_that_pass = (a == b - c) silently files every null-amount row wherever the negation happens to land; build each check as violations = ~condition | column.isna().
Collect the checks as a list of (name, mask) pairs and assemble the output in one pass, so adding a check is one line and every check reports in the same shape.
Order the output with structural failures (grain, null keys) above arithmetic failures, and report zero-count checks too — a check that silently disappears when it passes is indistinguishable from a check that was never run.

Worked solution (20 min)

key = ['account_id','workspace_id','sku_code','usage_date']; dup = df.duplicated(key, keep=False); record dup.sum() and df.loc[dup, key].iloc[0].to_dict() as the example.
neg = (df[['billable_quantity','included_quantity_applied','overage_quantity','list_amount_cents','net_amount_cents','cogs_cents']] < 0).any(axis=1); net_bad = df.net_amount_cents.isna() | (df.net_amount_cents != df.list_amount_cents - df.discount_amount_cents).
qty_bad = ~np.isclose(df.billable_quantity, df.included_quantity_applied + df.overage_quantity, atol=1e-6) | df.billable_quantity.isna(); restated_bad = df.is_restated & df.restated_at.isna().
Assemble pd.DataFrame([{'check': n, 'failing_rows': int(m.sum()), 'example': first_key(m)} for n, m in checks]) with the grain and null checks listed first.

Expected result: One row per check with failing_rows as an integer count of rows rather than groups, and a JSON-serialisable example key for any check with a non-zero count; a single doubled key contributes 2 to the grain check.

Verify it yourself

df.groupby(key).size().max() == 1 if and only if the grain check reports zero.
When the net check reports zero, sum(net_amount_cents) == sum(list_amount_cents) - sum(discount_amount_cents) exactly.
Running audit on df.sample(frac=1, random_state=0) gives identical counts; the checks must not depend on row order.
Corrupting one net_amount_cents by a single cent raises exactly one check's count by exactly 1.

Follow-up

Which of these should block a dashboard refresh and which should only warn?
Rows with is_restated = true legitimately change value after first write. How do you make yesterday's audit result reproducible?
How would you extend this to catch a partition that is missing entirely rather than wrong?

Can you explain how a hash table works?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Write a function to calculate the mean and median of a list of numbers.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Collapse retried requests into logical operations per account

Medium · window functions · deduplication · null semantics

fct_api_request carries request_id, account_id, api_key_id, idempotency_key (null when the caller supplied none), is_retry, request_at and http_status. For one ISO week, count logical operations rather than HTTP calls: requests sharing (account_id, api_key_id, idempotency_key) collapse to the earliest one, while every request with a null idempotency_key is its own operation. Return per account the raw request count, the logical operation count and the ratio between them, and state which direction a rising 5xx rate pushes that ratio.

Approach

Split the stream before deduplicating. GROUP BY and PARTITION BY treat NULLs as equal to one another, while the = operator does not, so a single grouping pass folds every null-key request in an account into one 'operation' and can cut the count by orders of magnitude.
Deduplicate the keyed rows with ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) and keep rn = 1. Including request_id in the ORDER BY makes the result deterministic when two rows share a millisecond, which they will.
UNION ALL the null-key rows back in unchanged; they need no dedup and must not pass through the partition.
Aggregate per account: count() over the raw week, count() over the union, and the ratio of the two. Report the ratio, not the difference, so accounts of very different sizes are comparable.
Cross-check against is_retry, which flags a resend of the same idempotency key: raw minus logical should sit close to the count of is_retry rows, and a large gap means clients are resending without a key and the dedup is under-counting duplicates.
Name the inversion explicitly: 5xx responses provoke client retries, so amplification rises exactly when reliability falls, and any engagement metric built on raw request counts will show growth during an outage.

Worked solution (25 min)

Filter to the ISO week with request_at >= :week_start AND request_at < :week_start + interval '7 days', and record the raw count per account as the baseline.
Build keyed_dedup: SELECT ... ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) AS rn FROM the week WHERE idempotency_key IS NOT NULL, then keep rn = 1.
Build unkeyed: the same week's rows WHERE idempotency_key IS NULL, taken as-is.
UNION ALL the two, then GROUP BY account_id selecting the logical count; join back to the raw counts and compute raw::numeric / logical.
Validate on one account by comparing raw - logical against COUNT(*) FILTER (WHERE is_retry) for that account and explaining any gap.

Expected result: One row per account with raw_requests, logical_operations and an amplification ratio at or above 1.0, where logical_operations counts each distinct (account_id, api_key_id, idempotency_key) once and each null-key request once.

Verify it yourself

logical_operations is never greater than raw_requests, and the two are equal for accounts whose keys are all distinct.
For accounts with no null idempotency_key, raw_requests - logical_operations is close to the count of is_retry rows.
Deliberately grouping on the key without the null split reproduces a visibly lower count, confirming the NULL collapse is the mechanism.
Re-running the dedup returns the same surviving request_id set, proving the ORDER BY tiebreak made it deterministic.

Follow-up

Which of these two counts belongs in the billable-units metric, and which in an engagement metric?
An account's amplification ratio jumps from 1.05 to 3.0 in a day. Name three causes and the query that separates them.

Explain how you would analyze customer usage patterns over time.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Describe your approach to designing a data pipeline for a new product feature.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you prioritize features based on data analysis?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Given a dataset, how would you approach feature selection?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Explain how you would implement data security in your models.

Medium

Approach

Work from the decision backwards to the evidence you would need.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Separate novelty from durable lift in a console redesign

Medium · novelty-effect · time-heterogeneity · holdout

A redesigned request console was tested for four weeks, randomised at the account level. The lift in weekly successful interactive requests per account was +9.2%, +5.1%, +1.8% and +0.4% in experiment weeks one through four, and the week-four 95% interval spans plus or minus 3.1 percentage points. Accounts entered on a rolling basis as they next signed in, so week one is a different calendar week for different accounts. Say what the decaying series can and cannot establish, specify the readout that would settle it, and state what you would ship on Monday.

Approach

Name the distinct explanations the same curve is consistent with, because the shipping decision differs across them: a novelty effect decaying to zero; a novelty effect decaying to a small positive plateau the test has no power to see; and a genuine effect whose measured size shrinks as control accounts learn about the change from colleagues or release notes.
Fix the time axis before interpreting anything. With rolling entry, calendar week and weeks-since-first-exposure are different variables, and pooling them mixes the decay curve with a change in cohort composition, since the accounts that signed in on day one are systematically the most engaged. Recut on weeks-since-first-exposure and verify that plan_tier and pre-period usage mix are stable across entry cohorts.
Read the interval width honestly. Plus or minus 3.1 percentage points at week four cannot separate a durable +1% from zero. The defensible statement is that no durable effect larger than roughly 3.5% was detected, not that there is no durable effect, and those two sentences lead to different decisions.
Separate novelty from primacy using account tenure. Accounts created after launch have no prior console to be surprised by, so they cannot show novelty; if the redesign is genuinely better their curve should be flat or rising, while established accounts show the spike and decay. If both cohorts decay to zero, the effect really was novelty.
Specify the settling readout rather than arguing about the four weeks you have. Hold back 5% of accounts as a long-run holdout for 90 days, and compare whole ISO weeks only: usage in this domain follows a hard five-to-two weekday cycle, and a window containing four business days instead of five moves this metric by several percent with no product change.
Give the Monday answer. Ship if guardrails are clean and maintenance cost is low, because a decayed-to-zero effect with no harm is a neutral trade; but book none of the +9.2% in any forecast, and do not run a follow-up test on the same accounts inside the novelty window, because their baseline has not returned to steady state.

Worked solution (30 min)

Recut the four weekly estimates on weeks-since-first-exposure and confirm entry-week cohorts are comparable on plan_tier and pre-period usage.
Split each week's estimate by account tenure, created before versus after launch, and compare the shapes of the two curves.
Compute the horizon needed to halve the week-four interval: precision scales with the square root of exposure, so about four times the account-weeks are required.
Write the ship note with the 5% holdout design, the 90-day re-read date, and an explicit statement of the effect sizes that remain unexcluded.

Expected result: The series is consistent with pure novelty, with a durable plateau anywhere below about 3.5% left unexcluded. Separating the explanations requires the tenure split plus a 5% holdout re-read at 90 days, and halving the week-four interval to plus or minus 1.55 points needs roughly four times the account-weeks.

Verify it yourself

Interval width scales with one over the square root of the sample, so quadrupling exposure should roughly halve 3.1 points. A plan claiming a larger gain is double-counting repeated weeks on the same accounts, which are correlated rather than independent.
Accounts created after launch cannot exhibit novelty against an experience they never had, so a week-one spike in that cohort points at an assignment or instrumentation problem rather than at novelty.
Every weekly comparison must be a whole ISO week. A window containing a different number of business days than its comparator is not a like-for-like contrast in this domain.

Follow-up

What sample or horizon would you need to rule out a durable +1.5% at 80% power, given the week-four interval you have?
How would you distinguish a novelty effect from a control arm that gradually learned about the change?
The metric that decides renewal is up to eleven months away on annual contracts. What proxy do you use in the meantime, and how do you validate it once renewals land?

Error rate halves while severe support tickets double

Hard · missing data · aggregation bias · reconciliation

The fleet-wide customer-visible error rate fell from 1.8 percent to 0.9 percent, while sev1 and sev2 tickets and reopened_count rose across the same fortnight. Using fct_api_request (account_id, environment, sdk_name, traffic_class, http_status, request_at, billable_units), fct_usage_daily (account_id, sku_code, usage_date, billable_quantity) and fct_support_ticket (account_id, severity, opened_at, reopened_count, linked_incident_id), determine whether reliability improved, and if not, identify precisely which rows are missing and from when. Deliverable: a diagnosis backed by an independent corroborating source.

Approach

Distrust an improvement that contradicts an independent operational signal. Two sources disagreeing is itself the finding; decide which one is more likely to be broken before explaining either.
Recompute the rate as the metric tree defines it, per account first and then as the share of accounts above the reliability target. A single global average is dominated by whichever account sends the most traffic, so a fleet number can fall while a quarter of accounts get worse.
Audit for missingness rather than for badness: count fct_api_request rows per hour split by environment, sdk_name and status class, indexed against the trailing same-hour baseline. A partial ingestion failure shows as a step drop confined to one slice, not a uniform decline.
Reconcile against a source the request pipeline does not feed, such as implied request volume from fct_usage_daily for the same accounts and dates. If the usage table is flat while request rows fell, rows are missing rather than traffic.
Test whether the missingness is differential by status, which is the mechanism that fakes an improvement: if 5xx rows are written on a path that stopped while 2xx rows were unaffected, the numerator falls faster than the denominator and the rate drops with nothing improving.
Close with the affected window, the affected slice and a restated series marked unreliable across that window, rather than a silently patched number.

Follow-up

The missing rows are unrecoverable. How do you present that fortnight in a series people compare week over week?
What monitor would have caught this within an hour, and what is its false-positive cost on a normal quiet weekend?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Your part, not the team's

Most data work is done by groups, so an interviewer has to work out which piece was yours. An answer that runs on 'we' for several minutes gets interrupted with a question about what you personally did, and by then the answer sounds defensive even when it is true. Mark your own contribution as you go, and name the parts that belonged to someone else instead of leaving them ambiguous. Keep a few specifics back as well, like the name of the metric or who actually objected, so a probe can be answered with something you had not already said.

How do you handle missing data in a dataset?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
Name the disagreement or constraint, and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Announce a metric fix that cuts the headline number

Medium · metric definitions · stakeholders · communication

Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.

Approach

The interviewer is probing whether you can land a correction as an operational change with a plan attached, rather than as an announcement other people then have to clean up after.
Quantify each exclusion separately before telling anyone: internal accounts, synthetic monitors, load tests. Three known quantities are a discussion; one alarming total is an argument.
Be precise about which side of the metric each exclusion touches, because one team's goal is on a count and the other's is on a ratio. The traffic-class filters remove requests, so they shrink the numerator only. Dropping internal accounts removes them from the ratio's denominator as well, since internal accounts carry ordinary account_status values and therefore sit in that denominator. Internal accounts are active in almost every week while the real base is not, so the numerator loses a larger share than the denominator and the ratio falls by less than the count does. Compute both and say which one the 11 percent is before anybody assumes.
Check whether the trend changes, not only the level. A constant 11 percent shift is a rebasing and nothing more. A shift that widens over time means the reported growth was partly internal or synthetic, which makes the existing goals unachievable as written and changes what you are asking teams to do.
Sequence the disclosure: the metric owner and the two teams whose goals move first and privately, then the board channel with a written bridge, then the dashboard. The dashboard is last because a number that changes without explanation is read as instability rather than as a fix.
Run both series for one reporting period with the bridge visible, restate history rather than letting the series break at a date, and set the date the old series is removed.
Propose the goal treatment yourself: rebase each target by the shift measured on the metric that target is written against, rather than leaving each team to negotiate individually, which is where corrections of this kind usually die.

Follow-up

One team's quarterly goal is now unreachable. Rebase the target or let it miss, and what does each choice teach the organisation?
How would this have been caught when the metric was first defined?
What else on that dashboard shares this failure mode, and how would you find out this week?

Defend a churn number twelve times the one in the board deck

Hard · stakeholder management · retention metrics · definitions

You recompute logo churn on the renewal-eligible base from fct_subscription_period, counting only accounts whose term_end_date fell in the month and allowing a 45-day grace for late paperwork. Annualised, about 14 percent of accounts that reach a renewal date do not renew. A revenue leader has been quoting 1.2 percent to the board for two quarters, computed by dividing non-renewed accounts by the entire customer base in each month and printing that monthly figure with no period attached. You have 20 minutes with that leader and the finance lead. Decide which number is reported from now on, and what happens to the two quarters already published.

Approach

The interviewer is probing whether you can hold a correct definition under social pressure without turning it into a competence dispute. Open by reproducing their 1.2 percent exactly, with their denominator and their months, so the disagreement is arithmetic both sides can see rather than a claim about who was careless.
Separate the two defects, because they are different in kind. The denominator is wrong: on annual contracts only about one twelfth of the base reaches a renewal date in any month, so an account eleven months from renewal sits in the denominator while being structurally incapable of entering the numerator, which suppresses the rate by a factor near twelve. The period is merely unstated: a monthly figure printed beside annual revenue targets gets read as an annual rate.
Say out loud that those two defects nearly cancel in the level, before the leader finds it. Twelve times 1.2 percent is about 14 percent, which is your number. That is the strongest thing you can say in the room, because it proves both figures rest on the same non-renewal count and moves the meeting onto which denominator and which period get published rather than onto whose query is right.
The level is recoverable; the series is not. Non-renewals in a month are the eligible base for that month times the churn rate, so dividing by a fixed whole base makes the published line proportional to how many contracts happen to come up that month. Where signings cluster at quarter ends, the eligible base in a quarter-end month can be several times a quiet month's, and the month-over-month moves the board has been reading as satisfaction are the signing calendar.
Separate the measurement change from a business change. Nothing got worse this week; the loss rate was always this. Bring net revenue retention over the same period as a ratio of sums on a cohort frozen twelve months earlier, because logo churn concentrated in small accounts can sit beside healthy revenue retention, and that combination is the actual story.
Offer a migration path rather than a correction. Report both rates for one quarter with a written bridge, restate the prior two quarters in an appendix instead of silently, and pin the definition, including the period it is stated over, somewhere finance and product both read it. Concede the limits of your own number: the 45-day grace means the most recent 45 days are not reportable, and churn must be dated on term_end_date rather than on updated_at. A strong answer volunteers this; a generic one only defends.

Follow-up

The leader multiplies their monthly figure by twelve, lands on your annual number, and concludes nothing was ever wrong. What do you say?
The leader says publishing the corrected rate costs the team its credibility with the board this quarter. What do you do?
Gross logo retention worsened while net revenue retention improved. Which do you lead with, and what does the combination tell you about who is leaving?