Hitachi Data Scientist Interview Questions & Guide 2026

The 60-second brief

At Hitachi, a Data Scientist plays a pivotal role in bridging the gap between advanced digital technology and physical social infrastructure. Operating at the intersection of information technology (IT) and operational technology (OT), Hitachi leverages data science to power its Social Innovation business. This means you will not just build isolated models; you will develop scalable algorithms that optimize smart grids, streamline industrial manufacturing, enhance transportation systems, and improve healthcare delivery globally.

Seniority shifts the scope more than the words in the title do. Earlier-career loops mostly check that you execute a well-posed analysis correctly; senior loops check that you can decide which question is worth answering and defend what you chose not to do.

Focus areas

• Power experiments for heavy-tailed account revenue
• Read NRR on a fixed account cohort
• Analyse at the account grain, cluster errors

Overview

As a Data Scientist, your work directly impacts Hitachi's core platforms, including Lumada—the company's advanced digital solutions and services suite. You will work alongside researchers, domain engineers, and product managers to extract actionable intelligence from massive, complex datasets. The challenges you face will require a unique blend of deep scientific curiosity and practical engineering to deliver robust, production-ready machine learning solutions.

This role is highly collaborative and intellectually demanding. Whether you are embedded in a specialized research lab or working within a business unit, you will be expected to tackle ambiguous, real-world problems. Success in this position requires not only technical excellence in statistics and machine learning but also the ability to translate complex data findings into strategic business value.

The Hitachi interview process

Hitachi candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

CV Screening

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

• Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
• The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
• Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

• Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
• Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
• Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Technical Screen

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Practical Assignment

Before any modelling, the dataset is itself the first test. Take-home data usually carries something broken: rows duplicated at an unexpected grain, a join that silently drops part of the population, timestamps stored in more than one timezone, or missingness correlated with the outcome. An hour spent profiling row counts, key uniqueness and date ranges is not overhead, because it decides whether every number after it is real. What separates submissions is whether you report the defects you found and adapt the analysis to them, rather than modelling over them quietly and hoping the aggregate absorbs it.

What to demonstrate

• Whether you established the grain of each table and checked row counts after every join, and said so in the writeup
• Whether data defects you found are surfaced with their effect on the conclusion, instead of being dropped without comment
• Whether filters and exclusions are reproducible from the submitted code, with the size of the excluded population quantified

How to prepare

• Write a short profiling script you can point at any unfamiliar table: row count, distinct key count, null rate per column, and the min and max of every date field, then run it before anything else
• Write the funnel or the join chain as one query and check the row count at each grain, so a silent fan-out shows up as a number rather than as a wrong answer later
• On a past dataset, list every exclusion you applied and how many rows each one removed, then draft the single sentence about it you would put in a report

Deep-Dive Technical Interview

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Discussion

This round decides whether you owned a decision or watched one happen nearby. Interviewers for data roles listen for the point where the analysis stopped being a report and started changing what someone did, so build each story around that hinge: what was going to happen by default, what you found, and what happened instead. The most common weakness is a story that ends at delivery. If you can name the decision your work changed and the number that moved because of it, most follow-ups become easy.

What to demonstrate

• Whether the decision was yours to influence, or whether you are narrating a team outcome in the first person
• The counterfactual: what would have been done without your analysis, and why that default was worse
• How far your involvement ran past the handoff, and whether you checked that the change did what you predicted

How to prepare

• Pick three projects and write one sentence for each naming the decision-maker, the choice in front of them, and what they chose after seeing your work. If you cannot name a person and a choice, the story is not ready for this round.
• Reconstruct the baseline for your strongest project from the original query or dashboard rather than memory, so the before-number survives a follow-up asking where it came from.
• Prepare an honest version of a project where your recommendation was overruled, including what you did with the analysis afterwards.

Compensation Negotiation

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Where candidates lose points

Reporting a mean over accounts when account revenue is heavy-tailed

When a small number of accounts hold most of the revenue, the sample mean is dominated by whichever of them happens to be in the sample, and the sample variance keeps growing as more data arrives instead of stabilising. In that regime the usual central-limit-based confidence interval understates uncertainty, and a single renewal or a single large account's batch job can flip the sign of a measured effect. The fixes are to pre-register a winsorisation or capping rule before looking at the outcome, to report account counts crossing a threshold alongside the revenue figure, or to define the estimand on a bounded transform. Choosing the cap after seeing the result is a separate and worse problem, because the cap then encodes the answer.

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Sizing estimates built on unnamed, unrevisable assumptions

Write each assumption as a named number you can change, then show the arithmetic so the interviewer can challenge one input instead of the whole answer. Finish by saying which assumption the result is most sensitive to, which matters more than the point estimate.

Over-explaining the method and under-explaining the implication

Lead with the answer and what you would do about it, then give the approach when asked. Roughly one sentence of method per three of implication is the right ratio for a stakeholder-facing answer; the interviewer already knows what a regression is.

Questions & practice

Walk me through the steps of conducting a chi-square test for independence.

Medium · statistics and probability

Approach

1. Quantify uncertainty explicitly rather than reporting a point estimate alone.
2. Translate the result into the decision it informs, in one plain sentence.
3. Say what the estimate is of, and over what population it generalises.

Follow-up

• What sample size would you need to detect an effect half this size?
• How would you explain this result to someone who does not know statistics?

Explain the concept of statistical power and the factors that influence it.

Medium · statistics and probability

Approach

1. Say what the estimate is of, and over what population it generalises.
2. Sanity-check the answer against a simple bound or a simulated case.
3. Translate the result into the decision it informs, in one plain sentence.

Follow-up

• How would you explain this result to someone who does not know statistics?
• What sample size would you need to detect an effect half this size?

How do you validate a predictive model when historical data is extremely sparse?

Medium · machine learning and modelling

Approach

1. Say how the offline result would be validated online before it is trusted.
2. Frame the prediction: the label, the moment of prediction, and the action it triggers.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

How do you prevent overfitting in deep neural networks?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• What would you monitor after launch to know the model is still valid?
• How would you choose the decision threshold, and who owns that choice?

Audit daily usage rows for grain and arithmetic violations

Easy · data-quality · pandas · grain

You are handed fct_usage_daily as a pandas DataFrame with account_id, workspace_id, sku_code, usage_date, billable_quantity, included_quantity_applied, overage_quantity, list_amount_cents, discount_amount_cents, net_amount_cents, cogs_cents, is_restated, first_written_at and restated_at. The declared grain is one row per (account_id, workspace_id, sku_code, usage_date). Write audit(df) returning a DataFrame with one row per failing check: check name, failing row count, and one example key. Cover at minimum grain duplication, negative quantities or amounts, the identity net = list - discount, billable = included + overage, and rows where is_restated is true but restated_at is null.

Approach

1. Check the grain before anything else with df.duplicated(subset=key, keep=False), and count rows rather than groups so a key appearing twice contributes 2 — if the grain is broken every arithmetic count below it is uninterpretable.
2. Express each invariant as a boolean Series over the whole frame. The cent columns are integers and compare exactly, so use !=; the numeric(18,6) quantity columns need np.isclose with atol=1e-6 because included + overage is a decimal sum.
3. Handle null as its own failure mode. Comparisons against NaN return False, so a check written as rows_that_pass = (a == b - c) silently files every null-amount row wherever the negation happens to land; build each check as violations = ~condition | column.isna().
4. Collect the checks as a list of (name, mask) pairs and assemble the output in one pass, so adding a check is one line and every check reports in the same shape.
5. Order the output with structural failures (grain, null keys) above arithmetic failures, and report zero-count checks too — a check that silently disappears when it passes is indistinguishable from a check that was never run.

Worked solution (20 min)

1. key = ['account_id','workspace_id','sku_code','usage_date']; dup = df.duplicated(key, keep=False); record dup.sum() and df.loc[dup, key].iloc[0].to_dict() as the example.
2. neg = (df[['billable_quantity','included_quantity_applied','overage_quantity','list_amount_cents','net_amount_cents','cogs_cents']] < 0).any(axis=1); net_bad = df.net_amount_cents.isna() | (df.net_amount_cents != df.list_amount_cents - df.discount_amount_cents).
3. qty_bad = ~np.isclose(df.billable_quantity, df.included_quantity_applied + df.overage_quantity, atol=1e-6) | df.billable_quantity.isna(); restated_bad = df.is_restated & df.restated_at.isna().
4. Assemble pd.DataFrame([{'check': n, 'failing_rows': int(m.sum()), 'example': first_key(m)} for n, m in checks]) with the grain and null checks listed first.

Expected result: One row per check with failing_rows as an integer count of rows rather than groups, and a JSON-serialisable example key for any check with a non-zero count; a single doubled key contributes 2 to the grain check.

Verify it yourself

• df.groupby(key).size().max() == 1 if and only if the grain check reports zero.
• When the net check reports zero, sum(net_amount_cents) == sum(list_amount_cents) - sum(discount_amount_cents) exactly.
• Running audit on df.sample(frac=1, random_state=0) gives identical counts; the checks must not depend on row order.
• Corrupting one net_amount_cents by a single cent raises exactly one check's count by exactly 1.

Follow-up

• Which of these should block a dashboard refresh and which should only warn?
• Rows with is_restated = true legitimately change value after first write. How do you make yesterday's audit result reproducible?
• How would you extend this to catch a partition that is missing entirely rather than wrong?

Net revenue retention on a cohort frozen twelve months back

Hard · retention · as-of join · cohort arithmetic

fct_subscription_period carries subscription_period_id, account_id, arr_cents, plan_code, term_start_date, term_end_date, booked_at, amendment_type, superseded_by_id (the subscription_period_id of the version that replaced this one, null on the live version of a lineage) and is_current. Compute net revenue retention for month M: the summed arr_cents at M for the set of accounts holding arr_cents > 0 at M-12, divided by that same set's arr_cents at M-12. An account can hold more than one live subscription, a churned account contributes zero rather than dropping out, and nothing signed after M-12 may enter either side. Return the ratio plus the expansion, contraction and churn components in cents.

Approach

1. Write one reusable as-of ARR snapshot parameterised by a date: rows whose term brackets the date AND whose booked_at is at or before the date, then only the version of each lineage that is still live at that date, then sum arr_cents per account. The booked_at guard matters because an amendment signed in advance otherwise co-exists with the term it replaces and double counts the account.
2. Collapse the lineage on superseded_by_id, not on any attribute of the contract. Keep a row when superseded_by_id IS NULL, or when the successor it points at was booked after the date. Ranking with ROW_NUMBER() OVER (PARTITION BY account_id, plan_code ...) instead is wrong in both directions: an amendment that moves the account from one plan_code to another puts the old and new versions in different partitions, so both are rank 1, both bracket the date, and the account's arr_cents is counted twice; and two genuinely concurrent subscriptions that happen to share a plan_code land in one partition, so one of them is deleted.
3. Resolve the successor with a LEFT JOIN back to fct_subscription_period on subscription_period_id, and treat a missing successor as not superseded. An inner join would silently delete an account's ARR on a dangling pointer, which is a data-quality bug in the source, not a retention movement.
4. Never use is_current for the M-12 side. is_current describes today; using it at the historical snapshot backdates the present contract onto last year's cohort and makes retention look like 100 percent by construction.
5. Freeze the cohort from the M-12 snapshot where arr_cents > 0, then LEFT JOIN the M snapshot onto it and COALESCE the missing side to zero. An inner join deletes exactly the churned accounts, which is the single largest way this number gets overstated.
6. Return a ratio of sums, not a mean of per-account ratios. The two are different estimands: contraction is floored at zero while expansion is unbounded, so the mean of ratios is both biased relative to the aggregate and far noisier on a skewed revenue base.
7. Decompose per account on the delta: positive delta is expansion, negative delta with a non-zero M value is contraction, a zero M value is churn. The three components must reconcile to numerator minus denominator.
8. Prove no leakage: any account whose first contract began after M-12 must be absent from both sides, and the cohort row count must be identical in the numerator and denominator.

Worked solution (45 min)

1. Write arr_asof(d) as a CTE or lateral: from fct_subscription_period s take rows with term_start_date <= d AND term_end_date >= d AND booked_at <= d, LEFT JOIN fct_subscription_period succ ON succ.subscription_period_id = s.superseded_by_id, keep the row when s.superseded_by_id IS NULL OR succ.subscription_period_id IS NULL OR succ.booked_at > d, then sum arr_cents per account_id across every surviving version.
2. Sanity-check the lineage rule on one amended account before going further: at a date after the amendment, the account must contribute exactly one version per lineage even when the amendment changed plan_code, term dates or both.
3. Materialise base = arr_asof(M-12) filtered to arr_cents > 0, and curr = arr_asof(M).
4. LEFT JOIN curr onto base on account_id and COALESCE(curr.arr_cents, 0) AS arr_now.
5. Compute nrr = sum(arr_now)::numeric / NULLIF(sum(base.arr_cents), 0), and the three components with SUM(...) FILTER on the sign of arr_now - base.arr_cents and on arr_now = 0.
6. Reconcile: assert sum(arr_now) - sum(base.arr_cents) = expansion - contraction - churn, and assert the cohort account count is identical on both sides.

Expected result: A single row with nrr as a ratio, plus expansion_cents, contraction_cents and churn_cents as positive magnitudes satisfying numerator minus denominator equals expansion minus contraction minus churn, over a cohort whose membership is fixed at M-12.

Verify it yourself

• At each snapshot date, the surviving row count per account equals the number of distinct live subscription lineages on that account, not the number of contract versions bracketing the date.
• For an account whose amendment changed plan_code, the snapshot returns one version and its snapshot ARR equals the amended arr_cents; swapping in a ROW_NUMBER partition on (account_id, plan_code) returns two versions for that account and inflates the side it sits on, which is the fastest way to see the bug.
• The three components reconcile to the numerator-denominator delta to the cent.
• The distinct account count is the same in the numerator and the denominator.
• No account with a first term_start_date after M-12 appears anywhere in the output.
• Re-running after new accounts sign leaves the ratio unchanged, which is the defining property of a frozen cohort.

Follow-up

• Net revenue retention can rise while the business shrinks. Show one mechanism and name the guardrail that catches it.
• How do you handle an account that co-terms two subscriptions into one mid-window, so the subscription count changes but the money does not?
• Finance computes this from invoiced amounts and gets a different number. Which is right for which question?

Consecutive qualifying weeks before renewal, as a ranked worklist

Hard · gaps and islands · window functions · date boundaries

fct_api_request carries account_id, workspace_id, environment, request_at (timestamptz), http_status and traffic_class. fct_subscription_period carries account_id, term_end_date, auto_renew and is_current. A qualifying week for an account is an ISO week with at least 50 successful production requests in traffic_class ('interactive','batch'). Over the last 52 whole ISO weeks, and for accounts whose current term ends within 90 days, return the current qualifying-week streak length, the week it began, and the longest earlier streak. An account whose streak has broken must appear with a current length of zero.

Approach

1. Bucket weeks as date_trunc('week', request_at AT TIME ZONE 'UTC'). request_at is a timestamptz, so an unpinned date_trunc silently uses the session time zone, weeks start at a local midnight, and Monday-morning traffic lands in the previous week for part of the fleet. Pinning UTC also makes the seven-day arithmetic below exact across daylight-saving transitions.
2. Apply the exclusions before counting: environment = 'production', http_status < 400, traffic_class IN ('interactive','batch'). Then apply the volume floor and drop the current partial week, which can never meet a floor calibrated on whole weeks.
3. Build islands with the row-number anchor: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) as rn, then week_start - rn * interval '7 days' is constant inside a run of consecutive weeks. Group by that anchor to get each streak's start, end and length.
4. The current streak is the island whose end equals the last whole week; if none does, the account's current streak is zero and that is the interesting case. The longest earlier streak is the maximum length among the remaining islands.
5. Join the renewal filter from the current subscription row and LEFT JOIN the streak summary so an account with no qualifying week at all still appears, rather than vanishing from the risk list precisely because it went quiet.
6. Finish with an operating point. The list is worked by a team with finite capacity, so order it and cut it at that capacity, and say what happens to the accounts below the line.

Follow-up

• The floor of 50 requests was picked for this exercise. How would you calibrate it from data, and what would force a recalibration?
• A regional holiday week drops several accounts below the floor at once. How do you keep that out of the risk list?
• How would you evaluate whether contacting these accounts actually changed renewal, given that coverage is assigned deliberately?

How would you approach predicting customer churn for an enterprise software service?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

Describe how you would clean, preprocess, and feature-engineer a highly noisy, unstructured dataset.

Medium · product sense and metrics

Approach

1. Restate the decision this analysis has to support, and who acts on the answer.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Separate novelty from durable lift in a console redesign

Medium · novelty-effect · time-heterogeneity · holdout

A redesigned request console was tested for four weeks, randomised at the account level. The lift in weekly successful interactive requests per account was +9.2%, +5.1%, +1.8% and +0.4% in experiment weeks one through four, and the week-four 95% interval spans plus or minus 3.1 percentage points. Accounts entered on a rolling basis as they next signed in, so week one is a different calendar week for different accounts. Say what the decaying series can and cannot establish, specify the readout that would settle it, and state what you would ship on Monday.

Approach

1. Name the distinct explanations the same curve is consistent with, because the shipping decision differs across them: a novelty effect decaying to zero; a novelty effect decaying to a small positive plateau the test has no power to see; and a genuine effect whose measured size shrinks as control accounts learn about the change from colleagues or release notes.
2. Fix the time axis before interpreting anything. With rolling entry, calendar week and weeks-since-first-exposure are different variables, and pooling them mixes the decay curve with a change in cohort composition, since the accounts that signed in on day one are systematically the most engaged. Recut on weeks-since-first-exposure and verify that plan_tier and pre-period usage mix are stable across entry cohorts.
3. Read the interval width honestly. Plus or minus 3.1 percentage points at week four cannot separate a durable +1% from zero. The defensible statement is that no durable effect larger than roughly 3.5% was detected, not that there is no durable effect, and those two sentences lead to different decisions.
4. Separate novelty from primacy using account tenure. Accounts created after launch have no prior console to be surprised by, so they cannot show novelty; if the redesign is genuinely better their curve should be flat or rising, while established accounts show the spike and decay. If both cohorts decay to zero, the effect really was novelty.
5. Specify the settling readout rather than arguing about the four weeks you have. Hold back 5% of accounts as a long-run holdout for 90 days, and compare whole ISO weeks only: usage in this domain follows a hard five-to-two weekday cycle, and a window containing four business days instead of five moves this metric by several percent with no product change.
6. Give the Monday answer. Ship if guardrails are clean and maintenance cost is low, because a decayed-to-zero effect with no harm is a neutral trade; but book none of the +9.2% in any forecast, and do not run a follow-up test on the same accounts inside the novelty window, because their baseline has not returned to steady state.

Worked solution (30 min)

1. Recut the four weekly estimates on weeks-since-first-exposure and confirm entry-week cohorts are comparable on plan_tier and pre-period usage.
2. Split each week's estimate by account tenure, created before versus after launch, and compare the shapes of the two curves.
3. Compute the horizon needed to halve the week-four interval: precision scales with the square root of exposure, so about four times the account-weeks are required.
4. Write the ship note with the 5% holdout design, the 90-day re-read date, and an explicit statement of the effect sizes that remain unexcluded.

Expected result: The series is consistent with pure novelty, with a durable plateau anywhere below about 3.5% left unexcluded. Separating the explanations requires the tenure split plus a 5% holdout re-read at 90 days, and halving the week-four interval to plus or minus 1.55 points needs roughly four times the account-weeks.

Verify it yourself

• Interval width scales with one over the square root of the sample, so quadrupling exposure should roughly halve 3.1 points. A plan claiming a larger gain is double-counting repeated weeks on the same accounts, which are correlated rather than independent.
• Accounts created after launch cannot exhibit novelty against an experience they never had, so a week-one spike in that cohort points at an assignment or instrumentation problem rather than at novelty.
• Every weekly comparison must be a whole ISO week. A window containing a different number of business days than its comparator is not a like-for-like contrast in this domain.

Follow-up

• What sample or horizon would you need to rule out a durable +1.5% at 80% power, given the week-four interval you have?
• How would you distinguish a novelty effect from a control arm that gradually learned about the change?
• The metric that decides renewal is up to eleven months away on annual contracts. What proxy do you use in the meantime, and how do you validate it once renewals land?

Activation drops six points starting at a deploy hour

Medium · instrumentation · release analysis · metric inputs

Seven-day activation, defined as an account's first request with http_status < 400, api_key_id not null and traffic_class <> 'synthetic_monitor' within 168 hours of created_at, fell six points for sign-up cohorts after a Tuesday. A client SDK major version shipped that morning. From fct_api_request (account_id, api_key_id, sdk_name, sdk_version, http_status, traffic_class, request_at) and dim_account (account_id, created_at, is_internal), decide whether activation actually fell or the metric's inputs changed, and state in advance what evidence would convince you of each.

Approach

1. Decompose the definition and recompute activation under each relaxation: status only, status plus traffic_class, then the full definition. If the entire drop lives in the api_key_id clause, this is an instrumentation question rather than a behavioural one.
2. Measure the null rate of api_key_id by sdk_version and by hour. A stamping change appears as a step at the deploy boundary confined to the new version; a behavioural change appears as a ramp that grows with adoption and leaves old-version traffic untouched.
3. Hold the cohort's SDK mix fixed before comparing. New sign-ups adopt the newest version first, so a cohort-level drop can be pure composition even when no individual version changed at all.
4. Corroborate with a source the release did not touch: whether the same cohorts appear in fct_usage_daily with non-zero billable_quantity, and whether their fct_support_ticket rows with category = 'onboarding' rose.
5. Write the decision rule down before looking at the answer. An instrumentation artefact predicts unchanged downstream usage and a version-confined null step; a real regression predicts falling downstream usage and more onboarding tickets in the same cohorts.

Follow-up

• Old-version and new-version populations are not exchangeable, because new accounts adopt the new version first. How would you build a comparison that is not confounded by cohort age?
• What backfill or metric-versioning policy keeps the historical series interpretable once you fix the stamping?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

What is a p-value, and how do you interpret statistical significance to non-technical stakeholders?

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Name the disagreement or constraint, and how you resolved it with evidence.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• How did you know the outcome was caused by your change?
• What would you do differently if you ran that project again?

Disagree with a product manager about an adoption claim

Medium · conflict · telemetry hygiene · adoption metrics

A product manager is about to present that a new SDK release drove a 40 percent rise in requests among adopting accounts, computed from fct_api_request counts grouped by sdk_version. You find the rise is concentrated in traffic_class equal to ci, that rows with is_retry true grew alongside it, and that restricting to interactive non-retry traffic leaves a 3 percent lift. The launch review is in two days. Decide how you raise this, with whom and in what order, and what you propose the claim becomes.

Approach

1. The interviewer is probing whether you can correct a colleague without ambushing them, and whether your own counter-analysis carries the caveats theirs lacked. Go to the product manager privately before the review. A correction delivered in the room is a status move and loses the argument you are actually trying to win.
2. Bring a decomposition rather than a verdict: the same accounts and window, requests split by traffic_class with retries held out as their own column, so their 40 percent and your 3 percent reconcile line by line and neither has to be taken on trust.
3. Reproduce their figure exactly first. If you cannot land on 40 percent with their method, you do not yet know what you are disagreeing with.
4. Ask whether the continuous-integration lift is itself valuable. An account wiring the SDK into its pipeline has increased integration depth, which is the dominant switching cost in this domain, so the honest claim may be that integration depth rose while interactive usage moved 3 percent. Improving the claim beats deleting it.
5. Name the mechanism that makes the raw count dangerous: clients retry when the platform degrades, so retry volume climbs exactly when the customer is most at risk. Pull the 5xx rate for the same accounts and window before anyone concludes anything, and note that billable_units is zero on 5xx rows, so request counts and billable quantities diverging is itself the signal.
6. Close with a standing definition for launch metrics so the next release does not repeat the exercise.

Follow-up

• The product manager argues that continuous-integration traffic is real usage and declines to split it out. Is that position defensible, and under what metric definition?
• Suppose the 5xx rate for those same accounts also rose 40 percent. What is the claim now?
• The review happens and the raw number is presented regardless. What do you do next, and what do you not do?

Allocate one analyst week across three competing requests

Easy · prioritisation · stakeholders · scoping

Three requests arrive the same morning and you have one week. Finance wants per-account gross margin from fct_usage_daily for a pricing review in three weeks. Sales wants a renewal-risk list for accounts with term_end_date inside 60 days. A product manager wants an experiment readout for a decision being taken on Thursday. Produce your allocation with hours attached, what you say to whoever receives less, and one thing you refuse to do this week, with the reason each decision is defensible to the person it costs.

Approach

1. The interviewer is probing whether you prioritise on decision timing and reversibility or on who asked most forcefully. Sort by the date each decision is actually taken and by what the default outcome is if nothing arrives.
2. Apply that sort concretely. The Thursday readout has a hard irreversible deadline and no value afterwards. The pricing review has three weeks of slack. The renewal list has a rolling deadline set by term_end_date, so part of it is urgent this week and the rest is not, which means it can be split rather than deferred whole.
3. Find the cheapest sufficient version of each request rather than the full version. The readout goes in full. The renewal list ships as a filtered query over renewal-eligible accounts ranked by two inspectable signals rather than as a model. The margin work is scoped to the accounts that dominate the pricing decision, since revenue is heavily skewed and the tail will not change the conclusion.
4. Make the trade visible in one written note to all three at once, with dates. Telling each person separately that they are the priority is how an allocation becomes a credibility problem.
5. Refuse something explicitly and say why. The model version of the renewal list is the usual candidate, because it cannot be evaluated without a holdout nobody has agreed to yet, and building it this week forecloses that.
6. Leave slack. A plan with none is a plan to miss the one deadline that cannot move.

Follow-up

• The sales leader escalates to your manager. What did you already do that makes that a short conversation?
• Which of the three deadlines would you push back on, and what exactly would you ask for?
• What would you change about how these requests reach you so next week is not the same?