Interview guides · Factored

Factored Data Scientist Interview Questions & Guide 2026

Diagnose a metric move for Factored data scientist prep: split by segment, rule out a logging change, separate mix shift from behaviour.

The 60-second brief

At Factored, a Data Scientist is not just an analyst; you are an elite technical consultant tasked with solving some of the most complex data challenges for top-tier global companies. Founded with the support of Andrew Ng’s AI Fund, Factored builds high-caliber data science, machine learning, and data engineering teams. As a Data Scientist, you will be directly embedded with clients—ranging from fast-growing Silicon Valley startups to massive enterprise organizations—to design, build, and deploy robust analytical solutions and predictive models.

Learn the economics of the product category before the loop. Marketplaces, subscription products and ad-supported products turn on different core quantities (match rate and liquidity, retention and churn, fill rate and yield) and fail in different characteristic ways.

Focus areas

Cluster inference at the account, not the engagement
Correct late-entered timesheets before trending recent weeks
Reconstruct pipeline stage history from mutable rows

Overview

Your work will directly influence product roadmaps, optimize operational efficiencies, and drive strategic decision-making. Whether you are building advanced time-series forecasting models, designing real-time recommendation engines, or structuring complex natural language processing pipelines, you will act as a critical bridge between raw data and business value. The role demands a unique combination of rigorous mathematical foundations, strong software engineering practices, and client-facing communication skills.

Because clients demand excellence, you will work alongside some of the brightest minds in Latin America and globally. You will have the opportunity to continuously upskill, tackle diverse projects across multiple industries, and make a tangible impact on products used by millions of people. This position is highly competitive and requires candidates who are not only technically proficient but also highly adaptable and proactive.

The Factored interview process

Factored candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

HR Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Online Assessments

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Live Technical Evaluations

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

System Design Discussion

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Candidate Pool Placement

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Client Matching

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Using calendar hours, or a current rate card, as the denominator

Available hours is not days x 8. It must subtract approved leave and region-specific public holidays, scale by fte_fraction, and prorate for hire_date and termination_date inside the period; skipping this produces cross-region utilisation gaps that are purely holiday-calendar artifacts and a fake dip every time a cohort of new hires starts. Equally, bill and cost rates change over time, so joining today's dim_consultant row to last year's hours restates history. Use the rate snapshotted on the time entry, or join dim_consultant on work_date BETWEEN effective_from AND COALESCE(effective_to, 'infinity').

Treating accounts as independent observations

Revenue is concentrated: a small number of client_ids typically carries a large share of fees, and engagements within one account share a partner, a rate card and a delivery team. Ordinary standard errors computed over engagements therefore understate uncertainty badly. Cluster at client_id, and with fewer than roughly 40 clusters use a wild cluster bootstrap or a CR2 correction, because cluster-robust standard errors are downward-biased in that regime and will manufacture significance that a replication will not reproduce.

Ignoring interference between units in a marketplace experiment

Ask whether one unit's treatment can change another unit's outcome through shared inventory, a matching pool, a social graph or a common budget. Where it can, randomise at a level that contains the spillover, such as region or time slice, and say explicitly what that costs you in statistical power.

Solving silently instead of narrating the reasoning

Say which branch you are taking and why you chose it over the alternative, for example checking the denominator first because it changes what the comparison means. A correct answer that arrives with no visible path scores below a rigorous one that needed a hint.

Questions & practice

Explain the difference between generative and discriminative models.

Medium · statistics and probability

Approach

Translate the result into the decision it informs, in one plain sentence.
Quantify uncertainty explicitly rather than reporting a point estimate alone.
Sanity-check the answer against a simple bound or a simulated case.

Follow-up

How would you explain this result to someone who does not know statistics?
Which assumption here is most likely to be violated in practice?

What is the difference between covariance and correlation?

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Say what the estimate is of, and over what population it generalises.
Write down the assumption the method needs before you use the method.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

How do you explain complex machine learning model decisions to non-technical business stakeholders?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Kaplan-Meier days to payment with unpaid invoices censored

Hard · survival analysis · censoring · kaplan-meier

invoice_lines has invoice_line_id, engagement_id, line_type, issued_at, due_date, paid_at (null when unpaid), amount_usd and status in draft, issued, partially_paid, paid, disputed, written_off. At a given snapshot_date, estimate the median days from issue to full payment. Implement Kaplan-Meier yourself; no lifelines or equivalent. Treat issued, partially_paid and disputed as right-censored at snapshot_date, and decide and justify what to do with written_off. Report the naive mean over paid lines alongside your estimate and state the sign of its bias.

Approach

Build the duration and event table explicitly. Drop draft lines, which have no clock. For status paid, duration = (paid_at - issued_at).days with event = 1. For issued, partially_paid and disputed, duration = (snapshot_date - issued_at).days with event = 0.
Handle written_off as a competing event rather than a censor. Censoring it makes the estimator answer 'time to payment if written-off invoices could still pay', which overstates collection. Either report a cumulative-incidence version alongside, or censor them and say plainly that the result is conditional on eventual collection.
Implement the estimator directly: sort unique event times, at each t take n_i as the count with duration >= t and d_i as the payments at exactly t, and accumulate S(t) = product of (1 - d_i / n_i). Censored rows leave the risk set without causing a drop, which is the whole mechanism and the reason the answer differs from any completed-case average.
Read the median as min{t : S(t) <= 0.5}. If S never reaches 0.5 within observed follow-up, report 'not reached'; interpolating past the last observation invents data that the snapshot does not contain.
Add Greenwood's formula for Var(S(t)) to put a band on the curve, then invert the band at 0.5 for an interval on the median rather than quoting a point estimate alone.
Compare against the mean over paid lines only and name the direction: at any snapshot the paid set over-represents fast payers, so the naive mean is biased low, and the bias widens exactly when collections deteriorate.

Worked solution (40 min)

Assemble a frame of (duration_days, event) per non-draft line, with snapshot-based durations for the three open statuses and an explicit, documented rule for written_off.
times = np.sort(df.loc[df.event == 1, 'duration_days'].unique()); for each t, n = (df.duration_days >= t).sum() and d = ((df.duration_days == t) & df.event).sum().
S = np.cumprod(1 - d / n) aligned to times; prepend S(0) = 1.0 so the curve starts where it should.
median = times[np.argmax(S <= 0.5)] if (S <= 0.5).any() else 'not reached'; compute Greenwood variance as S(t)^2 * cumsum(d / (n * (n - d))).
naive = df.loc[df.event == 1, 'duration_days'].mean(); report both, plus the censored share, since the gap between the two scales with it.

Expected result: A step function starting at S(0) = 1.0 that drops only at observed payment days, an integer median or an explicit 'not reached', a Greenwood-based interval, and a naive completed-case mean that sits below the Kaplan-Meier median on data with material censoring.

Verify it yourself

S is non-increasing, starts at 1.0, and never goes below 0; the number at risk at the first event time equals the count of all non-draft lines.
Re-running with every censored row deleted makes the Kaplan-Meier median equal the empirical median of paid durations, since with no censoring the estimator reduces to the empirical survival function.
A six-row toy example worked by hand reproduces the implementation's S values exactly.
The gap between the naive mean and the Kaplan-Meier median grows when you move snapshot_date earlier, which raises the censored share.

Follow-up

A large account moved to a monthly payment run. Is administrative censoring still independent of payment time, and what would you check?
Finance wants one DSO number against a target. What do you give them, and what do you refuse to give them?
Stratify by line_type. Do milestone lines behave like fees lines, and what would it mean for the firm if they do not?

How would you optimize a slow-running SQL query that involves multiple large table joins?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Write a script to perform an ETL process that extracts raw JSON data, flattens it, handles missing values, and loads it into a structured DataFrame.

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Given a dataset of user transactions, write a Pandas script to calculate the rolling 7-day average of purchase amounts.

Medium · sql and data manipulation

Approach

State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Cost hours at the rate in force on the work date

Medium · type-2 scd · effective dating · date boundaries

dim_consultant is a type-2 dimension with consultant_sk, consultant_id, level, practice_area, standard_cost_rate_usd, effective_from TIMESTAMPTZ, effective_to TIMESTAMPTZ NULL and is_current. fct_time_entry holds consultant_id, work_date DATE, hours, status and a cost_rate_usd snapshotted when the row was written. Recompute last year's delivery cost per practice_area by joining each approved entry to the dimension row valid on its work_date, then reconcile against the snapshotted rate and list entries where the two differ by more than one cent. Say what a session timezone change does to your join predicate.

Approach

Join on a half-open interval: te.work_date >= d.effective_from AND (d.effective_to IS NULL OR te.work_date < d.effective_to). This assumes effective_to holds the next row's effective_from. Verify that on one consultant with three versions before committing, because if effective_to instead stores the last valid instant the upper bound must be inclusive, and getting it backwards either double counts or drops a boundary day.
Refuse is_current = TRUE as a join predicate. It prices last year's hours at today's rate, so a promotion or an annual rate change silently restates history, and the restatement is invisible in the output.
Be explicit about the DATE to TIMESTAMPTZ comparison. Postgres casts work_date to midnight in the session TimeZone while effective_from is an absolute instant, so a version that opens at 2026-03-01 00:00 UTC lands on one side of a 2026-03-01 work_date under a UTC session and the other side under a negative-offset session. Normalise both to one convention, for example comparing work_date against (effective_from AT TIME ZONE 'UTC')::date, and write down which convention the warehouse uses.
Validate one-to-one before trusting any sum: group the joined result by time_entry_id and assert no id appears twice. Overlapping validity intervals in the dimension are a real defect and present as inflated cost, not as an error message.
Reconcile the recomputed rate against the snapshotted cost_rate_usd. Differences are either late-corrected dimension rows or entries snapshotted at the wrong moment; both are findings with a named cause, not rounding noise to be smoothed over.

Worked solution (30 min)

Count approved time entries in the window; this is the exact row count the join must preserve.
Write the half-open interval join and immediately compare the post-join row count against that number.
Check for duplicates by grouping on time_entry_id and, if any appear, inspect that consultant's dimension rows for overlap.
Aggregate cost by practice_area using the dimension rate, then again using the snapshotted rate, and diff the two totals.
List the individual entries whose two rates differ by more than a cent, with both rates and the dimension row's validity interval.

Expected result: A cost total per practice_area plus an exception list. The joined row count equals the approved-entry count exactly, and the two cost totals agree except for the entries on the exception list.

Verify it yourself

COUNT() after the join equals COUNT() before it.
No time_entry_id appears more than once in the joined set.
Recomputed total cost is within the exception list's aggregate difference of the snapshot-based total.
Re-running after SET TIME ZONE to a non-UTC zone assigns every entry to the same dimension row, or the query documents exactly where it does not.

Follow-up

How would you detect overlapping or gapped validity intervals across the whole dimension in one query?
Which of the two rates belongs in a client-facing margin number, and which in an internal cost review?
A consultant changes practice_area mid-engagement. Which practice_area owns the hours, and what breaks either way?

Describe a situation where you had to make a difficult technical trade-off to meet a tight project deadline.

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Explain the Central Limit Theorem and its practical implications in A/B testing.

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

What would you do if you could not randomise at all?
How would you handle interference between treated and control units?

How would you design a scalable data pipeline to ingest, process, and analyze real-time streaming data from IoT devices?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer would contain.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Measure a staggered process rollout across five practice areas

Hard · difference-in-differences · staggered adoption · wild cluster bootstrap · parallel trends

A delivery-review process was rolled out one practice area at a time over six quarters, and all five practice areas now have it, so there is no never-treated group. You have engagement-quarter gross margin reconstructible from fct_invoice_line and fct_time_entry, plus pricing_model on fct_engagement. Leadership wants the effect on margin. Explain why a two-way fixed effects regression of margin on a treated indicator is not trustworthy here, name the estimator you would use instead and what identifies it, and specify how you would obtain a defensible p-value from five clusters.

Approach

Explain the two-way fixed effects failure concretely. Under staggered timing the coefficient is a weighted average of every available two-by-two comparison, and that set includes comparisons that use already-treated units as controls for later-treated ones. Those comparisons difference out not only the common time shock but also the earlier cohort's ongoing treatment-effect growth, so when effects grow after adoption the aggregate is pulled toward zero and can flip sign. Decomposed onto the underlying ATT(g,t), the implied weights can be negative, which is why the coefficient is not an average of treatment effects at all. This is a bias in the estimand, not a small-sample artefact, so more data does not fix it.
Switch to a heterogeneity-robust estimator: Callaway and Sant'Anna group-time ATT(g,t) using not-yet-treated units as the comparison group, aggregated into an event-study path and a single summary. Sun and Abraham's interaction-weighted estimator or Borusyak-Jaravel-Spiess imputation answer the same objection. With every area eventually treated, identification rests entirely on not-yet-treated windows, so the last-adopting area carries disproportionate weight; say that out loud rather than burying it.
Test parallel trends honestly. Plot event-time leads with confidence intervals and refuse to read a non-rejection as confirmation, because at five clusters that test is badly underpowered. Pair it with a sensitivity analysis that bounds the post-treatment violation as a multiple of the largest pre-period violation and reports the multiple at which the conclusion breaks.
Fix the inference, and be precise about what the bootstrap can deliver at five clusters. Five sits far below the roughly forty at which cluster-robust standard errors become approximately valid, and below that they are sharply downward-biased. A wild cluster bootstrap with Rademacher weights draws from 2^5 = 32 sign vectors, but the null-imposed bootstrap t-statistic is odd in the sign vector: negating every weight negates t* and leaves |t*| unchanged, because the coefficient deviation is linear in the weights while the cluster-robust variance is quadratic in them. The 32 vectors therefore collapse to 2^4 = 16 distinct values of |t*|, so the smallest attainable two-sided p-value is 1/16 = 0.0625. That is above 0.05, so at five clusters a Rademacher bootstrap cannot reject at the conventional level however large the true effect is. Use Webb six-point weights, whose 6^5 = 7,776 vectors collapse to 3,888 distinct |t*| and a floor near 0.00026, or randomisation inference permuting the five observed adoption dates across the five practice areas, which gives 5! = 120 assignments and a floor of 1/120 = 0.0083.
Deal with the mix confound before reporting anything. Fixed-fee margin falls with hours worked while uncapped time-and-materials margin rises with them, so a rollout that coincided with a pricing-mix shift contaminates the pooled estimate. Estimate within pricing_model and report the within-model effects alongside the mix decomposition.

Worked solution (45 min)

Build the engagement-quarter panel with an adoption quarter per practice area, keyed on the quarter the process went live rather than when it was announced.
Run the Goodman-Bacon decomposition of the two-way fixed effects estimate and read off how much of the total weight sits on already-treated-as-control comparisons; that share is the diagnostic there, since those weights are non-negative by construction. For negative weights on the underlying ATT(g,t), run the de Chaisemartin and D'Haultfoeuille decomposition and report the negative share.
Estimate ATT(g,t) with not-yet-treated controls, aggregate to event time, and plot leads and lags with confidence bands.
Compute the p-value three ways, cluster-robust, Rademacher wild bootstrap and Webb wild bootstrap, and report the Webb figure while noting that the Rademacher floor at five clusters is 1/16 = 0.0625 and therefore cannot clear 0.05 at all.
Re-estimate separately for fixed_fee and time_and_materials engagements and report the mix decomposition against the pooled change.

Expected result: An event-study path whose pre-period leads are statistically indistinguishable from zero, a post-period aggregate ATT reported with a Webb wild-bootstrap p-value, an explicit note that a Rademacher bootstrap at five clusters cannot produce a p below 1/16 = 0.0625 and so cannot reject at 0.05 on any data, and separate estimates by pricing model with the mix shift quantified.

Verify it yourself

Goodman-Bacon weights are non-negative and sum to one by construction, so a negative one there is an implementation bug rather than a finding; the number to read is the share of weight resting on already-treated-as-control comparisons. Negative weights on the underlying ATT(g,t) come from the de Chaisemartin and D'Haultfoeuille decomposition, and a large negative share means the two-way fixed effects coefficient should not be reported as an average effect.
Restricted to one adoption cohort and two periods against not-yet-treated units, the group-time estimator and two-way fixed effects must return the identical two-by-two difference in differences; a gap there is an implementation bug, not heterogeneity.
At five clusters, enumerate all 32 Rademacher sign vectors rather than drawing B of them. The resulting |t*| must come back as 16 identical pairs, one per sign-flip orbit; 32 distinct values means the null is not being imposed or the statistic being bootstrapped is not the cluster-robust t.
Wild bootstrap p-values should be weakly larger than cluster-robust ones at five clusters; if they come out smaller, check that the null is being imposed in the bootstrap.
Within-pricing-model effects plus the mix-shift term must reconcile to the pooled period-over-period change.

Follow-up

If the rollout order was not random, say the worst-performing practice area went first, what does that do to parallel trends and what would you do about it?
How would you choose between a synthetic control built on a single practice area and the group-time estimator?
What would the event-study path have to look like for you to believe the effect is real rather than a continuation of a pre-existing trend?

Realisation rate steps up overnight after a warehouse release

Medium · definition change · realisation · restatement

Realisation rate for one practice area steps from 0.93 to 1.02 on a single reporting date. Bill rates, hours worked and the client set are unchanged. The metric is SUM(amount_usd) from fct_invoice_line over SUM(hours * bill_rate_usd) from approved billable fct_time_entry rows for the same engagement and period. A warehouse release shipped that week. Determine what in the definition can produce a level shift with no operational change, and deliver a back-restated series plus the definition you would keep.

Approach

Treat the shape as evidence: a definition change is a discontinuity at one processing boundary, while a behaviour change ramps over weeks. Plot daily, not monthly, to see which one this is before forming any hypothesis.
Diff the metric's lineage across the release and enumerate the candidate predicates: which line_type values enter the numerator (dropping credit_note and write_off inflates it), whether the period join moved from period_start_date to issued_at, and whether the denominator counts status = 'approved' only or now includes 'submitted'.
Run both definitions over the same fixed pre-release month. If old and new differ by approximately the size of the step, the step is definitional; if they agree, look for a behaviour change coinciding with the release date.
Sanity-check the new level against the metric's physics: values above 1.0 are legitimate only on fixed_fee work delivered under its hour budget, so a practice dominated by time_and_materials sitting at 1.02 is itself a red flag.
Restate the full history under the chosen definition, publish old and new series together with the changeover marked, and add a reconciliation test that recomputes one known engagement-month from raw rows.

Follow-up

Which definition is correct, and what does excluding write-offs from the numerator do to the guardrail relationship with utilisation?
How do you prevent the next release from silently restating a headline metric?
If the release also changed the denominator's status filter, which direction does that push realisation and by roughly how much?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

For a Data Scientist (Time Series) role, how do you handle seasonality and stationarity in sequential data?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Quantify the outcome, including what you would not claim credit for.
Name the disagreement or constraint, and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Scoping a one-line question about practice profitability in twenty minutes

Medium · scoping · metric definition · exclusions

A practice leader asks, in one line, whether the data_and_ai engagements are profitable. You have fct_engagement, fct_time_entry and fct_invoice_line, and twenty minutes before they travel. Some engagements are retainer or outcome_based, so contracted_hours is NULL; some invoices are unpaid; pursuit hours sit on charge_code = 'client_pursuit' with engagement_id NULL. Produce the scoping conversation: the questions you ask, the metric you commit to in writing, the exclusions you will have to make, and what you deliver by when.

Approach

Name the probe: whether you convert an ambiguous one-liner into a decision with a metric attached, or start querying and return a number nobody can act on.
Ask what changes based on the answer. Repricing the rate card, changing the staffing pyramid, and deciding whether to keep selling fixed_fee are three different questions, and they need different cuts of the same data. Pick the one the leader actually has authority over.
Commit the metric in writing on the spot: fees plus milestone plus credit_note lines from fct_invoice_line as the revenue side, the sum of hours times cost_rate_usd over all approved entries, billable and non-billable alike, as the cost side, inception to date, reported separately by pricing_model.
List what you will not answer and why: overrun on retainer and outcome_based work, because contracted_hours is NULL there and needs an explicit predicate rather than a silent NULL drop; cash collected, because unpaid invoices are censored; and pursuit cost, which has no engagement_id and must either be allocated on a stated rule or reported as a separate line.
Commit two deliverables with dates rather than one vague one: the stratified margin table within two days, the mix decomposition within a week. Name the single assumption that, if wrong, flips the conclusion.

Follow-up

They insist on one number for a partner meeting. Which single number do you give, and what sentence goes with it?
If you had to allocate client_pursuit hours to engagements, what rule would you use and how would you show the answer is not sensitive to it?

Defending a fixed-fee overrun finding against the selling partner

Easy · metric definition · pricing model mix · stakeholder pushback

Over the trailing four quarters, fixed_fee engagements in one practice area show a median scope overrun ratio of 1.34, computed as approved client_delivery hours in fct_time_entry divided by fct_engagement.contracted_hours, and inception-to-date gross margin nine points below the time_and_materials book. The partner who sold most of that work says the denominator is wrong because change orders were signed, and each change order is a separate row linked by prior_engagement_id. You have ten minutes with the practice lead. Present and defend the finding.

Approach

Say out loud what is being probed: whether a specific methodological objection makes you recompute or makes you repeat yourself. Restate the metric as numerator, denominator and window before defending anything.
Test the objection empirically instead of debating it. Rebuild the denominator by walking the prior_engagement_id chain with a recursive CTE, summing contracted_hours across the original SOW and every extension, and summing approved delivery hours over the same chain.
Report both numbers and say what each one answers: the original-SOW denominator answers whether the team held the scope that was signed, the chained denominator answers whether the firm estimated the total work correctly. Neither is a trick; they are different questions.
Keep the margin claim separate from the overrun claim, and keep it stratified. Fixed_fee margin falls with hours worked while uncapped time_and_materials margin does not, so the nine-point gap is only meaningful within pricing_model, and a shift in the fixed_fee share of fees is reported alongside it.
Close with the decision and a falsifier: if the chained overrun is near 1.0, the problem is scope control on the original SOW, and the check is whether change orders repriced at the standard rate card or at the original blended rate.

Follow-up

The chained denominator narrows the overrun to 1.11. Does your recommendation change, and which number goes in the practice review?
How would you distinguish an estimation problem at sale time from a delivery problem during execution, using only these tables?