Interview guides · Next Tier Concepts

Next Tier Concepts Data Scientist Interview Questions & Guide 2026

Data scientist prep for Next Tier Concepts: practice defining a B2B software and infrastructure metric, picking its guardrails and defending the tradeoff.

The 60-second brief

A Data Scientist at Next Tier Concepts plays a pivotal role in transforming complex, multi-source datasets into actionable intelligence. Operating at the intersection of advanced analytics, software engineering, and mission-critical domain knowledge, you will design and deploy analytical models that directly support national security, federal civilian missions, and enterprise decision-making. The solutions you build do not just sit in notebooks; they are integrated into production systems that process massive volumes of structured and unstructured data in real time.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Strip CI, retry and synthetic traffic first
Measure churn only on renewal-eligible accounts
Separate contracted seats from actively used seats

Overview

The impact of this position is profound. By developing predictive models, natural language processing (NLP) pipelines, and anomaly detection systems, you help federal partners navigate highly ambiguous environments and identify critical patterns that would otherwise remain hidden. Your work directly influences system capabilities, operational efficiency, and strategic outcomes for high-stakes programs located in defense and intelligence hubs like Chantilly and Herndon, Virginia.

What makes this role exceptionally rewarding is the scale and diversity of the problem spaces. You will collaborate closely with data engineers, software developers, and mission stakeholders to build end-to-end pipelines. Whether you are optimizing search algorithms, fine-tuning large language models, or establishing statistical baselines for complex workflows, your contributions will drive the technical direction of ' most critical delivery teams.

The Next Tier Concepts interview process

Next Tier Concepts candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Contact

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What to demonstrate

Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Technical Deep Dive

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Behavioral Assessment

Rounds of this kind usually include one question about work that did not go well, and it is the part that carries the most information. Anyone can narrate a shipped win. What the interviewer learns from a project that stalled is how you behave without a result to hide behind: whether you noticed the problem yourself, how long it took, and who you told. Answers that route the failure onto a data pipeline or a reorganisation close the topic without answering it, and the follow-up comes back to your own part.

What to demonstrate

Whether you found the error yourself or someone else found it, and how long it sat before anyone knew
What you changed afterwards, stated as a check you now run rather than a lesson you now believe
Whether the mistake you choose has real cost attached, such as a quarter of misdirected roadmap or a metric that was reported upward, instead of one that flatters you

How to prepare

Choose a failure you caught yourself and be ready to say what tipped you off. A story where someone else caught it is still usable, but you will be asked why you missed it.
Write down the check you added afterwards and where it lives now, so the correction is a concrete artefact rather than a resolution.
Rehearse saying the cost out loud. Candidates shrink the number by instinct once the interviewer is in the room.

Team Integration

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Final Offer Stage

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Treating raw request or usage volume as engagement

Most traffic in this domain is emitted by machines. Continuous-integration pipelines, scheduled batch jobs, synthetic monitors, backfills and client retries can all grow by an order of magnitude from one configuration change made by one engineer, and none of it represents a new decision to use the product. The inversion is what makes it dangerous: when the platform degrades, clients retry, so error-driven retry volume rises at the exact moment the customer is most likely to leave, and an engagement dashboard built on raw counts shows growth immediately before a churn. Filter on traffic_class and on successful status before anything else, and keep failed-request volume as its own separate series.

Reporting a mean over accounts when account revenue is heavy-tailed

When a small number of accounts hold most of the revenue, the sample mean is dominated by whichever of them happens to be in the sample, and the sample variance keeps growing as more data arrives instead of stabilising. In that regime the usual central-limit-based confidence interval understates uncertainty, and a single renewal or a single large account's batch job can flip the sign of a measured effect. The fixes are to pre-register a winsorisation or capping rule before looking at the outcome, to report account counts crossing a threshold alongside the revenue figure, or to define the estimand on a bounded transform. Choosing the cap after seeing the result is a separate and worse problem, because the cap then encodes the answer.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Ignoring interference between units in a marketplace experiment

Ask whether one unit's treatment can change another unit's outcome through shared inventory, a matching pool, a social graph or a common budget. Where it can, randomise at a level that contains the spillover, such as region or time slice, and say explicitly what that costs you in statistical power.

Questions & practice

Explain the bias-variance tradeoff and how you would address high variance in a predictive model.

Medium · statistics and probability

Approach

Say what the estimate is of, and over what population it generalises.
Write down the assumption the method needs before you use the method.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

What sample size would you need to detect an effect half this size?
Which assumption here is most likely to be violated in practice?

What metrics would you use to evaluate the performance of an NLP model designed for entity extraction?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.

Follow-up

How would you choose the decision threshold, and who owns that choice?
Where could label leakage enter this setup?

Walk us through how you would define and measure success metrics for a newly deployed search recommendation algorithm.

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Set a baseline first, so any model has something honest to beat.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Permutation-test a consumption experiment randomised at account level

Hard · permutation-test · heavy-tails · cuped · experiment-readout

An experiment randomised 900 accounts into two arms. You have one row per account: account_id, arm, consumption_28d (billable units after launch) and consumption_pre (the 28 days before). Consumption is heavy-tailed and the largest account is several percent of the total. Write a permutation test from scratch: winsorise at the pooled 99th percentile as a pre-registered rule, use the difference in arm means of the winsorised outcome as the statistic, and obtain a two-sided p-value from 20,000 relabellings of the account-level arm vector. Report the observed effect, the p-value, and the same test on a CUPED-adjusted outcome.

Approach

Be precise about what the permutation test needs. Under the sharp null of no effect for any account, the outcomes are exchangeable across arm labels, and the test is valid for ANY statistic T(outcomes, labels) provided the identical function is applied to the observed labels and to all 20,000 relabellings. The pooled 99th percentile is a function of the outcome vector alone, so recomputing it inside the loop returns the same number 20,000 times: that is wasted CPU, not a bias, and hoisting it out is an optimisation rather than a correctness fix. Say plainly that capping at all changes the estimand from mean consumption to mean capped consumption; it is not a neutral cleaning step.
The mistake that does invalidate the test is an asymmetry between the observed statistic and the permuted ones, and the easiest way to create it is to derive the cleaning rule from the observed arm labels and then freeze it — winsorise each arm at its own observed 99th percentile, hold those two caps fixed, and permute. The observed value is then computed with caps matched to its own partition while every relabelling is scored with caps belonging to a different one, so the null distribution no longer answers the question the p-value claims to answer. A per-arm cap recomputed consistently inside every permutation is a valid test, but it estimates a contrast whose two sides are capped at different thresholds, so prefer the pooled cap on estimand grounds and pre-register it.
Permute the account-level arm vector, because the account is the randomisation unit. Relabelling anything finer — users, workspaces, requests — generates a null distribution narrower than the design actually supports and returns p-values that are anti-conservative.
Vectorise the null: tile the treatment indicator into a (B, n) matrix and permute along axis 1 with rng.permuted(..., out=...). The statistic is a difference of means, so the treated sum alone determines it and the whole null is one matrix-vector product. Use the two-sided p-value (1 + count(|stat_perm| >= |stat_obs|)) / (B + 1); the plus-one on each side is not cosmetic, it keeps the p-value away from exactly zero and keeps the test valid at finite B.
For CUPED, fit theta = cov(y, x) / var(x) on the pooled data and use that same theta for the observed statistic and every relabelling. Pooled theta, like the pooled cap, carries no label information, so where in the loop you compute it is again only a performance question; fitting theta within arms is what goes wrong, because the adjusted outcome then depends on the labels and an observed-label fit frozen across all 20,000 relabellings breaks the match between observed and permuted statistics. x must be measured entirely before launch, which consumption_pre is. Expected variance reduction is about 1 - corr(y, x)^2; measure the achieved reduction from the two null distributions rather than asserting it.

Worked solution (45 min)

cap_y = np.quantile(df.consumption_28d, 0.99); y = np.minimum(df.consumption_28d.to_numpy(float), cap_y); cap_x = np.quantile(df.consumption_pre, 0.99); x = np.minimum(df.consumption_pre.to_numpy(float), cap_x)
t = (df.arm == 'treatment').to_numpy(); n1 = int(t.sum()); n0 = len(t) - n1; obs = y[t].mean() - y[~t].mean()
rng = np.random.default_rng(11); L = np.tile(t.astype(np.int8), (20_000, 1)); rng.permuted(L, axis=1, out=L); s1 = L @ y; stats = s1/n1 - (y.sum() - s1)/n0
p = (1 + int(np.sum(np.abs(stats) >= abs(obs)))) / (20_000 + 1)
theta = np.cov(y, x, ddof=1)[0,1] / np.var(x, ddof=1); y_adj = y - theta*(x - x.mean()); repeat steps 2 to 4 on y_adj and compare stats.std(ddof=1) between the two runs.

Expected result: A two-sided p-value strictly between 1/20001 and 1, an observed effect expressed in capped billable units per account, and a CUPED null whose standard deviation is smaller than the unadjusted one by roughly sqrt(1 - corr(y, x)^2). The CUPED point estimate stays close to the unadjusted one, since the adjustment removes variance rather than shifting the effect.

Verify it yourself

The permutation null is centred near zero: |mean(stats)| is a small fraction of std(stats).
An A/A run, relabelling arm at random before the test and repeating over seeds, gives roughly uniform p-values with about 5% below 0.05.
std(stats_cuped) < std(stats) whenever corr(y, x) is materially non-zero, and the ratio matches sqrt(1 - r^2) to within simulation noise.
Moving the pooled cap computation inside the permutation loop reproduces the same p-value bit for bit at the same seed — the direct demonstration that a pooled cap carries no label information — while setting the cap to infinity reproduces the uncapped test exactly, so the winsorisation switch is independently verifiable.

Follow-up

The p-value is 0.04 with the cap and 0.31 without it. What do you report, and what did you pre-register?
Colleagues in a shared workspace can see the treated behaviour. How does that change the design and the estimate?
How many accounts would you need to detect a 5% lift given this outcome's distribution?

How would you optimize a slow-performing SQL query that aggregates millions of rows?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Write a SQL query using window functions to identify the top three most frequent events per user session.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How would you verify this result without re-running the same query?
How does the query change if the join becomes one-to-many?

Explain the difference between an inner join, outer join, and a cross join, and describe the performance implications of each on large datasets.

Medium · sql and data manipulation

Approach

Say which table is the grain you start from, and join outward from it.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Running commitment burn-down and the date consumption crosses it

Medium · window functions · running totals · committed consumption

fct_subscription_period gives committed_amount_cents, term_start_date and term_end_date for each account's current version where pricing_model = 'committed_consumption'. fct_usage_daily gives account_id, workspace_id, sku_code, usage_date and net_amount_cents, at one row per workspace and SKU per day. Inside each account's current term, return the running total of net_amount_cents by usage_date, the first usage_date on which that running total reaches committed_amount_cents, and the fraction of the term elapsed at that point. Accounts that have not reached their commitment must still appear, with a null crossing date.

Approach

Collapse usage to one row per (account_id, usage_date) first. The fact is grained by workspace and SKU, so a raw running total leaves several rows per date and the first-crossing date becomes dependent on the arbitrary order of rows inside that day.
Restrict to the term with usage_date BETWEEN term_start_date AND term_end_date on the account's current row, so no prior term's consumption leaks into this term's burn-down.
Compute SUM(net_cents) OVER (PARTITION BY account_id ORDER BY usage_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). Name ROWS explicitly: the default frame is RANGE, which includes all peer rows at the same ORDER BY value and would hand back the whole day's total on each of that day's rows.
Extract the crossing with MIN(usage_date) FILTER (WHERE running_cents >= committed_amount_cents) grouped per account, which returns NULL for accounts still under commitment instead of dropping them.
Elapsed fraction is (crossing_date - term_start_date)::numeric / NULLIF(term_end_date - term_start_date, 0); Postgres date subtraction yields whole days, so the guard matters for same-day terms.
Exclude the trailing days still inside the metering settling window, measured from first_written_at against restated_at, and say how many days you cut and why.

Worked solution (30 min)

Build terms: SELECT account_id, committed_amount_cents, term_start_date, term_end_date FROM fct_subscription_period WHERE is_current AND pricing_model = 'committed_consumption' AND committed_amount_cents IS NOT NULL.
Build daily: join fct_usage_daily to terms on account_id with usage_date inside the term, then GROUP BY account_id, usage_date summing net_amount_cents.
Add running_cents with SUM(...) OVER (PARTITION BY account_id ORDER BY usage_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW).
Aggregate per account with MIN(usage_date) FILTER (WHERE running_cents >= committed_amount_cents) AS crossing_date, then compute the elapsed fraction with the NULLIF guard.
LEFT JOIN that summary back onto terms so every committed account appears, and verify the last running_cents per account equals a plain SUM over the term.

Expected result: A per-account-per-day series of running_cents inside the current term, plus one summary row per committed account carrying crossing_date and term_elapsed_fraction, with crossing_date NULL exactly for accounts whose final running total is below committed_amount_cents.

Verify it yourself

The maximum running_cents per account equals SUM(net_amount_cents) over that account's term computed independently.
crossing_date is NULL if and only if the final running total is below the commitment.
term_elapsed_fraction falls in [0, 1] for every non-null crossing date.
running_cents is non-decreasing within each account, which fails immediately if a negative credit row was included without comment.

Follow-up

Turn this into an end-of-term overage forecast. What breaks if you extrapolate a linear run rate on a consumption product?
An account crosses its commitment at 40 percent of the term. Is that an expansion signal or a billing-surprise risk, and what would you check to tell them apart?

We have a massive collection of unstructured text documents. How would you design a system to automatically categorize these documents and extract key topics?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you prioritize your tasks when juggling multiple tight deadlines on a high-priority project?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Measure named coverage at an assignment threshold you cannot randomise

Hard · regression-discontinuity · instrumental-variables · selection-bias

Named customer-success coverage is assigned to accounts whose arr_cents at fiscal-year start is at least $50,000. About 88% of accounts above the line receive a named owner and about 9% below it do, through documented exceptions. Roughly 200 accounts sit within $15,000 of the cutoff, and gross logo retention on the renewal-eligible base runs near 0.88. Estimate the effect of coverage on renewal, name the design, state its identifying assumptions and the test for each, and say whether this data can answer the question at all.

Approach

Say first why the naive comparison is unusable, because the answer turns on it. Coverage is assigned to the largest accounts because they are valuable and to distressed accounts because they are at risk, so selection runs in both directions at once and the naive estimate can come out with either sign depending on which rule dominated. Matching on observed size does not fix it: the risk signal that triggered coverage is the same signal that predicts renewal.
Name the design and the estimand together. This is a fuzzy regression discontinuity with the $50,000 threshold as an instrument for coverage. The estimator is the Wald ratio, the jump in renewal at the cutoff divided by the jump in coverage probability of 0.88 - 0.09 = 0.79, and the estimand is a local average treatment effect for accounts near $50,000 whose coverage is actually decided by the rule, not the average effect of coverage across the book.
Fit it as local linear regression on each side with a triangular kernel and a data-driven bandwidth (Imbens-Kalyanaraman, or Calonico-Cattaneo-Titiunik with bias-corrected robust intervals). A global polynomial fitted across the whole ARR range is not a substitute and is well documented to manufacture spurious jumps at interior points.
Test the assumptions instead of asserting them. Run a density test for bunching just above $50,000, which is exactly what a sales team structuring a deal at $50,001 to win coverage would produce and which invalidates the design outright. Check continuity at the cutoff of employee_band, acquisition_channel, deployment_model and pre-period usage. Run placebo cutoffs at other ARR values, and plot the estimate against bandwidth.
Do the power arithmetic before promising an answer. With about 100 accounts per side and a renewal base rate of 0.88 (standard deviation 0.325), the sharp-design MDE at 80% power and two-sided 0.05 is 2.80 * 0.325 * sqrt(2/100) = about 12.9 percentage points, and dividing by the 0.79 first stage gives about 16 points on the fuzzy estimand. No plausible effect of assigning an account manager is that large.
Give the honest verdict with a route forward attached. This design cannot answer the question on one fiscal year. Pool several years at the same cutoff, accepting the added assumption that the coverage model and the threshold's meaning were stable across them, or ask for a randomised holdout among accounts near the threshold, which is cheap to justify precisely because the rule is already arbitrary there.

Worked solution (45 min)

Plot the density of arr_cents in bins around $50,000 and run the manipulation test; stop and report the design as invalid if bunching is present.
Plot the first stage, share of accounts with a named owner against ARR, and confirm the jump is close to 0.79 and locally flat on each side.
Fit local linear regressions of renewal on ARR either side of the cutoff with a triangular kernel and data-driven bandwidth, take the Wald ratio, and report bias-corrected robust intervals.
Run covariate-continuity checks, placebo cutoffs, and a bandwidth-sensitivity curve for the point estimate.
Compute the MDE at the realised effective sample and write the verdict, powered or not, with the alternative design named.

Expected result: A fuzzy RD local average treatment effect at the $50,000 cutoff with a first stage of about 0.79, and an MDE of roughly 16 percentage points on the fuzzy estimand at about 100 accounts per side. The honest finding is that the design is underpowered by an order of magnitude relative to any credible coverage effect, so the recommendation is pooled fiscal years or a randomised holdout at the margin.

Verify it yourself

The first-stage jump and the outcome jump must be estimated on the same bandwidth. A Wald ratio assembled from two different bandwidths is not the regression discontinuity estimator.
Covariates that coverage could not have affected must be continuous at the cutoff. A jump in employee_band or acquisition_channel at $50,000 indicates either manipulation of the running variable or a second rule sharing the same threshold.
Point estimates should vary smoothly with bandwidth. An estimate that flips sign between a $10,000 and a $20,000 bandwidth is a fitting artefact rather than a finding.

Follow-up

The density test shows clear bunching just above $50,000. Is any part of the design salvageable, and what would you report instead?
Coverage also correlates with support ticket severity. Does adding severity as a control help, hurt, or do nothing in a regression discontinuity?
If you pool five fiscal years to gain power, what new identifying assumption have you taken on, and how would you check it?

Error rate halves while severe support tickets double

Hard · missing data · aggregation bias · reconciliation

The fleet-wide customer-visible error rate fell from 1.8 percent to 0.9 percent, while sev1 and sev2 tickets and reopened_count rose across the same fortnight. Using fct_api_request (account_id, environment, sdk_name, traffic_class, http_status, request_at, billable_units), fct_usage_daily (account_id, sku_code, usage_date, billable_quantity) and fct_support_ticket (account_id, severity, opened_at, reopened_count, linked_incident_id), determine whether reliability improved, and if not, identify precisely which rows are missing and from when. Deliverable: a diagnosis backed by an independent corroborating source.

Approach

Distrust an improvement that contradicts an independent operational signal. Two sources disagreeing is itself the finding; decide which one is more likely to be broken before explaining either.
Recompute the rate as the metric tree defines it, per account first and then as the share of accounts above the reliability target. A single global average is dominated by whichever account sends the most traffic, so a fleet number can fall while a quarter of accounts get worse.
Audit for missingness rather than for badness: count fct_api_request rows per hour split by environment, sdk_name and status class, indexed against the trailing same-hour baseline. A partial ingestion failure shows as a step drop confined to one slice, not a uniform decline.
Reconcile against a source the request pipeline does not feed, such as implied request volume from fct_usage_daily for the same accounts and dates. If the usage table is flat while request rows fell, rows are missing rather than traffic.
Test whether the missingness is differential by status, which is the mechanism that fakes an improvement: if 5xx rows are written on a path that stopped while 2xx rows were unaffected, the numerator falls faster than the denominator and the rate drops with nothing improving.
Close with the affected window, the affected slice and a restated series marked unreliable across that window, rather than a silently patched number.

Follow-up

The missing rows are unrecoverable. How do you present that fortnight in a series people compare week over week?
What monitor would have caught this within an hour, and what is its false-positive cost on a normal quiet weekend?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
Listen back and mark every moment you proposed a solution before the success metric existed.
Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Rigour against a deadline

Nearly every data role forces a trade between the analysis you want and the one that fits the decision window. Prepare a case where you deliberately shipped something less rigorous, named the weakness to the person relying on it, and said what would change your answer. The naming is the part interviewers listen for.

How do you handle missing values, outliers, and duplicate records in a large-scale data pipeline?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that project again?
What did you decide not to do, and why?

Report an underpowered consumption test to a non-technical executive

Medium · communication · uncertainty · experimentation

An account-randomised packaging change ran six weeks across 900 paying accounts. The effect on billable units per account per month is plus 4.1 percent, with a 95 percent interval from minus 3.2 to plus 11.8 after clustering standard errors at the account and applying the pre-registered winsorisation at the 99th percentile. An executive with no statistical background wants one number this week to decide a full rollout. Produce a three-sentence spoken answer, one chart, and an explicit recommendation of ship, stop or keep running, with the cost of each option stated.

Approach

The interviewer is probing whether you can be decision-useful without either hiding the uncertainty or hiding behind it. Start from the decision rather than the statistics: establish what the executive would do differently at plus 4 percent versus zero, because if the action is identical the interval does not matter.
Translate the interval into consequences in units the executive already reasons about. Multiply both endpoints by the cohort's baseline consumption and contracted rates to give an annualised revenue range, so the answer is a range of dollars rather than a range of percentages.
Price the option to wait. Using the observed variance, state roughly how many additional account-weeks halve the interval width, so keep running becomes a quantified choice instead of a stall.
Offer a cheaper path to the same decision: a lower-variance proximate outcome such as successful billable units on the new SKU, or CUPED using each account's pre-period consumption, quoting the expected variance reduction as one minus the squared pre-post correlation.
Give a recommendation and name the single observation that would reverse it. A strong answer commits; a generic one recites the interval and leaves the decision on the table.

Follow-up

The executive says it clearly works and is just not provable, so ship it. What is your answer?
How much of the interval width comes from clustering and how much from the revenue tail, and what would you do about each?
If you had to ship this week with no more data, which guardrail would you watch for the first fortnight and at what threshold would you roll back?

Scope an open-ended request to predict account churn

Medium · scoping · churn modelling · operating point

A customer success director asks for a list of accounts about to churn. You know only that the team has six people and that contracts are annual. Available data is fct_subscription_period, fct_usage_daily, fct_api_request, fct_support_ticket and dim_account. Before writing any code, produce the questions you need answered, a proposed definition of about to churn, and the shape of the artefact you would hand back, including the operating point that turns a score into a decision.

Approach

The interviewer is probing whether you convert a vague request into a decision with a capacity constraint attached. A candidate who starts talking about model families has already failed the exercise.
Pin the event and the horizon first. Churn is only possible at term_end_date, so the population is accounts renewing in the next 60 to 90 days, not the whole base. Ask explicitly whether contraction and downgrade count as churn or only full non-renewal, because the three have different base rates and different interventions.
Pin the action and the capacity. Six people times a realistic number of meaningful interventions per week gives k, and k is what the list is ranked to. Evaluate on precision at k rather than a global AUC over accounts that will never be contacted.
Audit leakage before choosing features. Every feature needs a timestamp proving it existed before the prediction date. A downgrade amendment, a churn reason code, and a ticket opened after the renewal conversation started are all leaks that will make the offline number look excellent and the live list useless.
Ask for the counterfactual now rather than later. Coverage is assigned deliberately, so without a held-out slice agreed at the start the intervention can never be evaluated, and you will be asked for its impact in nine months regardless.
Propose the smallest artefact that closes the loop: a weekly ranked list sized to capacity with two or three inspectable reasons per row, plus a stated policy for accounts below the line.

Follow-up

The director insists all accounts are in scope, not only those renewing soon. How do you answer without simply refusing?
Historical non-renewals number about 30 a year. At what point do you tell them a model is the wrong tool and a rules list is better?
Which candidate features would you drop purely because you cannot date them?