Interview guides · BAE Systems

BAE Systems Data Scientist Interview Questions & Guide 2026

Practice explaining analysis before a BAE Systems data scientist interview: the recommendation, the uncertainty, the follow-ups.

The 60-second brief

As a Data Scientist at BAE Systems, you occupy a critical junction between advanced analytical research and the practical, mission-critical engineering that defines the company’s global footprint. Your work is not merely about model accuracy; it is about providing actionable intelligence that supports complex systems, ranging from aerospace and defense technologies to large-scale operational logistics. You will be tasked with transforming raw, high-dimensional data into robust solutions that enhance decision-making and improve the efficiency of systems that operate in some of the most demanding environments on Earth.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

Separate contracted seats from actively used seats
Power experiments for heavy-tailed account revenue
Strip CI, retry and synthetic traffic first

Overview

This role requires a blend of rigorous technical expertise and a pragmatic, product-focused mindset. You will often work within multidisciplinary teams, collaborating with software engineers and domain experts to solve problems where the stakes are high and the data is often noisy or incomplete. Whether you are optimizing predictive maintenance models or evaluating the performance of new system features through rigorous experimentation, your contributions directly influence the reliability and strategic capabilities of ’ diverse product portfolio.

The BAE Systems interview process

BAE Systems candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Automated Assessments

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Technical Coding Challenges

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Behavioral Interviews

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

Whether you can state the other side's argument accurately before you explain why you disagreed
What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Project Presentations

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Complex Problem-Solving

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Treating raw request or usage volume as engagement

Most traffic in this domain is emitted by machines. Continuous-integration pipelines, scheduled batch jobs, synthetic monitors, backfills and client retries can all grow by an order of magnitude from one configuration change made by one engineer, and none of it represents a new decision to use the product. The inversion is what makes it dangerous: when the platform degrades, clients retry, so error-driven retry volume rises at the exact moment the customer is most likely to leave, and an engagement dashboard built on raw counts shows growth immediately before a churn. Filter on traffic_class and on successful status before anything else, and keep failed-request volume as its own separate series.

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Averaging per-user rates to produce a population rate

Decide which quantity you want: the mean of per-user ratios and the ratio of summed numerator to summed denominator are different estimands, and heavy users dominate one but not the other. For a ratio metric, aggregate numerator and denominator separately and use the delta method for its variance.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

Audit daily usage rows for grain and arithmetic violations

Easy · data-quality · pandas · grain

You are handed fct_usage_daily as a pandas DataFrame with account_id, workspace_id, sku_code, usage_date, billable_quantity, included_quantity_applied, overage_quantity, list_amount_cents, discount_amount_cents, net_amount_cents, cogs_cents, is_restated, first_written_at and restated_at. The declared grain is one row per (account_id, workspace_id, sku_code, usage_date). Write audit(df) returning a DataFrame with one row per failing check: check name, failing row count, and one example key. Cover at minimum grain duplication, negative quantities or amounts, the identity net = list - discount, billable = included + overage, and rows where is_restated is true but restated_at is null.

Approach

Check the grain before anything else with df.duplicated(subset=key, keep=False), and count rows rather than groups so a key appearing twice contributes 2 — if the grain is broken every arithmetic count below it is uninterpretable.
Express each invariant as a boolean Series over the whole frame. The cent columns are integers and compare exactly, so use !=; the numeric(18,6) quantity columns need np.isclose with atol=1e-6 because included + overage is a decimal sum.
Handle null as its own failure mode. Comparisons against NaN return False, so a check written as rows_that_pass = (a == b - c) silently files every null-amount row wherever the negation happens to land; build each check as violations = ~condition | column.isna().
Collect the checks as a list of (name, mask) pairs and assemble the output in one pass, so adding a check is one line and every check reports in the same shape.
Order the output with structural failures (grain, null keys) above arithmetic failures, and report zero-count checks too — a check that silently disappears when it passes is indistinguishable from a check that was never run.

Worked solution (20 min)

key = ['account_id','workspace_id','sku_code','usage_date']; dup = df.duplicated(key, keep=False); record dup.sum() and df.loc[dup, key].iloc[0].to_dict() as the example.
neg = (df[['billable_quantity','included_quantity_applied','overage_quantity','list_amount_cents','net_amount_cents','cogs_cents']] < 0).any(axis=1); net_bad = df.net_amount_cents.isna() | (df.net_amount_cents != df.list_amount_cents - df.discount_amount_cents).
qty_bad = ~np.isclose(df.billable_quantity, df.included_quantity_applied + df.overage_quantity, atol=1e-6) | df.billable_quantity.isna(); restated_bad = df.is_restated & df.restated_at.isna().
Assemble pd.DataFrame([{'check': n, 'failing_rows': int(m.sum()), 'example': first_key(m)} for n, m in checks]) with the grain and null checks listed first.

Expected result: One row per check with failing_rows as an integer count of rows rather than groups, and a JSON-serialisable example key for any check with a non-zero count; a single doubled key contributes 2 to the grain check.

Verify it yourself

df.groupby(key).size().max() == 1 if and only if the grain check reports zero.
When the net check reports zero, sum(net_amount_cents) == sum(list_amount_cents) - sum(discount_amount_cents) exactly.
Running audit on df.sample(frac=1, random_state=0) gives identical counts; the checks must not depend on row order.
Corrupting one net_amount_cents by a single cent raises exactly one check's count by exactly 1.

Follow-up

Which of these should block a dashboard refresh and which should only warn?
Rows with is_restated = true legitimately change value after first write. How do you make yesterday's audit result reproducible?
How would you extend this to catch a partition that is missing entirely rather than wrong?

Implement seven-day activation rate from its written definition

Easy · metrics · pandas · activation

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

Approach

Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Follow-up

Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
How would you decide whether 168 hours is the right window rather than 72 or 336?
An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Join daily usage to the contract version live that day

Medium · as-of-join · pandas · numpy · contracts

fct_usage_daily has account_id, usage_date and net_amount_cents. fct_subscription_period has account_id, subscription_period_id, plan_tier, term_start_date, term_end_date, arr_cents and is_current, with one row per contract term version and every amendment inserting a new row. Attach to each usage row the subscription_period_id whose term brackets usage_date (term_start_date <= usage_date <= term_end_date). pd.merge_asof and an interval-condition merge are unavailable; use sorting and numpy.searchsorted. Then report monthly net revenue by plan_tier. Terms for one account do not overlap, and usage may fall outside every term.

Approach

Say out loud why the cheap version is wrong: joining on is_current stamps today's plan tier onto last year's usage, so every account that upgraded has its history reclassified and revenue-by-tier becomes a function of when the query ran.
Sort each account's terms by term_start_date and use np.searchsorted(term_start, usage_date, side='right') - 1 to get the last term that started on or before the usage date. Do it per account group, or globally after encoding (account_id, date) into one monotone key.
searchsorted only enforces the left edge. Validate the right edge afterwards — usage_date <= the candidate's term_end_date — and set the match to NA where it fails. That NA is usage in a gap between contracts and must stay visible instead of being folded into the expired term.
Assert non-overlap before trusting the lookup, and write the assertion so it is capable of passing. prev_end = terms.groupby('account_id').term_end_date.shift() is NaT on each account's first row, and NaT < Timestamp evaluates to False rather than NA, so a comparison followed by .fillna(True) has nothing left to fill and the assertion fires on every account's first term whatever the data looks like. Guard the null yourself: assert (prev_end.isna() | (prev_end < terms.term_start_date)).all(). The failure mode of getting this wrong is not a false alarm you notice once — it is an assertion someone deletes because it never passes, after which overlapping terms make searchsorted return one of them with no trace in the output.
Aggregate after the join, grouping by (usage_date month, plan_tier) with dropna=False so the unmatched bucket appears as its own row and the total still ties to the ungrouped sum of net_amount_cents.

Follow-up

An amendment takes effect on the 17th of a month. How do you report that month's revenue by tier?
What changes if terms can overlap because of a co-term amendment?
How would you verify this against a SQL implementation using a BETWEEN condition?

How would you handle missing values or anomalies in a large time-series dataset?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.
Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Write a query using SQL window functions to calculate a rolling 30-day average of system errors.

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Find paid seats that never called the API

Easy · anti-join · null semantics · seat utilisation

dim_user_membership carries user_id, account_id, seat_type, is_service_account, deactivated_at and last_seen_at. fct_api_request carries user_id, which is null whenever the caller is a service account or an unattended key, plus account_id, request_at and http_status. For a single account, list every licensed_paid seat with is_service_account = false and deactivated_at null that issued no request at all in the trailing 90 days, returning user_id and last_seen_at. A first draft writes NOT IN against a subquery over fct_api_request.user_id. State exactly what that draft returns and why, then write the correct query.

Approach

Read the column comment first: user_id is nullable on the fact, so the subquery almost certainly contains at least one NULL for any account that runs a service account or an unattended key.
Work the three-valued logic out loud. x NOT IN (a, NULL) expands to NOT (x = a OR x = NULL); the second disjunct is UNKNOWN, so for any x not equal to a the whole predicate is UNKNOWN and the row is filtered out. The draft returns zero rows, which reads as full seat utilisation.
Rewrite as NOT EXISTS with the 90-day and status predicates inside the correlated subquery. Putting them in the outer WHERE instead turns the anti-join into a different question and silently changes the answer.
Correlate on both user_id and account_id. A membership is (user_id, account_id) and one person can hold memberships in several accounts, so correlating on user_id alone marks a seat as active because that human was busy somewhere else.
Filter the seat side to seat_type = 'licensed_paid', is_service_account = false and deactivated_at IS NULL, and say what the resulting count means next to contracted_seats on the current subscription row.

Worked solution (15 min)

Confirm the hazard with one query: SELECT count(*) FROM fct_api_request WHERE user_id IS NULL AND account_id = :account_id. Any non-zero result proves the draft returns nothing.
Write the seat side: SELECT user_id, last_seen_at FROM dim_user_membership WHERE account_id = :account_id AND seat_type = 'licensed_paid' AND NOT is_service_account AND deactivated_at IS NULL.
Attach AND NOT EXISTS (SELECT 1 FROM fct_api_request r WHERE r.user_id = m.user_id AND r.account_id = m.account_id AND r.request_at >= now() - interval '90 days').
Compare the row count against the same query written as a LEFT JOIN with a WHERE r.user_id IS NULL; the two must agree exactly.
Divide the active seat count by contracted_seats from the account's current fct_subscription_period row and state the utilisation figure.

Expected result: A list of user_id and last_seen_at for licensed, non-service, active memberships on that account with no request in 90 days. The NOT IN draft returns zero rows whenever the account has any service-account or unattended-key traffic, and returns the right answer only by accident on accounts that have none.

Verify it yourself

The NOT EXISTS result and the LEFT JOIN ... IS NULL result return identical user_id sets.
Idle seats plus seats with at least one request in 90 days equals the total licensed non-service active membership count for the account.
Forcing a NULL into the subquery (union a SELECT NULL) leaves NOT EXISTS unchanged and empties the NOT IN version, which demonstrates the failure on demand.

Follow-up

Adding AND user_id IS NOT NULL to the subquery also fixes NOT IN. Why is NOT EXISTS still the form you would leave in the repository?
last_seen_at looks like a shortcut for the whole query. What does it actually record, and where does it disagree with API activity?
This is a seat-reduction risk list. What threshold would you attach before handing it to an account team, and what happens to seats below it?

If you notice a sudden, significant drop in a core platform metric, what is your step-by-step process for diagnosing the root cause?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you define "user engagement" for a complex internal dashboard used by engineering teams?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you design a metric to measure the success of a new predictive maintenance feature?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you balance the need for short-term performance gains with long-term system stability?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

What are the most common experimentation pitfalls you have encountered when running A/B tests?

Medium · a/b testing and experimentation

Approach

Decide the analysis before seeing data, including how long it runs and when you look.
Name the randomisation unit first; it decides the variance and what the test can detect.
Say whether units interfere with each other, and switch design if they do.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

If an A/B test shows a positive result but the p-value is high, how do you interpret those findings?

Medium · a/b testing and experimentation

Approach

Name the guardrails that would stop a launch even on a positive primary result.
Say whether units interfere with each other, and switch design if they do.
State the primary metric and the minimum effect worth shipping, then size the test.

Follow-up

What would you do if you could not randomise at all?
What would you conclude if the result is positive but the test is underpowered?

Triage a sample ratio mismatch before reading the result

Easy · srm · assignment · instrumentation

An account-randomised test assigned arms 50/50 by hashing account_id. Exposure logging recorded 4,200 accounts: 1,987 in treatment and 2,213 in control. A chi-square goodness-of-fit test against the intended 2,100/2,100 split gives chi-square(1) = 12.2, p about 0.0005. The analyst reports the primary metric up 6% in treatment and asks to ship. Explain what the imbalance implies about that 6%, rank the causes you would investigate first given this domain's data, and name for each the single query against fct_api_request or dim_account that confirms or eliminates it.

Approach

State the rule before diagnosing anything: a sample ratio mismatch below a 0.001 alarm threshold invalidates the readout. Whatever mechanism removed 113 accounts from one arm almost certainly removed them non-randomly, which makes it a selection effect on the outcome. The 6% is not to be adjusted, caveated or shipped; it is unusable until the mechanism is named.
Verify the test is testing the right thing. chi-square = (1987 - 2100)^2 / 2100 + (2213 - 2100)^2 / 2100 = 6.08 + 6.08 = 12.16 on one degree of freedom. Confirm the denominator is the intended assignment count rather than the observed total, and that you are testing assignment rather than an analysis population that has already been filtered.
Rank causes by how this domain actually breaks rather than by textbook order: is_internal accounts filtered after assignment instead of before; exposure logged on a code path that returns 5xx more often in one arm, silently dropping those accounts; assignment recorded at first request but exposure at a later event, so accounts that churned in between appear in only one arm; dim_account type-2 versioning giving one account_id two surrogate keys and two hash inputs; deployment_model = 'self_hosted' accounts that never emit exposure telemetry at all.
Attach one discriminating query to each: count arms before the is_internal filter; compare the http_status >= 500 rate by arm on the exposure endpoint in fct_api_request; compare the distribution of created_at and churned_at by arm in dim_account; count distinct account_sk per account_id inside the experiment window; cross-tabulate arm against deployment_model.
Compare counts at three fixed stages, assignment, first exposure, and analysis population, and localise the divergence to one of them. The stage where the arms first separate names the subsystem, and everything downstream of it is a symptom.
Report the outcome as abort, fix, rerun, and be explicit about what survives: the variance estimate for re-powering, the instrumentation fix, and nothing whatsoever about the effect size.

Worked solution (20 min)

Recompute the statistic: two terms of 113^2 / 2100 = 6.08, total 12.16, p about 0.0005 on one degree of freedom, below the 0.001 alarm line.
Pull arm counts at assignment, at first exposure, and in the analysis population, and find the first stage at which they diverge.
At that stage, cross-tabulate arm against is_internal, deployment_model, and the 5xx rate on the exposure endpoint.
Write the abort note naming the mechanism, the corrected code path, and the rerun date, with the effect estimate explicitly withheld.

Expected result: chi-square(1) = 12.16, p about 0.0005, so a 0.001 alarm threshold fires and the readout is abandoned. The divergence is localised to a single pipeline stage by comparing arm counts at assignment, at exposure and at analysis, and the most likely causes in this domain are post-assignment internal-account filtering or exposure logging that fails asymmetrically on 5xx responses.

Verify it yourself

A 113-account shortfall on 2,100 expected is a 5.4% deficit in one arm, which under a sound hash is roughly a 1-in-2,000 event. That is what the 0.001 threshold is calibrated to catch.
Counts at the assignment stage should already be balanced if the bucketing is correct. If they are not, the bug is in the hash or the salt, not in the logging.
Any fix that restores balance by filtering on a variable measured after exposure has conditioned on a post-treatment variable and reintroduces the bias it appears to remove.

Follow-up

The imbalance disappears once you restrict to accounts with at least one successful request. Does that fix the experiment or confirm the bug?
What alarm threshold would you set for this check, and why is 0.05 the wrong one for a diagnostic you run on every experiment every day?
How would you detect a mismatch confined to one segment while the overall split looks clean?

Net revenue retention jumps sixteen points in one month

Medium · versioned tables · fan-out · metric definition

Trailing-twelve-month net revenue retention printed around 108 percent for months and now reads 124 percent, with no unusual deals closed. The query sums arr_cents from fct_subscription_period (account_id, arr_cents, term_start_date, term_end_date, amendment_type, superseded_by_id, is_current, booked_at) filtered on is_current = true at month M, across accounts holding arr_cents > 0 at month M-12. Find the defect, correct the number, and rewrite the definition so the next person cannot reintroduce it.

Approach

Audit the grain before the arithmetic: count account_ids holding more than one row with is_current = true and superseded_by_id null. A versioned contract table that double counts one amendment batch inflates the numerator while leaving the denominator untouched.
Replace is_current with an as-of selection on both dates, taking the version whose term_start_date and term_end_date bracket the reporting date and tie-breaking on latest booked_at. The numerator is read as of M and the denominator as of M-12; neither uses today's live version.
Verify the cohort is frozen. The account set is fixed at M-12 and nothing acquired since may enter the numerator, so check that no join to a current-period table quietly re-admits new accounts.
Confirm the estimand is a ratio of sums rather than a mean of per-account ratios. Contraction is floored at zero while expansion is unbounded, so the two constructions differ systematically and the second is far noisier.
Reissue the definition with the failure modes written into it: exactly one row per account per date by construction, cohort frozen at M-12, churned accounts contributing zero rather than dropping out of the numerator.

Follow-up

A churned account should contribute zero rather than disappear. What does the ratio do under each treatment, and which one is correct?
How would you unit-test this metric so a future amendment batch with the same defect fails a check instead of reaching a board slide?

Suggested practice checklist

For someone who has spent the last year in notebooks, dashboards or modelling work and has not written raw SQL under time pressure. The first four days rebuild query fluency against a fixture you control and can verify by hand; the last three attach that fluency to the rest of the loop.

Day 1 — Build a fixture you can check answers against

Create a local Postgres or SQLite database with four tables (users, sessions, events, orders) holding roughly 200 rows you generated yourself, so you know the contents well enough to predict every result.
Deliberately seed the cases that break queries: a user with no sessions, a session with no events, two orders sharing a timestamp, a NULL in one join key, and one duplicated user row.
Before writing any SQL, hand-compute five answers on paper (how many users placed at least one order, median orders per ordering user, and three others) and save them as the ground truth for the week.

Deliverable: A one-command seed script plus a text file of five hand-computed answers to grade every later query against.

Day 2 — Joins, filters and NULL semantics

Answer "which users have no orders" three ways (LEFT JOIN with IS NULL, NOT EXISTS, NOT IN) and confirm that the NOT IN version returns zero rows once the subquery contains a NULL, because the comparison is never TRUE.
Reproduce the LEFT JOIN that silently collapses to an inner join by putting a right-table predicate in WHERE, then fix it by moving the predicate into the ON clause, and record both row counts.
Create a fan-out bug on purpose by joining orders to order_items and summing the order total, then correct it with a pre-aggregated subquery and explain in one line which table changed the grain.

Deliverable: One annotated .sql file holding the three join traps, each with the wrong result and the corrected result side by side.

Day 3 — Window functions and frames

Write three window queries against the fixture: a running order total per user, the rank of each order within its user by value, and the day gap to that user's previous order, then check each against the day-one ground truth.
Run ROW_NUMBER, RANK and DENSE_RANK over a column containing ties, print all three side by side, and write one sentence on when each is the correct choice.
Switch one query from the default frame (RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which is what you get when ORDER BY is present and no frame is written) to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and explain why the output differs only when the ORDER BY column has duplicates.

Deliverable: Three verified window queries plus a short note explaining the RANGE versus ROWS difference in your own words.

Day 4 — The four analytical query patterns

Write a monthly retention grid: first order month per user, then months-since-first as the column, and verify that month zero equals the cohort size exactly.
Sessionize the events table under a 30-minute inactivity rule using LAG plus a cumulative sum over a new-session flag.
Build a four-step funnel that counts distinct users rather than events at each step, and state the rule you applied to a user who reaches step three without ever logging step two.

Deliverable: One file with the retention, sessionization and funnel patterns, each carrying a one-line note on the assumption it bakes in.

Day 5 — Write SQL the way you will have to write it live

Set a 12-minute timer and solve three medium prompts in a plain editor with no execution and no autocomplete, then run them and tally syntax errors separately from logic errors.
Narrate one solution aloud while writing it, stating the grain of each intermediate result (one row per user, one row per user-day) before you type its body.
Rewrite your slowest solution as a CTE chain where every CTE name states its grain, and time yourself re-solving it from blank.

Deliverable: A recording of one narrated solution plus an error tally that separates syntax from logic.

Day 6 — One day for everything that is not SQL

Write the preconditions of the two-sample t-test from memory, then check them: independent observations, and a difference in means whose sampling distribution is approximately normal, which at large sample sizes follows from the central limit theorem rather than from normality of the raw values.
Write the difference between an odds ratio from logistic regression and a relative risk, and state the condition under which the two are close (low outcome prevalence).
Prepare a 90-second answer to "how would you know this model is any good" that names the metric, the baseline you would beat, and the cost of the errors you care about.

Deliverable: One page of notes covering test preconditions, the odds-ratio caveat and the model-quality answer.

Day 7 — Full loop rehearsal

Run a 45-minute mock with someone willing to interrupt: 20 minutes of SQL, 15 minutes defining a metric, 10 minutes on a past project.
Re-solve from blank the two queries you were slowest on this week and compare the times against day five.
Write a five-line answer to "walk me through a project" that puts a number in the first sentence and names the decision the work changed.

Deliverable: Mock feedback notes plus a timed project narrative you can deliver without reading it.

Getting an analysis acted on

Work that nobody used is a common and unflattering pattern in data careers, and interviewers probe for it. Have a story about an analysis that changed a decision, and be specific about how you got it in front of the person who could act. Also have one about work that went nowhere, with your reading of why.

How do you handle disagreements within a team regarding the choice of a machine learning model or analytical approach?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

Scope an open-ended request to predict account churn

Medium · scoping · churn modelling · operating point

A customer success director asks for a list of accounts about to churn. You know only that the team has six people and that contracts are annual. Available data is fct_subscription_period, fct_usage_daily, fct_api_request, fct_support_ticket and dim_account. Before writing any code, produce the questions you need answered, a proposed definition of about to churn, and the shape of the artefact you would hand back, including the operating point that turns a score into a decision.

Approach

The interviewer is probing whether you convert a vague request into a decision with a capacity constraint attached. A candidate who starts talking about model families has already failed the exercise.
Pin the event and the horizon first. Churn is only possible at term_end_date, so the population is accounts renewing in the next 60 to 90 days, not the whole base. Ask explicitly whether contraction and downgrade count as churn or only full non-renewal, because the three have different base rates and different interventions.
Pin the action and the capacity. Six people times a realistic number of meaningful interventions per week gives k, and k is what the list is ranked to. Evaluate on precision at k rather than a global AUC over accounts that will never be contacted.
Audit leakage before choosing features. Every feature needs a timestamp proving it existed before the prediction date. A downgrade amendment, a churn reason code, and a ticket opened after the renewal conversation started are all leaks that will make the offline number look excellent and the live list useless.
Ask for the counterfactual now rather than later. Coverage is assigned deliberately, so without a held-out slice agreed at the start the intervention can never be evaluated, and you will be asked for its impact in nine months regardless.
Propose the smallest artefact that closes the loop: a weekly ranked list sized to capacity with two or three inspectable reasons per row, plus a stated policy for accounts below the line.

Follow-up

The director insists all accounts are in scope, not only those renewing soon. How do you answer without simply refusing?
Historical non-renewals number about 30 a year. At what point do you tell them a model is the wrong tool and a rules list is better?
Which candidate features would you drop purely because you cannot date them?

Disagree with a product manager about an adoption claim

Medium · conflict · telemetry hygiene · adoption metrics

A product manager is about to present that a new SDK release drove a 40 percent rise in requests among adopting accounts, computed from fct_api_request counts grouped by sdk_version. You find the rise is concentrated in traffic_class equal to ci, that rows with is_retry true grew alongside it, and that restricting to interactive non-retry traffic leaves a 3 percent lift. The launch review is in two days. Decide how you raise this, with whom and in what order, and what you propose the claim becomes.

Approach

The interviewer is probing whether you can correct a colleague without ambushing them, and whether your own counter-analysis carries the caveats theirs lacked. Go to the product manager privately before the review. A correction delivered in the room is a status move and loses the argument you are actually trying to win.
Bring a decomposition rather than a verdict: the same accounts and window, requests split by traffic_class with retries held out as their own column, so their 40 percent and your 3 percent reconcile line by line and neither has to be taken on trust.
Reproduce their figure exactly first. If you cannot land on 40 percent with their method, you do not yet know what you are disagreeing with.
Ask whether the continuous-integration lift is itself valuable. An account wiring the SDK into its pipeline has increased integration depth, which is the dominant switching cost in this domain, so the honest claim may be that integration depth rose while interactive usage moved 3 percent. Improving the claim beats deleting it.
Name the mechanism that makes the raw count dangerous: clients retry when the platform degrades, so retry volume climbs exactly when the customer is most at risk. Pull the 5xx rate for the same accounts and window before anyone concludes anything, and note that billable_units is zero on 5xx rows, so request counts and billable quantities diverging is itself the signal.
Close with a standing definition for launch metrics so the next release does not repeat the exercise.

Follow-up

The product manager argues that continuous-integration traffic is real usage and declines to split it out. Is that position defensible, and under what metric definition?
Suppose the 5xx rate for those same accounts also rose 40 percent. What is the claim now?
The review happens and the raw number is presented regardless. What do you do next, and what do you not do?