Interview guides · Clearancejobs

Clearancejobs Data Scientist Interview Questions & Guide 2026

Data scientist interview prep for Clearancejobs: original practice on experiment design, SQL and metric reasoning, with worked solutions.

The 60-second brief

Within the defense, intelligence, and national security sectors, a Data Scientist is not just an analyst; they are a critical asset driving mission-focused insights from some of the world's most complex and sensitive datasets. Working with elite federal defense contractors such as Sentar, Northrop Grumman, Wyetech, and AeroVironment, you will develop, sustain, and scale advanced analytics tools that directly impact national security, aerospace systems, and strategic military operations.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Size an experiment before anyone launches it
Define numerator, denominator and window precisely
Separate novelty effects from durable behaviour change

Overview

This role requires a unique intersection of high-level mathematics, computer science, and domain-specific defense knowledge. Whether you are optimizing manufacturing workflows for aerospace systems, implementing predictive modeling and Monte Carlo simulations, or parsing unorganized and unstructured agency data, your work will directly influence executive decision-making and tactical operations.

The environments you will work in are highly secure, fast-paced, and collaborative. Your day-to-day will involve translating complex mission requirements into technical specifications, building reproducible machine learning pipelines, and presenting your findings to both technical peers and non-technical military commanders or corporate executives. It is a high-stakes, intellectually stimulating role where your contributions have a tangible impact on global safety and technological superiority.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Watching an experiment daily and stopping when it crosses significance

A fixed-sample test controls type I error at one pre-declared look. Checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate to roughly 0.15 to 0.20 for ten looks, and it rises further with more frequent checks, because the p-value takes a random walk that will eventually dip below the threshold under the null. The usual defences are a fixed horizon declared before launch, group-sequential boundaries such as O'Brien-Fleming that spend alpha across a planned number of looks, or always-valid confidence sequences that are correct under continuous monitoring. Compounding it, the effect size reported conditional on having crossed the threshold is biased away from zero, and the bias is larger the lower the power was, so an underpowered test that 'won' typically overstates the lift it found.

Interpreting a change before checking data quality and logging

Spend the first pass on row volume by day, null rates, duplicate keys, and whether the step change lands on a release or tracking-migration date. A discontinuity that coincides with a deploy is an instrumentation hypothesis before it is a behavioural one.

Generalising beyond the population the sample actually supports

State the frame the sample was drawn from and where it diverges from the population you want to talk about: time window, platform, geography, opt-in. If a group is excluded from the frame, either weight to known margins or narrow the claim rather than quietly extending it.

Questions & practice

Explain the mathematical difference between L1 (Lasso) and L2 (Ridge) regularization, and how you would choose between them for a highly sparse dataset.

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Check what information would not exist at prediction time, and exclude it.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

What are the core assumptions of a linear regression model, and how do you test for and mitigate violations like heteroscedasticity in Python?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

How do you address the challenges of class imbalance when training a predictive model on highly sensitive, low-frequency security events?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Say how the offline result would be validated online before it is trusted.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Describe the process of designing and running a Monte Carlo simulation to forecast project schedule risks or manufacturing bottleneck probabilities.

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Say how the offline result would be validated online before it is trusted.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

Audit a one-day event extract for structural defects

Easy · data quality · late arrival · pandas

You receive a one-day extract of fct_event as a DataFrame with event_id, occurred_at_utc, received_at_utc, visitor_id, user_id, account_id, event_name, is_bot_flagged and surface. Write a function returning one row per data-quality rule with the rule name, the failing row count and the failing share of the extract. Cover at minimum: duplicate event_id, received_at_utc earlier than occurred_at_utc, occurred_at_utc later than the extract's maximum received_at_utc, account_id present while user_id is NULL, and rows whose occurred_at date differs from their received_at date. Do not drop rows; report only.

Approach

Compute the extract's own reference clock first: max(received_at_utc). Wall-clock now() is wrong here because the extract may be replayed days later, which would turn every row into a future-dated failure.
Express each rule as a boolean Series over the same index so the checks compose, then aggregate with .sum() and divide by len(df). Building a list of (name, mask) pairs keeps the rule set extensible and keeps one code path for counting.
For the duplicate rule, decide and state the convention: df.duplicated('event_id', keep=False).sum() counts every member of a duplicated group, df.duplicated('event_id').sum() counts only the surplus copies. Either is defensible; an unstated choice is not. The rest of this item assumes keep=False.
Treat received_at < occurred_at as clock skew, not corruption: occurred_at is client-supplied. Separate it from the date-mismatch rule, which is the one that actually breaks a daily metric keyed on occurred_at.
Know which rules imply which before you read the counts. A row whose occurred_at exceeds max(received_at_utc) has its own received_at no later than that maximum, so it is necessarily a clock-skew row as well: the future-dated mask is a subset of the skew mask, always. Neither is a subset of the date-mismatch mask, because skew of a few minutes inside one UTC date mismatches nothing.
Return a tidy DataFrame sorted by failing_share descending, and add a boolean column saying whether the rule should block publication, so the output is a decision rather than a list of numbers.

Worked solution (20 min)

Parse both timestamp columns with utc=True and assert the dtype, since a silently-object column makes every comparison string-wise and wrong.
Set ref = df['received_at_utc'].max() and build the five masks against it.
Assemble results as pd.DataFrame(rows) with columns rule, failing_rows, failing_share, blocks_publication.
Keep the masks addressable (a dict of name to Series) rather than only their sums, so the overlap between rules can be asserted rather than assumed.
Verify the function is pure: assert the input frame's shape is unchanged after the call.

Expected result: A DataFrame with one row per rule, failing_share equal to failing_rows divided by len(df) for every row, and the input frame returned unmodified. The rules overlap rather than partition the extract, and one containment is structural: every future-dated row is also a clock-skew row. The counts therefore must not be summed or presented as a total.

Verify it yourself

Plant violations deliberately and assert an exact expected vector rather than assuming one hit per rule. On a 10-row frame carrying two rows that share an event_id, one same-date skew row, one row whose occurred_at sits an hour past max(received_at_utc), one row with account_id set and user_id NULL, and one row whose occurred_at falls on the previous UTC date: duplicate_event_id is 2 under the keep=False convention, clock_skew is 2 (the planted skew row plus the future-dated row, which cannot avoid tripping it), future_dated is 1, orphan_account is 1, date_mismatch is 1.
Assert the containment directly rather than by counting: (future_dated_mask & ~clock_skew_mask).sum() is 0. A non-zero value means the reference clock was not max(received_at_utc).
Feed a clean frame and confirm all counts are 0 and no rule errors on empty masks.
Confirm len(input) is identical before and after, since the deliverable is a report and not a cleaned frame.

Follow-up

The date-mismatch count is 2.1 percent on this extract. What late-arrival rule would you write for a daily metric, and how many days would you hold the number open?
Duplicate event_id values appear only on the 'core_action_completed' event. What upstream cause would you check before deduplicating?
Which of these rules should fire an alert at the pipeline, and which should only appear in a weekly review?

Find reactivation gaps in account paid-period history

Hard · gaps-and-islands · window-functions · subscriptions

fct_subscription_period holds account_id, subscription_id, period_start_utc, period_end_utc, period_status and change_reason. A mid-period plan or seat change closes one row and opens another, so a single continuous paid tenure is often many rows, and an account may hold two overlapping subscriptions. Collapse rows with period_status in ('active','past_due') into continuous tenures per account, treating gaps of three days or less as continuous. Return account_id, tenure_start, tenure_end, and for every tenure after the first, the gap in days that preceded it.

Approach

Filter to paid rows only: period_status IN ('active','past_due'). Trialing periods are not tenure, and including them turns every trial that never converted into a one-period tenure followed by a fake churn.
Order by period_start_utc and take a running maximum of all prior ends: MAX(period_end_utc) OVER (PARTITION BY account_id ORDER BY period_start_utc, period_end_utc, subscription_id ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). Those three columns are the only stable ordering this schema exposes, so check first that they are unique within an account; if rows tie on all three, the island numbering is order-dependent between runs and you need a real row key before the result is reproducible.
LAG on its own is wrong here because with overlapping or nested periods the immediately preceding row by start date is not the one that ends latest, so the running maximum is the part that cannot be shortcut.
Flag a new island when prior_max_end IS NULL OR period_start_utc > prior_max_end + interval '3 days', then number islands with a running SUM of the flag over the same ordering and an explicit ROWS frame.
Group to (account_id, island) taking MIN(period_start_utc) and MAX(period_end_utc), then LAG(tenure_end) OVER (PARTITION BY account_id ORDER BY tenure_start) to compute the preceding gap in days for every tenure after the first.
Sanity-check with change_reason, which is the only lineage this schema carries: list its distinct values first, then confirm that rows recording a plan or seat change sit inside a tenure rather than opening one, and that every tenure after the first opens on a row whose reason records a restart rather than an ordinary renewal. Do not reconcile against a churn timestamp on dim_account, which this schema does not define; and where such a column does exist, a cancellation timestamp records when the request was made and routinely sits weeks before the period it ends.

Follow-up

Why three days of grace? What do 0 and 30 days each do to the count of accounts classed as reactivated?
An account runs two concurrent subscriptions for different teams. One tenure or two, and what does the revenue reader expect?
How would you turn these tenures into a monthly gross logo churn series without double-counting an account that churned and returned in the same month?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
Write the active-account spine and the 28-day activity CTE.
Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
No account_id appears twice in the output.
Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

Halve the runtime of a flat test with pre-period data

Medium · cuped · variance reduction · stratification

A test on 28-day core actions per user is powered for six weeks and the team will not wait. You have fct_event (user_id, occurred_at_utc, is_core_action) covering the period before each unit's first_exposed_at_utc in fct_experiment_exposure, and the pre-period count correlates with the in-experiment count at r = 0.6. Apply CUPED, state precisely how much runtime it buys, and handle the units with no pre-period because they signed up after the experiment started.

Approach

Define the adjusted outcome: Y_cuped = Y - theta (X - Xbar), with theta = Cov(Y, X) / Var(X) estimated on the pooled sample so theta itself carries no treatment effect. It is unbiased because X is pre-exposure and therefore has equal expectation in both arms.
Quantify the gain exactly: Var(Y_cuped) = Var(Y)(1 - r^2) = Var(Y) x 0.64, so 36% of the variance is removed. Required sample scales with variance, so six weeks becomes 3.8 weeks, and at the original sample the MDE shrinks by sqrt(0.64) = 0.8.
State the precondition that actually bites: X must be measured strictly before first_exposed_at_utc, not before assigned_at_utc and not before the calendar start of the experiment. Any post-exposure information in the covariate lets the treatment effect leak in and biases the estimate rather than merely failing to help.
Handle missing pre-periods as a stratum, never by dropping rows. Dropping changes the population; setting X to the pooled mean leaves those units' adjustment at zero, which is unbiased but buys them no reduction, so report the covered share and expect the blended reduction to fall short of 36%.
Note that stratification is the same mechanism with a categorical covariate. Post-stratifying on device_type and referrer_channel removes the variance those stratum means explain, and it composes with CUPED provided the strata are also fixed before exposure.

Follow-up

Here the covariate is the same metric as the outcome. When would you deliberately pick a different one, and what is the risk if that covariate is itself affected by treatment?
The point estimate moves by 0.4 standard errors when CUPED is applied. Is that reassuring or alarming, and what would you check?
How does CUPED combine with cluster randomisation on account_id, and at which grain is the covariate built?

A test that won on day four and lost by day twelve

Medium · peeking · sequential testing · multiple comparisons · winner's curse

A two-week test was checked every morning. On day four the primary metric crossed p = 0.03 and the team shipped. By day twelve, with logging still running, the estimate had fallen to a 0.2% lift at p = 0.61. The day-four deck also reported a significant 9% lift among paid_search visitors on mobile_web, one of twenty segments inspected. Explain both patterns quantitatively, say what the day-four evidence was actually worth, and specify the monitoring rule that prevents a repeat.

Approach

Price the peeking. A fixed-sample test controls type I error at one pre-declared look. Ten daily looks at a nominal 5% raise the false positive rate to roughly 0.15 to 0.20, because the running test statistic takes a random walk that will eventually cross the boundary under the null.
Price the winner's curse separately, because it is a different error. Conditioning an estimate on having crossed the threshold biases it away from zero, by a factor of roughly 3 when true power is 20% and only about 1.1 at 80% power. Day four sits at roughly 29% of planned information, which is exactly the low-power case.
Read day twelve as the better estimate rather than as a contradiction. Nothing regressed; the day-four value was an extreme draw that was selected precisely for being extreme.
Price the segment claim. Across twenty segments at a nominal 5%, at least one crosses about 64% of the time under a global null, since 1 - 0.95^20 = 0.64, and the slice that surfaces is by construction the one with the most favourable noise.
Specify the rule instead of scolding. Either fix the horizon and look once, or use group-sequential boundaries such as O'Brien-Fleming that spend almost no alpha early, or use always-valid confidence sequences that stay correct under continuous monitoring. Pre-register the small set of segments, correct across the ones tested, and treat anything found afterwards as a hypothesis needing its own powered test.

Worked solution (25 min)

Count the looks actually taken, twelve daily, and state the inflated alpha band of 0.15 to 0.20 for ten looks, rising with more.
Place day four on the information fraction, about 4/14 = 29% of planned sample, and apply the exaggeration factor for low power.
Compute 1 - 0.95^k for the number of segments actually inspected; with k = 20 this is 0.64.
Report the day-twelve interval as the estimate and say plainly that the experiment supports no effect at any size the team would ship for.
Write the rule into the experiment template: horizon and segments declared at launch, sequential boundaries if interim looks are needed at all.

Expected result: The day-four result is consistent with pure noise: ten looks inflate the false positive rate to roughly 0.15 to 0.20, and an estimate conditioned on crossing at about 29% information is exaggerated by roughly a factor of three at low power. The segment finding appears about 64% of the time across twenty slices under a global null. The supportable conclusion is a 0.2% lift whose interval covers zero.

Verify it yourself

1 - 0.95^20 = 0.6415. If your family-wise number lands near 0.05 you have computed the per-test rate instead.
The day-twelve standard error should be roughly sqrt(12/4) = 1.7 times smaller than day four's. If the interval did not narrow, sample was not accumulating and there is a second problem.
A Bonferroni threshold for twenty segments is 0.0025; check whether the mobile_web result would have survived it.

Follow-up

Under O'Brien-Fleming with five planned looks, roughly what nominal p-value would the first look have needed to stop?
The feature already shipped. What analysis can you still run, and what can it not tell you?
If mobile_web paid_search were genuinely the only affected segment, how would you design the confirmatory test and how large would it have to be?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

Repeat the depth protocol on the second-ranked area with the same six-problem structure.
Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

Re-run the six prompts from day one under the same clock and compare both correctness and time.
Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

Quantify your own impact without claiming the topline you touched

Hard · self-assessment · attribution · communication

You are writing the impact section of your own review. Over the year you ran four experiments, one of which shipped and three of which were flat; you corrected the definition of gross monthly revenue churn so that cancellation is recognised at period_end_utc; and you built a self-serve funnel dashboard. Weekly active accounts rose 14% over the same period. Your reviewer knows the data well. Write the three impact claims you would defend, stating for each what you contributed, what evidence supports it, and what portion of the outcome you are not claiming.

Approach

Recognise what is being probed: whether you apply to your own work the causal standard you would apply to somebody else's roadmap claim. Nearly everyone who would reject 'accounts that do Y retain better' will write 'I drove a 14% increase' without noticing it is the same error with a friendlier subject.
Sort the work by the kind of evidence it can carry. The shipped experiment is the only item with a randomised estimate, so it is the only one where an effect size is defensible, and you claim the interval rather than the point estimate.
Claim the three flat experiments as decisions prevented and price them. Features not built, or built differently, on evidence, with the engineering weeks reallocated as the number somebody else can verify. A defensible null is a delivered decision and should be written as one.
Claim the definition fix as correctness, not as improvement. The old figure was overstated by a specific percentage and appeared in a specific set of recurring documents; the impact is the change it produced in the forecast built on top of it, not a change in churn itself.
Claim the dashboard on usage and displacement: distinct weekly users of it, and the ad-hoc request count for six months before against six months after. If the request log does not exist, record the claim as unverified rather than estimating it upward.
Disclaim the 14% explicitly and once. State that it cannot be separated from seasonality, other teams' launches and a pricing change, and bound your own contribution from above using the shipped experiment's interval converted into headline units.

Follow-up

Your shipped experiment's interval was +0.2pp to +1.4pp on activation. How much of the 14% can that account for, and how do you say so without undercutting yourself?
A peer in the same cycle claims the full 14%. What, if anything, do you do about it?
If you could only keep two of your three claims, which do you drop, and why that one?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

The executive says to give the midpoint and they will manage the risk. What do you do?
How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?

Choose between three teams' requests with one analyst-week

Medium · prioritisation · stakeholder · judgement

You have one analyst-week. Three requests land the same morning. A growth team wants a paid-channel readout before a Friday spend decision. A billing team wants gross monthly revenue churn rebuilt, because the current figure recognises cancellation at canceled_at_utc rather than period_end_utc and is therefore wrong. A product team wants a dashboard for a feature launching in six weeks. All three sponsors are peers of your manager. Deliver your ranking, the explicit rule that produced it, and the message you send to the two teams you defer.

Approach

Recognise what is being probed: whether you prioritise on decision value and reversibility or on who asked most recently and most loudly. The generic answer sorts by importance; the strong one states a rule, applies it, and accepts the ranking it produces even where that is uncomfortable.
Score each request on three statable things: the decision it unblocks and the date that decision is made, the cost of being wrong in the meantime, and whether the work is one-off or compounding. A wrong published churn figure compounds, because it is quoted downstream and enters forecasts; the channel readout has a fixed date that cannot move; the dashboard has six weeks of slack.
Notice the tension between value and urgency rather than resolving it by feel. The churn defect is the most valuable item and the least urgent one, which is exactly the shape of work that never gets done.
Break the churn item in two. A one-hour severity check, sizing the gap between the two recognition points in MRR, is cheap enough to do before ranking anything and may promote the item outright. Do that first, then rank.
Make the deferrals concrete. Each deferred team gets a date, a reason expressed as another team's decision deadline rather than as relative importance, and the smallest thing you can hand them immediately.

Follow-up

The dashboard team escalates to your manager. What do you say in that conversation?
Your severity check shows churn is overstated by 15%. Does the ranking change, and does anybody need to be told today regardless of the ranking?
A fourth request arrives Wednesday with a Thursday deadline. What comes off the list, and who do you tell first?