Impact Analytics Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at Impact Analytics, you are at the core of driving data-informed decisions for high-stakes business environments. This role is not merely about building models; it is about bridging the gap between complex statistical theory and actionable business strategy. You will work on products and solutions that tackle real-world challenges, requiring a blend of technical rigor and a sharp, intuitive grasp of how data influences business outcomes.

A large share of questions open as "how would you measure X", where the real work is choosing the metric, fixing its denominator, and defining the population it applies to. Any computation comes last and is frequently not required at all.

Focus areas

• Power experiments for heavy-tailed account revenue
• Separate contracted seats from actively used seats
• Strip CI, retry and synthetic traffic first

Overview

Your work will directly impact how the organization approaches problem-solving at scale. Whether you are optimizing supply chains, refining retail analytics, or developing predictive engines, your contributions will be central to the company’s value proposition. Candidates report that the interviews favor people who are excited by ambiguity and can translate vague business requirements into robust, scalable technical solutions.

Impact Analytics values candidates who can explain the 'why' behind their models. Be prepared to discuss the business trade-offs of your technical decisions.

The Impact Analytics interview process

Impact Analytics candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Technical Discussions

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Leadership Interviews

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

• Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
• Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
• How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

• Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
• Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
• Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Where candidates lose points

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Reporting a mean over accounts when account revenue is heavy-tailed

When a small number of accounts hold most of the revenue, the sample mean is dominated by whichever of them happens to be in the sample, and the sample variance keeps growing as more data arrives instead of stabilising. In that regime the usual central-limit-based confidence interval understates uncertainty, and a single renewal or a single large account's batch job can flip the sign of a measured effect. The fixes are to pre-register a winsorisation or capping rule before looking at the outcome, to report account counts crossing a threshold alongside the revenue figure, or to define the estimand on a bounded transform. Choosing the cap after seeing the result is a separate and worse problem, because the cap then encodes the answer.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Questions & practice

Describe the difference between bagging and boosting techniques.

Medium · statistics and probability

Approach

1. Sanity-check the answer against a simple bound or a simulated case.
2. Write down the assumption the method needs before you use the method.
3. Translate the result into the decision it informs, in one plain sentence.

Follow-up

• Which assumption here is most likely to be violated in practice?
• What sample size would you need to detect an effect half this size?

Given a dataset, how would you perform data cleaning and feature engineering using Pandas?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• How would you choose the decision threshold, and who owns that choice?
• What would you monitor after launch to know the model is still valid?

How would you implement a machine learning pipeline in Python without relying exclusively on pre-built libraries?

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

Implement seven-day activation rate from its written definition

Easy · metrics · pandas · activation

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

Approach

1. Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
2. Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
3. Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
4. Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
5. Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Worked solution (20 min)

1. cohort = accounts.loc[~accounts.is_internal & accounts.created_at.between(week_start, week_start + pd.Timedelta(days=7), inclusive='left'), ['account_id','created_at']].drop_duplicates('account_id')
2. ok = requests[(requests.http_status < 400) & requests.api_key_id.notna() & (requests.traffic_class != 'synthetic_monitor')]
3. first_ok = ok.groupby('account_id', as_index=False).request_at.min(); m = cohort.merge(first_ok, on='account_id', how='left'); assert len(m) == len(cohort)
4. activated = (m.request_at - m.created_at) <= pd.Timedelta(hours=168); return {'rate': float(activated.mean()), 'numerator': int(activated.sum()), 'denominator': len(m)}

Expected result: A dict whose denominator is the count of distinct non-internal accounts created in that ISO week, numerator <= denominator, and rate = numerator / denominator; accounts with no qualifying request sit in the denominator with NaT and evaluate to False.

Verify it yourself

• len(m) == cohort.account_id.nunique(); the merge must not add rows.
• Dropping the synthetic_monitor filter can only raise the numerator, never lower it.
• Widening the window from 168h to 336h is monotone: the numerator must not decrease.
• An account whose only qualifying request falls at exactly created_at + 168h counts as activated, since the comparison is inclusive.

Follow-up

• Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
• How would you decide whether 168 hours is the right window rather than 72 or 336?
• An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Describe the difference between INNER JOIN and LEFT JOIN in a practical scenario.

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Say which table is the grain you start from, and join outward from it.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

If you have 15 items and can only carry 5, what criteria do you use to select them?

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Say which table is the grain you start from, and join outward from it.
3. State the window function and its partition and ordering out loud before writing it.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

Write a SQL query using window functions to identify top-performing segments.

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. Check whether any join is one-to-many before aggregating, or the sums inflate.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Find paid seats that never called the API

Easy · anti-join · null semantics · seat utilisation

dim_user_membership carries user_id, account_id, seat_type, is_service_account, deactivated_at and last_seen_at. fct_api_request carries user_id, which is null whenever the caller is a service account or an unattended key, plus account_id, request_at and http_status. For a single account, list every licensed_paid seat with is_service_account = false and deactivated_at null that issued no request at all in the trailing 90 days, returning user_id and last_seen_at. A first draft writes NOT IN against a subquery over fct_api_request.user_id. State exactly what that draft returns and why, then write the correct query.

Approach

1. Read the column comment first: user_id is nullable on the fact, so the subquery almost certainly contains at least one NULL for any account that runs a service account or an unattended key.
2. Work the three-valued logic out loud. x NOT IN (a, NULL) expands to NOT (x = a OR x = NULL); the second disjunct is UNKNOWN, so for any x not equal to a the whole predicate is UNKNOWN and the row is filtered out. The draft returns zero rows, which reads as full seat utilisation.
3. Rewrite as NOT EXISTS with the 90-day and status predicates inside the correlated subquery. Putting them in the outer WHERE instead turns the anti-join into a different question and silently changes the answer.
4. Correlate on both user_id and account_id. A membership is (user_id, account_id) and one person can hold memberships in several accounts, so correlating on user_id alone marks a seat as active because that human was busy somewhere else.
5. Filter the seat side to seat_type = 'licensed_paid', is_service_account = false and deactivated_at IS NULL, and say what the resulting count means next to contracted_seats on the current subscription row.

Worked solution (15 min)

1. Confirm the hazard with one query: SELECT count(*) FROM fct_api_request WHERE user_id IS NULL AND account_id = :account_id. Any non-zero result proves the draft returns nothing.
2. Write the seat side: SELECT user_id, last_seen_at FROM dim_user_membership WHERE account_id = :account_id AND seat_type = 'licensed_paid' AND NOT is_service_account AND deactivated_at IS NULL.
3. Attach AND NOT EXISTS (SELECT 1 FROM fct_api_request r WHERE r.user_id = m.user_id AND r.account_id = m.account_id AND r.request_at >= now() - interval '90 days').
4. Compare the row count against the same query written as a LEFT JOIN with a WHERE r.user_id IS NULL; the two must agree exactly.
5. Divide the active seat count by contracted_seats from the account's current fct_subscription_period row and state the utilisation figure.

Expected result: A list of user_id and last_seen_at for licensed, non-service, active memberships on that account with no request in 90 days. The NOT IN draft returns zero rows whenever the account has any service-account or unattended-key traffic, and returns the right answer only by accident on accounts that have none.

Verify it yourself

• The NOT EXISTS result and the LEFT JOIN ... IS NULL result return identical user_id sets.
• Idle seats plus seats with at least one request in 90 days equals the total licensed non-service active membership count for the account.
• Forcing a NULL into the subquery (union a SELECT NULL) leaves NOT EXISTS unchanged and empties the NOT IN version, which demonstrates the failure on demand.

Follow-up

• Adding AND user_id IS NOT NULL to the subquery also fixes NOT IN. Why is NOT EXISTS still the form you would leave in the repository?
• last_seen_at looks like a shortcut for the whole query. What does it actually record, and where does it disagree with API activity?
• This is a seat-reduction risk list. What threshold would you attach before handing it to an account team, and what happens to seats below it?

You are tasked with deciding which metrics to prioritize for a new client project; how do you approach this?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Estimate the number of units a retail chain might sell during a holiday season.

Medium

Approach

1. State your assumptions explicitly before working the problem.
2. Work from the decision backwards to the evidence you would need.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Measure named coverage at an assignment threshold you cannot randomise

Hard · regression-discontinuity · instrumental-variables · selection-bias

Named customer-success coverage is assigned to accounts whose arr_cents at fiscal-year start is at least $50,000. About 88% of accounts above the line receive a named owner and about 9% below it do, through documented exceptions. Roughly 200 accounts sit within $15,000 of the cutoff, and gross logo retention on the renewal-eligible base runs near 0.88. Estimate the effect of coverage on renewal, name the design, state its identifying assumptions and the test for each, and say whether this data can answer the question at all.

Approach

1. Say first why the naive comparison is unusable, because the answer turns on it. Coverage is assigned to the largest accounts because they are valuable and to distressed accounts because they are at risk, so selection runs in both directions at once and the naive estimate can come out with either sign depending on which rule dominated. Matching on observed size does not fix it: the risk signal that triggered coverage is the same signal that predicts renewal.
2. Name the design and the estimand together. This is a fuzzy regression discontinuity with the $50,000 threshold as an instrument for coverage. The estimator is the Wald ratio, the jump in renewal at the cutoff divided by the jump in coverage probability of 0.88 - 0.09 = 0.79, and the estimand is a local average treatment effect for accounts near $50,000 whose coverage is actually decided by the rule, not the average effect of coverage across the book.
3. Fit it as local linear regression on each side with a triangular kernel and a data-driven bandwidth (Imbens-Kalyanaraman, or Calonico-Cattaneo-Titiunik with bias-corrected robust intervals). A global polynomial fitted across the whole ARR range is not a substitute and is well documented to manufacture spurious jumps at interior points.
4. Test the assumptions instead of asserting them. Run a density test for bunching just above $50,000, which is exactly what a sales team structuring a deal at $50,001 to win coverage would produce and which invalidates the design outright. Check continuity at the cutoff of employee_band, acquisition_channel, deployment_model and pre-period usage. Run placebo cutoffs at other ARR values, and plot the estimate against bandwidth.
5. Do the power arithmetic before promising an answer. With about 100 accounts per side and a renewal base rate of 0.88 (standard deviation 0.325), the sharp-design MDE at 80% power and two-sided 0.05 is 2.80 * 0.325 * sqrt(2/100) = about 12.9 percentage points, and dividing by the 0.79 first stage gives about 16 points on the fuzzy estimand. No plausible effect of assigning an account manager is that large.
6. Give the honest verdict with a route forward attached. This design cannot answer the question on one fiscal year. Pool several years at the same cutoff, accepting the added assumption that the coverage model and the threshold's meaning were stable across them, or ask for a randomised holdout among accounts near the threshold, which is cheap to justify precisely because the rule is already arbitrary there.

Worked solution (45 min)

1. Plot the density of arr_cents in bins around $50,000 and run the manipulation test; stop and report the design as invalid if bunching is present.
2. Plot the first stage, share of accounts with a named owner against ARR, and confirm the jump is close to 0.79 and locally flat on each side.
3. Fit local linear regressions of renewal on ARR either side of the cutoff with a triangular kernel and data-driven bandwidth, take the Wald ratio, and report bias-corrected robust intervals.
4. Run covariate-continuity checks, placebo cutoffs, and a bandwidth-sensitivity curve for the point estimate.
5. Compute the MDE at the realised effective sample and write the verdict, powered or not, with the alternative design named.

Expected result: A fuzzy RD local average treatment effect at the $50,000 cutoff with a first stage of about 0.79, and an MDE of roughly 16 percentage points on the fuzzy estimand at about 100 accounts per side. The honest finding is that the design is underpowered by an order of magnitude relative to any credible coverage effect, so the recommendation is pooled fiscal years or a randomised holdout at the margin.

Verify it yourself

• The first-stage jump and the outcome jump must be estimated on the same bandwidth. A Wald ratio assembled from two different bandwidths is not the regression discontinuity estimator.
• Covariates that coverage could not have affected must be continuous at the cutoff. A jump in employee_band or acquisition_channel at $50,000 indicates either manipulation of the running variable or a second rule sharing the same threshold.
• Point estimates should vary smoothly with bandwidth. An estimate that flips sign between a $10,000 and a $20,000 bandwidth is a fitting artefact rather than a finding.

Follow-up

• The density test shows clear bunching just above $50,000. Is any part of the design salvageable, and what would you report instead?
• Coverage also correlates with support ticket severity. Does adding severity as a control help, hurt, or do nothing in a regression discontinuity?
• If you pool five fiscal years to gain power, what new identifying assumption have you taken on, and how would you check it?

Consumption per account always dips in the last four days

Easy · pipeline lag · restatement · reporting windows

Your consumption dashboard reads billable units per paying account from fct_usage_daily (account_id, sku_code, usage_date, billable_quantity, net_amount_cents, is_restated, first_written_at, restated_at, updated_at). Every refresh shows the final three to five days declining, and a review is scheduled on that shape. Establish empirically how long a usage_date takes to settle, separately per sku_code, and specify the trailing exclusion window every reported figure should use. Derive the number from the data; do not adopt a convention.

Approach

1. Treat settlement as a measurable curve rather than a belief: for each historical usage_date, compare the total captured as of age d days against that date's final settled total.
2. Do it per sku_code, because metering paths differ. A per-request SKU lands within hours, while a storage-month SKU is produced by a daily sweep and lands later, so one global lag number is wrong for at least one of them.
3. Pick the age at which a stated percentile of dates reaches a stated completeness threshold, for example the tenth percentile date reaching 99.5 percent of final, and take the slowest SKU's age as the dashboard's exclusion window.
4. Separate late arrival from restatement. Late rows raise totals and are fixed by waiting; restatements can move either way and are not, so report the share of the gap from each.
5. Encode the exclusion inside the query rather than in the chart, so anyone reusing the SQL inherits the rule, and never compare a fresh partial period against a settled one.

Follow-up

• Month-end invoicing needs a number before settlement completes. How would you publish an early estimate with an honest uncertainty band attached?
• What monitor would tell you the settling time has changed, without anyone remembering to re-run this analysis?

Suggested practice checklist

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.

Ambiguous requests are the test

Most of the questions in this section reduce to one thing: can you be handed a vague request and come back with something useful? Prepare an example where the ask was underspecified, you chose an interpretation, and you said out loud which interpretation you chose. Describing how you narrowed the question matters more than the technique you eventually used.

How would you explain a complex model's output to a non-technical stakeholder?

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Close with what you would do differently, concretely.
3. Pick a story where you drove the decision, not one where you observed it.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that project again?

Scope an open-ended request to predict account churn

Medium · scoping · churn modelling · operating point

A customer success director asks for a list of accounts about to churn. You know only that the team has six people and that contracts are annual. Available data is fct_subscription_period, fct_usage_daily, fct_api_request, fct_support_ticket and dim_account. Before writing any code, produce the questions you need answered, a proposed definition of about to churn, and the shape of the artefact you would hand back, including the operating point that turns a score into a decision.

Approach

1. The interviewer is probing whether you convert a vague request into a decision with a capacity constraint attached. A candidate who starts talking about model families has already failed the exercise.
2. Pin the event and the horizon first. Churn is only possible at term_end_date, so the population is accounts renewing in the next 60 to 90 days, not the whole base. Ask explicitly whether contraction and downgrade count as churn or only full non-renewal, because the three have different base rates and different interventions.
3. Pin the action and the capacity. Six people times a realistic number of meaningful interventions per week gives k, and k is what the list is ranked to. Evaluate on precision at k rather than a global AUC over accounts that will never be contacted.
4. Audit leakage before choosing features. Every feature needs a timestamp proving it existed before the prediction date. A downgrade amendment, a churn reason code, and a ticket opened after the renewal conversation started are all leaks that will make the offline number look excellent and the live list useless.
5. Ask for the counterfactual now rather than later. Coverage is assigned deliberately, so without a held-out slice agreed at the start the intervention can never be evaluated, and you will be asked for its impact in nine months regardless.
6. Propose the smallest artefact that closes the loop: a weekly ranked list sized to capacity with two or three inspectable reasons per row, plus a stated policy for accounts below the line.

Follow-up

• The director insists all accounts are in scope, not only those renewing soon. How do you answer without simply refusing?
• Historical non-renewals number about 30 a year. At what point do you tell them a model is the wrong tool and a rules list is better?
• Which candidate features would you drop purely because you cannot date them?

Allocate one analyst week across three competing requests

Easy · prioritisation · stakeholders · scoping

Three requests arrive the same morning and you have one week. Finance wants per-account gross margin from fct_usage_daily for a pricing review in three weeks. Sales wants a renewal-risk list for accounts with term_end_date inside 60 days. A product manager wants an experiment readout for a decision being taken on Thursday. Produce your allocation with hours attached, what you say to whoever receives less, and one thing you refuse to do this week, with the reason each decision is defensible to the person it costs.

Approach

1. The interviewer is probing whether you prioritise on decision timing and reversibility or on who asked most forcefully. Sort by the date each decision is actually taken and by what the default outcome is if nothing arrives.
2. Apply that sort concretely. The Thursday readout has a hard irreversible deadline and no value afterwards. The pricing review has three weeks of slack. The renewal list has a rolling deadline set by term_end_date, so part of it is urgent this week and the rest is not, which means it can be split rather than deferred whole.
3. Find the cheapest sufficient version of each request rather than the full version. The readout goes in full. The renewal list ships as a filtered query over renewal-eligible accounts ranked by two inspectable signals rather than as a model. The margin work is scoped to the accounts that dominate the pricing decision, since revenue is heavily skewed and the tail will not change the conclusion.
4. Make the trade visible in one written note to all three at once, with dates. Telling each person separately that they are the priority is how an allocation becomes a credibility problem.
5. Refuse something explicitly and say why. The model version of the renewal list is the usual candidate, because it cannot be evaluated without a holdout nobody has agreed to yet, and building it this week forecloses that.
6. Leave slack. A plan with none is a plan to miss the one deadline that cannot move.

Follow-up

• The sales leader escalates to your manager. What did you already do that makes that a short conversation?
• Which of the three deadlines would you push back on, and what exactly would you ask for?
• What would you change about how these requests reach you so next week is not the same?