Dealerware Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at Dealerware, your role is pivotal in transforming data into actionable insights that drive strategic decisions. This position is essential for enhancing product offerings and improving user experience within the automotive finance sector. By leveraging advanced analytics, statistical modeling, and machine learning, you will influence key areas such as pricing strategies, customer segmentation, and operational efficiencies.

How much statistics you need depends on the flavour of the seat. Experiment-facing work wants you deep enough to notice that repeated looks at accumulating data inflate the false positive rate of a fixed-sample test; modelling-facing work wants estimation and honest uncertainty intervals.

Focus areas

• Separate novelty effects from durable behaviour change
• Pick a randomisation unit that respects interference
• Define numerator, denominator and window precisely

Overview

You will collaborate with cross-functional teams, including engineering, product management, and marketing, to tackle complex challenges that directly impact the business and its users. The complexity of the data you will work with, paired with the scale at which Dealerware operates, provides an exciting opportunity to contribute to innovative solutions in a fast-paced environment. Expect to work on projects that not only refine existing products but also lay the groundwork for future advancements in the automotive industry.

The Dealerware interview process

Dealerware candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

• Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
• Whether you name what you have not done instead of stretching to cover every line of the posting
• Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

• Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
• Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
• Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Technical Interviews

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Management Interview

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

• Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
• The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
• Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

• Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
• Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
• Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Slicing a flat experiment until a segment reaches significance

Testing one metric across twenty segments at a nominal 5% level produces a significant result about two thirds of the time when nothing is happening anywhere, and the segment that surfaces is by construction the one with the most favourable noise. The reported effect in that slice is then badly overstated, because selection on significance conditions the estimate on being large. What makes it dangerous rather than merely wrong is that a post-hoc segment always has a plausible story attached, so it survives the meeting. The controls are declaring the small number of segments of interest before launch, correcting across the ones tested, and treating anything discovered afterwards as a hypothesis that needs its own adequately-powered test rather than a finding.

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Reaching for a model before the target metric exists

Before naming an algorithm, write down the label, the prediction time, and the action that changes when the score crosses a threshold. If you cannot say what decision the output drives, any modelling choice is guesswork dressed up as method.

Questions & practice

Write a function to calculate the mean and standard deviation of a list.

Medium · statistics and probability

Approach

1. Quantify uncertainty explicitly rather than reporting a point estimate alone.
2. Say what the estimate is of, and over what population it generalises.
3. Sanity-check the answer against a simple bound or a simulated case.

Follow-up

• What sample size would you need to detect an effect half this size?
• Which assumption here is most likely to be violated in practice?

How would you implement a decision tree algorithm from scratch?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Say how the offline result would be validated online before it is trusted.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• What would you monitor after launch to know the model is still valid?
• Where could label leakage enter this setup?

Rebuild per-visitor ordering without groupby convenience methods

Easy · pandas · vectorisation · window logic

You have a DataFrame of 2 million fct_event rows with visitor_id, occurred_at_utc and event_id, unsorted and containing duplicate timestamps within a visitor. Produce three new columns: event_rank, the 1-based position of the event within its visitor ordered by occurred_at_utc; seconds_since_prev, the gap to that visitor's previous event, NULL for the first; and is_first_for_visitor. You may use sort_values, shift, cumsum, numpy and boolean masking. You may not use groupby.transform, groupby.apply, groupby.cumcount, groupby.rank or merge_asof. Break timestamp ties on event_id.

Approach

1. Sort once by ['visitor_id', 'occurred_at_utc', 'event_id'] and reset the index. The whole exercise reduces to row arithmetic on a sorted frame, and the tiebreak on event_id is what makes the result reproducible across runs.
2. Mark visitor boundaries with is_first = df['visitor_id'].ne(df['visitor_id'].shift()). This is the single fact every other column derives from.
3. Compute seconds_since_prev as the diff of the timestamp column, then overwrite it with NaT/NaN wherever is_first is True. The shift crosses the boundary between visitors and will otherwise hand the first row of each visitor the last event of the previous one.
4. Build event_rank from a running counter that resets at boundaries: take a global cumulative position (np.arange(len(df))) and subtract, per row, the global position at which that visitor started. Get the start position by forward-filling the positions where is_first is True, which is a cumsum-free reset and is O(n).
5. Verify against the forbidden method once, as a test rather than as the implementation, and confirm the two agree on every row.

Worked solution (20 min)

1. Sort on the three-key tuple and reset_index(drop=True).
2. Compute is_first via .ne(.shift()), which is True for row 0 because the shifted value is NaN.
3. pos = np.arange(len(df)); start = pd.Series(np.where(is_first, pos, np.nan)).ffill(); event_rank = (pos - start + 1).astype(int).
4. gap = df['occurred_at_utc'].diff().dt.total_seconds(); gap[is_first] = np.nan.
5. Assert event_rank equals df.groupby('visitor_id').cumcount() + 1 on the sorted frame.

Expected result: Three columns on the sorted frame: event_rank starting at 1 for every visitor and increasing by 1 with no gaps, seconds_since_prev null exactly where is_first_for_visitor is True, and is_first_for_visitor summing to df['visitor_id'].nunique().

Verify it yourself

• is_first_for_visitor.sum() equals the number of distinct visitor_id values.
• seconds_since_prev.isna().sum() equals that same number.
• event_rank.max() per visitor equals that visitor's row count, and event_rank is non-negative everywhere.
• A visitor with two events at the identical timestamp gets ranks 1 and 2 with a gap of 0.0, ordered by event_id.

Follow-up

• The frame does not fit in memory. How does your approach change if you can only process one visitor-partitioned chunk at a time?
• occurred_at_utc is client-supplied and sometimes runs backwards within a visitor. Does your seconds_since_prev go negative, and should it?
• How would you extend this to reset the counter at every change of surface as well as visitor?

Discuss the trade-offs between a SQL and NoSQL database for a specific use case.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. State the window function and its partition and ordering out loud before writing it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Paying accounts with no active seat in 28 days

Medium · anti-join · null-semantics · retention-risk

dim_account holds account_id, account_type, lifecycle_status, seats_licensed. fct_event holds account_id, user_id, occurred_at_utc, is_core_action, and its account_id is NULL for every signed-out and pre-signup event. Find accounts with lifecycle_status = 'active' and account_type <> 'internal' that had no distinct user complete a core action in the trailing 28 days. Return account_id, seats_licensed and days since that account's most recent core action, with NULL where the account has never emitted one. Order by seats_licensed descending.

Approach

1. Build the recent-activity set first: fct_event rows with is_core_action = TRUE, occurred_at_utc >= now() - interval '28 days', and an explicit account_id IS NOT NULL. Making the NULL exclusion explicit in the CTE is what lets you reason about the anti-join afterwards.
2. Express the exclusion with NOT EXISTS (correlated on account_id) or a LEFT JOIN with an IS NULL guard. Do not use NOT IN against this column: it is nullable, and SQL's three-valued logic turns the whole predicate UNKNOWN, returning zero rows.
3. Compute last-seen separately as MAX(occurred_at_utc) per account over all history, LEFT JOINed on, so an account that has never emitted a core action (NULL) is distinguishable from one that went quiet six weeks ago. Those two cases have different causes and different owners.
4. Rank by seats_licensed, or better by the account's current mrr_cents_constant_fx if you are allowed the subscription table, because a silent fifty-seat account is a renewal conversation and a silent one-seat account is noise.
5. Before shipping, check whether the never-seen group is a cluster by signup date or surface. A block of accounts with no events at all is usually an instrumentation gap, not a set of customers who stopped using the product.

Worked solution (25 min)

1. Count NULL account_id rows in fct_event over the window so you know the trap is live in this data rather than hypothetical.
2. Write the active-account spine and the 28-day activity CTE.
3. Write the anti-join with NOT EXISTS, then deliberately run the NOT IN version and record that it returns zero rows.
4. Add the all-time MAX(occurred_at_utc) LEFT JOIN and derive days_since as a date difference.
5. Split the output into 'quiet' and 'never seen' and eyeball the never-seen group for a shared signup window or surface.

Expected result: One row per silent active account with seats_licensed and days_since_last_core_action, ordered by seats_licensed descending. Accounts that never emitted a core action show NULL and are reported as a separate group rather than as the worst churn risks.

Verify it yourself

• Silent accounts plus accounts with at least one active user equals the total count of active non-internal accounts.
• The NOT IN variant returns zero rows, which is the demonstration that a NULL is present.
• No account_id appears twice in the output.
• Every days_since_last_core_action is either NULL or greater than 28.

Follow-up

• How would you distinguish a genuinely idle account from one whose events lost their account_id after an instrumentation change?
• Would you count on fct_event.account_id or resolve user_id through dim_user instead, and what does each choice miss?
• Licensed-seat utilisation is the continuous version of this. How would you turn this boolean into that ratio?

Suppose you are given a dataset with thousands of entries; how would you identify trends?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Fix the population and the time window before naming any metric.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How would you approach a project that involves predicting customer churn?

Medium · product sense and metrics

Approach

1. Fix the population and the time window before naming any metric.
2. Decompose the metric into the rates that drive it, and say which one you would check first.
3. Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

Provide an example of a successful project where you used data to influence business decisions.

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Fix the population and the time window before naming any metric.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• Which segment would you cut first, and what would that rule out?
• How would you detect that the metric is being gamed rather than genuinely improving?

Describe your process for tackling a complex data problem.

Medium · product sense and metrics

Approach

1. State what result would change your recommendation, so the answer is falsifiable.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• Which segment would you cut first, and what would that rule out?

Can you explain A/B testing and its significance in analytics?

Medium · a/b testing and experimentation

Approach

1. State the primary metric and the minimum effect worth shipping, then size the test.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Say whether units interfere with each other, and switch design if they do.

Follow-up

• What would you do if you could not randomise at all?
• What would you conclude if the result is positive but the test is underpowered?

What methods would you use to handle missing data?

Medium

Approach

1. State your assumptions explicitly before working the problem.
2. Say what you would check first and why it is the highest-information step.
3. Work from the decision backwards to the evidence you would need.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Randomise a shared workspace feature without contaminating control

Medium · randomisation unit · interference · design effect · clustering

A feature changes a collaborative surface inside a workspace: when one member uses it, other members of the same account see the result in their own view. You have dim_user (user_id, account_id, is_internal), dim_account (account_id, seats_assigned, lifecycle_status) and fct_event. Among active accounts the mean seats_assigned is 6, the coefficient of variation of that count is 1.5, and the intraclass correlation of the weekly core-action rate within an account is 0.10. Choose the randomisation unit, quantify what that choice costs in sample, and specify how you would compute inference.

Approach

1. State the interference before choosing anything: a treated user changes what an untreated colleague sees, so user-level randomisation puts both arms inside one account and biases the contrast toward zero. Randomise on account_id.
2. Price the clustering properly. With equal clusters the design effect is 1 + (m - 1) rho = 1 + 5(0.10) = 1.5. Sizes here are far from equal, so use 1 + ((CV^2 + 1) m - 1) rho = 1 + (3.25 x 6 - 1)(0.10) = 2.85. The equal-size shortcut understates the cost by nearly half.
3. Decide the estimand before the estimator. An account-weighted mean gives every workspace one vote; a user-weighted mean lets the largest workspaces dominate. With this size skew the two can move in opposite directions, so pick the one the decision needs and write it down.
4. Compute standard errors on the account, not the user: cluster-robust on account_id, or collapse each account to a single number and test those. Below roughly 40 clusters per arm, cluster-robust errors are biased downward, so use a wild cluster bootstrap or randomisation inference over the assignment.
5. Buy back variance where you can. Stratify assignment by seat band and lifecycle_status before randomising, and decide in advance how the handful of very large accounts are handled, since one enterprise workspace can carry more users than a hundred single-seat ones.

Worked solution (30 min)

1. Write the interference down: the outcome for user i depends on the treatment of other users in account(i), so the no-interference assumption fails at the user level and holds at the account level.
2. Compute both design effects, 1.5 equal-size and 2.85 unequal-size, and use 2.85.
3. Take the user-level sample requirement from the proportion shortcut, multiply by 2.85, then divide by the mean of 6 users per account to express it in accounts per arm.
4. Specify the analysis: collapse to one row per account, regress the account-level outcome on variant with stratum fixed effects, and use a wild cluster bootstrap for inference.
5. State the stopping rule up front: if the required account count exceeds the eligible population, the test is not runnable, and the alternatives are a longer window, a larger target effect, or a non-experimental read.

Expected result: Randomise on account_id. The unequal-size design effect is 2.85, so a user-level requirement of 12,000 per arm becomes 34,200 user-equivalents, which is 5,700 accounts per arm, with inference computed on accounts rather than users.

Verify it yourself

• The design effect must be at least 1, and must equal 1 when rho = 0 or when mean cluster size is 1. A value below 1 means a term is inverted.
• Recomputing the unequal-size formula with CV = 0 must return the equal-size value of 1.5.
• The cluster-robust standard error on account_id should come out about sqrt(2.85) = 1.69 times the naive user-level one. A ratio near 1 means the clustering was not applied.

Follow-up

• Suppose the feature is not workspace-scoped but changes a globally shared ranking model, so no clean cluster exists. What design gets you a causal read, and what does it cost you?
• You have 900 eligible active accounts in total. Given the design effect, what absolute lift can this test detect, and is the honest answer 'do not run it'?
• The intraclass correlation is an estimate from last quarter. What happens to your sizing if the true value is 0.25?

Decide whether a one-day core-action drop is real

Easy · seasonality · day-of-week · late-arriving-data

A daily dashboard counts distinct fct_event.user_id with is_core_action = TRUE, filtered on occurred_at_utc, and is read at 09:00 UTC. This morning it shows yesterday down 22% against the day before. fct_event is partitioned on received_at_utc. You have fct_event, fct_session and dim_user with thirteen months of history. Deliver a one-paragraph verdict, escalate or do not escalate, with the evidence that settles it, before anyone proposes a product hypothesis.

Approach

1. Identify which two weekdays the comparison actually spans, then pull the same weekday-pair transition for the last 52 weeks and place the observed 22% inside that distribution. A day-over-day comparison in a product with a weekday pattern is a comparison of two different populations, so the reference class is the same transition historically, not the prior day.
2. Measure partition completeness rather than assuming it. For each of the last 30 days compute the share of that day's occurred_at_utc rows that had landed by 09:00 UTC the following morning, split by surface; mobile clients buffer events offline, so the freshest partition is systematically short and the shortfall is not uniform across surfaces.
3. Recompute the same series keyed on received_at_utc. If the drop survives on both keys it is not a lateness artefact; if it exists only on occurred_at_utc it is the partition filling in.
4. Check the two exclusion flags before segmenting anything: a change in is_bot_flagged coverage or a batch of is_internal accounts entering or leaving moves a distinct-user count with no user behaviour behind it.
5. Only if the movement survives all of the above, begin the segment decomposition. Say explicitly in the verdict which of these four checks the movement passed, so the next reader does not repeat them.

Follow-up

• What publication lag would you set for this dashboard, and how would you derive the number rather than pick it?
• If you switch the metric to received_at_utc, what does that break for anyone comparing to historical figures?
• How would you detect the same problem automatically, so a human does not have to notice it each morning?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Your part, not the team's

Most data work is done by groups, so an interviewer has to work out which piece was yours. An answer that runs on 'we' for several minutes gets interrupted with a question about what you personally did, and by then the answer sounds defensive even when it is true. Mark your own contribution as you go, and name the parts that belonged to someone else instead of leaving them ambiguous. Keep a few specifics back as well, like the name of the metric or who actually objected, so a probe can be answered with something you had not already said.

Discuss a failure you've experienced and what you learned from it.

Medium · behavioural and stakeholder questions

Approach

1. Pick a story where you drove the decision, not one where you observed it.
2. Name the disagreement or constraint, and how you resolved it with evidence.
3. Quantify the outcome, including what you would not claim credit for.

Follow-up

• How did you know the outcome was caused by your change?
• What did you decide not to do, and why?

Handle a request for numbers supporting a decision already made

Hard · integrity · framing · stakeholder

A senior leader has already decided to sunset a plan tier and asks you for the analysis showing it is the right call. Accounts on that tier carry 6% of MRR at constant FX and have the highest licensed-seat utilisation in the book. The leader's support matters to your next review cycle, and the decision is being presented in four days. Deliver what you produce, what you decline to produce, and the exact sentence you will say in the meeting where the number appears on a slide.

Approach

1. Recognise what is being probed: whether you can find the legitimate request inside an illegitimate framing instead of either complying or refusing on principle. The generic answer promises to push back; the strong one produces something genuinely useful and states its limits in the room, without ambushing anybody.
2. Separate the decision from the justification. Sunsetting the tier may be correct for reasons the data does not hold, such as support cost, roadmap surface area or sales motion. What you decline is a one-sided document. What you produce is the case read both ways, which also happens to be more useful to the leader.
3. Build the symmetric analysis: MRR at risk at constant FX, the share of affected accounts with a plausible migration path given seats_licensed and billing_term, the recovery rate assumed for that migration and where it came from, and the downside case in which high-utilisation accounts treat the sunset as a reason to re-evaluate the vendor entirely.
4. Surface the inconvenient fact privately and early. The highest seat utilisation in the book is a retention signal, and the leader should hold it before the room does, so they can incorporate it rather than be caught by it.
5. Agree the meeting sentence in advance with the leader, so that nobody is surprised. Something to the effect that the tier is 6% of MRR and its accounts are the most heavily used in the book, and that the case for sunsetting rests on cost and focus rather than on revenue. That is true, it supports the decision on its real grounds, and it stops the deck claiming the numbers endorse it.
6. Decide your own line before you need it: what you will not put your name to, and that the route if asked anyway is your own manager rather than a confrontation in the meeting.

Follow-up

• The deck circulates with your analysis included and the downside case removed. What do you do, and by when?
• What changes if the honest analysis says the sunset is clearly the wrong call?
• How do you write the same memo when the leader is your skip-level and the meeting is tomorrow?

Disagree with a product manager's roadmap claim using data

Medium · causal inference · selection bias · stakeholder

A product manager proposes building a feature on the argument that accounts connecting an integration in week one retain three times better at week four. The figure is correctly computed from dim_user and fct_event, and it has already been shown to leadership. You have one scheduled 1:1 before the roadmap locks. Deliver the specific analysis you would run to test whether the relationship is causal, the result that would change your own mind, and how you open the conversation so that the PM is not put in the position of defending the number in public.

Approach

1. Recognise what is being probed: whether you can separate a number being right from an inference being wrong, and do it without costing the PM face. The generic answer recites that correlation is not causation; the strong one names the specific confound and proposes the cheapest design that could distinguish the explanations.
2. State the alternative concretely. Accounts that connect an integration in week one are accounts that already have a workflow and a technical owner, so week-one intent plausibly drives both the connection and week-four retention. The selection is on intent, which no amount of post-hoc adjustment observes.
3. Order the discriminating analyses by cost. First, condition on pre-connection activity by comparing retention within strata of week-one core-action count, which removes the crude version of the confound but not unobserved intent. Second, look for variation in integration availability that was unrelated to intent, such as a staggered release or an outage window. Third, an encouragement design that randomises a prompt to connect and reads the intent-to-treat effect on week-four retention, which is the only version that identifies an effect.
4. Run the timing check, because it is nearly free and it is the most persuasive single piece of evidence. If the retention advantage among connectors is already visible before any of them connected, the causal story is largely finished.
5. Pre-commit to what would change your mind and say it before you show anything: if the gap survives stratification and the encouragement arm moves week-four retention at all, the feature has a case and you will say so.
6. Open the 1:1 by agreeing with the true part, that the correlation is real and worth chasing, then ask what effect size the roadmap plan assumes. That makes the size of the claim the topic instead of its authorship.

Follow-up

• The encouragement test needs six weeks and the roadmap locks in two. What do you recommend in the interim?
• Stratifying on week-one activity closes half the gap. What do you conclude, and what do you still not know?
• How would you word this in the roadmap document so the PM's original number is reframed rather than deleted?