Interview guides · Kalshi

Kalshi Data Scientist Interview Questions & Guide 2026

Modeling drills for a Kalshi data scientist interview: baselines, regression interpretation, class imbalance, model defense.

The 60-second brief

As a Data Scientist at Kalshi, you are at the forefront of the prediction market revolution. You will be tasked with transforming massive, complex datasets into actionable insights that drive product strategy, market liquidity, and user engagement. Your work directly influences how Kalshi builds its platform, ensuring that its event-based contracts are accurately priced and accessible.

In modelling rounds the live question is usually why this model class for this problem, and how you would know six weeks after launch that it is still working. Deriving gradients by hand is rarely what is being probed.

Focus areas

Turn a vague request into a measurable question
Separate novelty effects from durable behaviour change
Size an experiment before anyone launches it

Overview

This role requires a unique intersection of rigorous quantitative analysis and product intuition. You will not just be building models; you will be identifying new market opportunities, optimizing user funnels, and working closely with engineering and product teams to translate data into tangible business outcomes. If you are comfortable working in a high-stakes, fast-paced environment where your technical output has an immediate, visible impact on the business, this is the right challenge for you.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Reading a pooled rate that moved because the mix moved, not because any behaviour changed

A pooled conversion rate is a weighted average, and a shift in the weights can move it in the opposite direction to every one of its parts. A paid campaign that brings low-converting traffic drops overall signup conversion even if desktop, mobile web and app conversion each rose that week, which is Simpson's paradox and it is the single most common cause of an inexplicable dashboard move. The discipline is to decompose before explaining: recompute the rate holding last period's segment weights fixed, and compare that counterfactual to the actual, so the mix effect and the rate effect are separated numerically rather than argued about. Segment on the dimensions that actually reweight, which in this domain are almost always device_type, referrer_channel, country and new versus returning.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Extrapolating a first-week lift inflated by novelty effects

Plot the treatment effect by days since first exposure instead of quoting one pooled average. A lift that decays toward zero across the test window is behaviour that will not persist, and annualising it produces a forecast that misses by an order of magnitude.

Questions & practice

How do you determine if a model is overfitting, and what steps do you take to mitigate it?

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.
Say how the offline result would be validated online before it is trusted.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Explain the trade-offs between different evaluation metrics for a binary classification model.

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Worked solution (25 min)

Aggregate to one row per (week, segment tuple) with summed visitors and signups, then split into w0 and w1 frames and align with an outer join, filling missing visitors and signups with 0.
Compute w_s = visitors / visitors.sum() within each week, and r_s = signups / visitors only where visitors > 0. Where a week's visitors are 0, set that week's r_s to that week's pooled rate, the stated convention; never leave it NaN.
rate_effect = (w0 * (r1 - r0)).sum(); mix_effect = ((w1 - w0) * r0).sum(); interaction = ((w1 - w0) * (r1 - r0)).sum().
contribution = w0*(r1-r0) + (w1-w0)r0 + (w1-w0)(r1-r0) per segment, which reduces to w1r1 - w0r0; sort by abs and return the head.
assert abs(rate + mix + interaction - (pooled1 - pooled0)) < 1e-12.

Expected result: Three scalars summing exactly to the observed pooled change, plus a segment table whose contribution column also sums to that change. In the stated case rate and mix carry opposite signs: several segments converted better while volume moved toward a weaker-converting one, which is what makes the pooled move look inexplicable until it is split.

Verify it yourself

On synthetic data where every segment appears in both weeks, every segment rate is held constant and only the weights move, rate_effect and interaction are exactly 0.
On synthetic data where every segment appears in both weeks, weights are held constant and only rates move, mix_effect and interaction are exactly 0.
The per-segment contribution column sums to the pooled change, matching the sum of the three scalars.
Swapping the two weeks negates the total, as it must, but not the parts. Assert the three identities that actually hold: the interaction term is exactly invariant, because both of its factors change sign; the swapped rate effect equals -(rate_effect + interaction); and the swapped mix effect equals -(mix_effect + interaction). They still sum to the negated pooled change. Expecting all three to flip sign is the common wrong assertion.
Plant a segment present only in week 1, holding 3 percent of week-1 visitors at a rate well below pooled. Under the documented pooled-rate convention its mix contribution is w_s1 * r_pooled0 and its interaction contribution is w_s1 * (r_s1 - r_pooled0), with a rate contribution of exactly 0; under a zero-fill convention its mix contribution is exactly 0 and the whole w_s1 * r_s1 lands in interaction. Assert whichever one the function documents, and fail if mix is asserted non-zero while the code zero-fills.

Follow-up

The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Read an experiment from first exposure, not assignment

Hard · experiments · window-functions · srm

fct_experiment_exposure holds experiment_id, unit_type, unit_id, variant, user_id, assigned_at_utc, first_exposed_at_utc, is_in_analysis_population and planned_end_utc. fct_event holds user_id, occurred_at_utc, is_core_action, and carries events up to a known data cut, :data_cut_utc. For one experiment randomised on unit_type = 'user', return per variant: exposed units, units with at least one core action in the seven days after that unit's own first exposure, the rate, and the variant share of exposed units. Only units whose seven-day window has fully elapsed as of the data cut belong in the readout. Units appearing under more than one variant are excluded from both arms and counted separately.

Approach

Run the contamination pass as an aggregate, not a window: SELECT unit_id FROM fct_experiment_exposure WHERE experiment_id = :exp GROUP BY unit_id HAVING COUNT(DISTINCT variant) > 1, then anti-join it away. PostgreSQL rejects COUNT(DISTINCT variant) OVER (PARTITION BY unit_id) outright, since DISTINCT is not implemented for window functions; if you want the test inline, MIN(variant) OVER (PARTITION BY unit_id) <> MAX(variant) OVER (PARTITION BY unit_id) is the equivalent that does run.
Do not resolve contamination by keeping the earliest variant. A unit that saw both arms carries treatment from both, so assigning it to either one biases that arm.
Define the population as is_in_analysis_population = TRUE AND unit_type = 'user' AND first_exposed_at_utc < planned_end_utc AND first_exposed_at_utc + interval '7 days' <= :data_cut_utc. The horizon filter is what makes the readout reproducible next week instead of drifting with every re-run; the data-cut filter is the one that actually buys seven days of follow-up, since a unit exposed an hour before the horizon otherwise contributes an hour of observation to a seven-day rate.
Measure the outcome on a per-unit relative window: LEFT JOIN fct_event on user_id with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc, first_exposed_at_utc + interval '7 days'). LEFT JOIN so units with no outcome stay in the denominator at zero rather than being deleted by an inner join.
Check the sample ratio before reading the effect: variant share of exposed units against the intended split, tested as a binomial. Run it on the truncated population as well as on the full exposed set, because if one arm exposes later on average the data-cut filter removes more of that arm and can manufacture a ratio mismatch the randomisation did not have. A mismatch on the full set means the exposure data is not a valid randomisation and invalidates the readout rather than being a footnote under it.
Report the per-variant rate, the absolute difference, and the fact that the variance unit is unit_id. That is straightforward here only because the grain is already one row per user; a per-session outcome under user randomisation would need a delta-method or bootstrap standard error instead.

Follow-up

Some units were assigned days before they were exposed. What does analysing the assigned set instead do to the estimated effect, and in which direction?
The split is 51/49 on 400,000 exposed units. Do you read the result?
The treatment arm exposes on average two days later than control. What does that do to a fixed calendar outcome window, and which arm does it favour?

Seven-day activation rate by weekly signup cohort

Easy · cohorts · activation · date-bounds

dim_user holds user_id, account_created_at_utc, is_internal. fct_event holds user_id, occurred_at_utc, is_core_action. A user is activated when core-action events fall on at least two distinct UTC dates inside [account_created_at_utc, account_created_at_utc + 7 days). Return, for the last twelve complete weekly signup cohorts, the cohort week, cohort size, activated users and the activation rate. Exclude is_internal users. Every signup in the cohort week stays in the denominator, including users who never returned.

Approach

Start from dim_user as the denominator spine with is_internal = FALSE and DATE_TRUNC('week', account_created_at_utc) as the cohort key. Driving the query from the event table instead would silently condition on having events and delete the entire non-activating population.
Join fct_event on user_id with is_core_action = TRUE and a per-user bound, occurred_at_utc >= u.account_created_at_utc AND occurred_at_utc < u.account_created_at_utc + interval '7 days'. The bound is correlated to each user's own signup timestamp, not a single global date range.
Aggregate per user with COUNT(DISTINCT occurred_at_utc::date) >= 2, then LEFT JOIN that back onto the spine and COALESCE the flag to FALSE so non-activators contribute a zero rather than vanishing.
Restrict the published cohorts to those whose week ended at least eight days ago. A cohort younger than that has not finished its seven-day window, so its rate is mechanically low and reads as a decline.
Roll up by summing the numerator and denominator per cohort week, and state the two-distinct-days threshold next to the number since it is a choice that re-bases the whole history if changed.

Worked solution (20 min)

Write the cohort spine and confirm its total equals the count of non-internal signups in the date range.
Write the per-user distinct-active-days CTE with both interval bounds and inspect a handful of users manually.
LEFT JOIN, COALESCE the flag, aggregate to cohort week.
Apply the eight-day publication lag and drop the incomplete cohort.
Re-run with a closed upper bound (<= +7 days) and note how many users change state, to show the boundary is doing work.

Expected result: Twelve rows, one per cohort week, each with cohort_size, activated and rate. Cohort sizes sum to the total non-internal signup count across those twelve weeks. No cohort is younger than eight days.

Verify it yourself

activated is never greater than cohort_size in any row.
A user with three core actions on a single day is not counted as activated.
Removing the LEFT JOIN in favour of an inner join raises every rate, which demonstrates the denominator was being conditioned on survival.
Cohort sizes sum to an independent COUNT(*) over dim_user for the same range.

Follow-up

Why two distinct days rather than one event? What happens to the published history if someone changes it to three?
Invited seats and SSO-provisioned users get an account_created_at_utc at provisioning and may never sign in. Should they be in this denominator?
The rate rose 3 points this week. What do you check before believing it?

How would you measure the impact of a specific feature change on user retention?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How would you approach forecasting volume for a new event contract on the platform?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
State your assumptions explicitly before working the problem.
Work from the decision backwards to the evidence you would need.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Metric tree for a seat-expansion push on seat-based plans

Medium · revenue · seats · leading indicators

Sales wants a push to add seats to existing accounts. Tables: dim_account (account_id, seats_licensed, seats_assigned, lifecycle_status, current_plan_tier), fct_subscription_period (subscription_id, account_id, mrr_cents_constant_fx, seats_billed, period_start_utc, period_end_utc, change_reason), and fct_event (is_core_action, user_id, account_id, occurred_at_utc). Build the metric tree from net revenue retention down to the metric the push team is graded on, name the primary, and give the guardrail that fires before the damage shows up in NRR. Deliverable: the tree, the primary metric, and the lead-time argument for the guardrail.

Approach

Grade the team at the grain it controls. NRR is a trailing twelve-month revenue-weighted ratio that the push cannot move inside a quarter and that also moves for reasons the push does not touch, so it belongs at the top of the tree and nowhere near the target.
Define the primary as a delta rather than a count: the sum of period-over-period mrr_cents_constant_fx changes on rows with change_reason = 'seat_change' and a positive seats_billed delta. A COUNT of seat_change rows rewards churn of seats as much as growth.
Name the failure the target creates: seats sold and never assigned book revenue now and reappear as a seat reduction at the next renewal, six to twelve months later, comfortably outside the window the team is measured in.
Pick licensed-seat utilisation as the guardrail precisely because of that lag — distinct user_id with an is_core_action event in the trailing 28 days over seats_licensed, MRR-weighted across lifecycle_status = 'active' accounts — and read it against each account's renewal date rather than the calendar.
Close the horizon gap with a rule instead of a hope: the push books expansion MRR immediately but retains credit only for accounts still above the utilisation threshold at the following renewal.

Worked solution (30 min)

Build the tree downward: NRR at twelve months, then expansion plus contraction plus churned MRR, then seat_change deltas and tier changes, then seats billed per account, then licensed-seat utilisation.
Write the primary as a period-over-period MRR delta restricted to positive seat changes, in constant FX, on subscription_id.
Write the guardrail with its weighting and its read points: 30 and 60 days before each account's renewal date.
State the lead time in months: utilisation moves within one 28-day window of a seat sale, while the seat reduction it predicts reaches NRR only at the next renewal.
Write the credit rule that ties the two horizons together.

Expected result: A tree with expansion MRR as the primary, utilisation as the guardrail with its lead time stated in months, and a credit rule that makes the push team carry the renewal-horizon consequence.

Verify it yourself

Summing mrr_cents_constant_fx across all period rows in a month double-counts any account with a mid-period change; the monthly figure must be built from period boundaries rather than a naive SUM over rows.
seats_assigned can exceed seats_licensed during a grace period, so a utilisation computed on assigned seats can exceed 1. The denominator is seats_licensed.
The MRR-weighted and unweighted utilisation means should differ substantially. If they do not, single-seat accounts are probably being dropped somewhere in the aggregation.

Follow-up

How do you separate seat expansion from price and tier expansion inside NRR without double-counting either?
An account buys 40 seats and assigns 6 during a phased rollout. Is that expansion, and what would change your answer?
What does the utilisation threshold do to a genuinely growing customer three weeks into a rollout?

Tell a pipeline outage from a collapse in usage

Medium · pipeline-failure · instrumentation · event-volume

Weekly active accounts completing a core action fell 9%, and almost the entire fall sits in accounts whose events carry surface = 'ios'. App crash rates and store reviews are unchanged. You have fct_event with occurred_at_utc, received_at_utc, event_name, is_core_action, app_version and surface, plus fct_session and the ingestion job run log. Establish within the hour whether iOS engagement fell or iOS events stopped arriving, name the evidence that distinguishes them, and say what you would publish on the dashboard in the meantime.

Approach

Compare the event-name composition inside surface = 'ios' against the prior four weeks as shares, not counts. A behaviour collapse scales most event names together; a dropped event definition or a broken downstream filter hits specific event_name values while page_view and session-opening events hold steady. That shape difference is the fastest discriminator available.
Profile the received_at_utc minus occurred_at_utc distribution per day for surface = 'ios'. A stalled-then-backfilling pipeline shows a fat upper tail and a recovering p99; a silently dropped stream shows an unchanged lag distribution over a smaller volume. The two failure modes have different remedies and different histories.
Cut by app_version. A logging SDK change arrives with one build and ramps with its adoption curve; an infrastructure fault arrives across every build within the same hour. Checking this costs one group-by and rules out half the hypothesis space.
Cross-check against a signal that does not travel the suspect path: server-emitted events with session_id NULL, and subscription or billing activity for the same accounts. If those accounts are still transacting, the users did not leave.
Publish an ex-iOS total with an explicit annotated break rather than a blended total. A blended number during a known ingestion gap is wrong in a direction you can already name, and republishing it daily spreads the artefact into every downstream report.

Follow-up

Suppose the events do eventually backfill. What is your policy for restating the published weekly numbers, and who needs to be told?
What monitor would have caught this before a human noticed the weekly metric, and what would it alert on?
If is_core_action is maintained in the tracking plan, what governance would stop a change to that list from silently moving a north-star metric?

Suggested practice checklist

For someone who has spent the last year in notebooks, dashboards or modelling work and has not written raw SQL under time pressure. The first four days rebuild query fluency against a fixture you control and can verify by hand; the last three attach that fluency to the rest of the loop.

Day 1 — Build a fixture you can check answers against

Create a local Postgres or SQLite database with four tables (users, sessions, events, orders) holding roughly 200 rows you generated yourself, so you know the contents well enough to predict every result.
Deliberately seed the cases that break queries: a user with no sessions, a session with no events, two orders sharing a timestamp, a NULL in one join key, and one duplicated user row.
Before writing any SQL, hand-compute five answers on paper (how many users placed at least one order, median orders per ordering user, and three others) and save them as the ground truth for the week.

Deliverable: A one-command seed script plus a text file of five hand-computed answers to grade every later query against.

Day 2 — Joins, filters and NULL semantics

Answer "which users have no orders" three ways (LEFT JOIN with IS NULL, NOT EXISTS, NOT IN) and confirm that the NOT IN version returns zero rows once the subquery contains a NULL, because the comparison is never TRUE.
Reproduce the LEFT JOIN that silently collapses to an inner join by putting a right-table predicate in WHERE, then fix it by moving the predicate into the ON clause, and record both row counts.
Create a fan-out bug on purpose by joining orders to order_items and summing the order total, then correct it with a pre-aggregated subquery and explain in one line which table changed the grain.

Deliverable: One annotated .sql file holding the three join traps, each with the wrong result and the corrected result side by side.

Day 3 — Window functions and frames

Write three window queries against the fixture: a running order total per user, the rank of each order within its user by value, and the day gap to that user's previous order, then check each against the day-one ground truth.
Run ROW_NUMBER, RANK and DENSE_RANK over a column containing ties, print all three side by side, and write one sentence on when each is the correct choice.
Switch one query from the default frame (RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which is what you get when ORDER BY is present and no frame is written) to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and explain why the output differs only when the ORDER BY column has duplicates.

Deliverable: Three verified window queries plus a short note explaining the RANGE versus ROWS difference in your own words.

Day 4 — The four analytical query patterns

Write a monthly retention grid: first order month per user, then months-since-first as the column, and verify that month zero equals the cohort size exactly.
Sessionize the events table under a 30-minute inactivity rule using LAG plus a cumulative sum over a new-session flag.
Build a four-step funnel that counts distinct users rather than events at each step, and state the rule you applied to a user who reaches step three without ever logging step two.

Deliverable: One file with the retention, sessionization and funnel patterns, each carrying a one-line note on the assumption it bakes in.

Day 5 — Write SQL the way you will have to write it live

Set a 12-minute timer and solve three medium prompts in a plain editor with no execution and no autocomplete, then run them and tally syntax errors separately from logic errors.
Narrate one solution aloud while writing it, stating the grain of each intermediate result (one row per user, one row per user-day) before you type its body.
Rewrite your slowest solution as a CTE chain where every CTE name states its grain, and time yourself re-solving it from blank.

Deliverable: A recording of one narrated solution plus an error tally that separates syntax from logic.

Day 6 — One day for everything that is not SQL

Write the preconditions of the two-sample t-test from memory, then check them: independent observations, and a difference in means whose sampling distribution is approximately normal, which at large sample sizes follows from the central limit theorem rather than from normality of the raw values.
Write the difference between an odds ratio from logistic regression and a relative risk, and state the condition under which the two are close (low outcome prevalence).
Prepare a 90-second answer to "how would you know this model is any good" that names the metric, the baseline you would beat, and the cost of the errors you care about.

Deliverable: One page of notes covering test preconditions, the odds-ratio caveat and the model-quality answer.

Day 7 — Full loop rehearsal

Run a 45-minute mock with someone willing to interrupt: 20 minutes of SQL, 15 minutes defining a metric, 10 minutes on a past project.
Re-solve from blank the two queries you were slowest on this week and compare the times against day five.
Write a five-line answer to "walk me through a project" that puts a number in the first sentence and names the decision the work changed.

Deliverable: Mock feedback notes plus a timed project narrative you can deliver without reading it.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Describe a time you had to deal with a highly imbalanced dataset; what techniques did you use?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

How did you know the outcome was caused by your change?
What did you decide not to do, and why?

Explain a wide interval to a non-technical executive

Medium · communication · uncertainty · pricing

A pricing change is under consideration. Your best estimate of its effect on trial-to-paid conversion is a 1.8pp drop, with a 95% interval from a 4.6pp drop to a 1.0pp rise, read from a geo holdout rather than a randomised test. An executive preparing a board slide asks you for 'the number'. You have ninety seconds and one slide, and the words confidence interval, p-value and significance are not usable with this audience. Deliver the slide headline, the single supporting line, and what you say aloud.

Approach

Recognise what is being probed: whether you can carry uncertainty into a decision instead of either hiding it or hiding behind it. The generic answer promises to explain the interval in plain English; the strong one replaces the question 'what is the number' with 'across this range, where does the decision change'.
Find the threshold before you draft anything. Ask what the pricing case assumes, then compute the conversion drop at which the higher price stops adding revenue: price uplift on the conversions kept against the revenue lost from conversions forgone. That single figure is what makes the range legible.
Restate the estimate and both bounds in the unit the audience already reasons in. Convert percentage points into monthly first-paid conversions at current trial volume, then into mrr_cents_constant_fx, so the slide reads as money per month rather than as statistics.
Place the range against the break-even and say which part of it sits on each side. If most of the range clears the threshold, that is a recommendation to proceed with a monitoring plan; if the range straddles it, that is a recommendation to narrow the range first.
Name what would narrow it and what that costs in weeks, then give one recommendation with an explicit condition for revisiting it. Uncertainty stated without a next step is read as indecision and the midpoint gets used anyway.

Follow-up

The executive says to give the midpoint and they will manage the risk. What do you do?
How does the slide change if the interval were a 4.6pp to 0.2pp drop, with no positive outcomes in range?
Why is a geo holdout the credible read here rather than the attributed channel numbers you already have?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?