DICK's Sporting Goods Data Scientist Interview Questions & Guide 2026

The 60-second brief

As a Data Scientist at DICK'S Sporting Goods, you are at the forefront of transforming how the nation's leading sports retailer interacts with millions of athletes. Your work directly influences everything from personalized customer experiences and dynamic pricing to complex supply chain logistics and inventory optimization. By leveraging vast amounts of retail and customer data, you help ensure the right products are in the right stores at the right time.

How much statistics you need depends on the flavour of the seat. Experiment-facing work wants you deep enough to notice that repeated looks at accumulating data inflate the false positive rate of a fixed-sample test; modelling-facing work wants estimation and honest uncertainty intervals.

Focus areas

• Decompose a metric move by segment and mix
• Define numerator, denominator and window precisely
• Turn a vague request into a measurable question

Overview

This role is critical to the continued digital and omnichannel evolution of DICK'S Sporting Goods. You will not just be building models in a vacuum; you will be solving high-impact, real-world retail challenges. Whether you are optimizing predictive models for e-commerce or designing algorithms to streamline brick-and-mortar operations, your insights will drive strategic business decisions across the enterprise.

Expect a fast-paced environment where scale and complexity meet. You will collaborate closely with cross-functional teams within the data analytics department, engineering, and product management. To succeed here, you must be as passionate about understanding the retail landscape as you are about writing clean, efficient code and deploying robust machine learning models.

The DICK's Sporting Goods interview process

DICK's Sporting Goods candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Automated Coding Assessment

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

• Whether an ambiguous term becomes a specific column and filter before any computation happens
• Whether you read the schema for keys and cardinality rather than only for column names
• Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

• Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
• On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
• Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Super Interview

An extra round usually exists because something is still open after the standard loop: a skill the earlier interviews did not sample, a level decision, or two interviewers who disagreed. It is rarely a rerun of what you already did well. Ask the recruiter who you are meeting, what function they sit in, and how long the session runs. That is an ordinary scheduling question, and the answer changes what you should prepare. What separates a strong candidate here is treating the round as a fresh evaluation with its own bar, rather than assuming earlier performance carries you through or sinks you.

What to demonstrate

• Whether you can answer well on ground the earlier rounds did not cover, without leaning on what you already said to someone else
• Consistency of the facts in your stories: the same sample size, timeframe, team size and scope of your own role as in earlier conversations
• How you handle an unfamiliar format live, including whether you ask what kind of answer is wanted before producing one

How to prepare

• Ask the recruiter for the interviewer's function, the length, and whether to expect a coding surface, a discussion, or a presentation. Preparing for a 30 minute conversation with a partner team is not the same work as preparing for a 60 minute technical block.
• Write out what each earlier round actually covered, then list the two or three areas nobody probed. That gap is the most likely subject of the extra round.
• Re-read the numbers in the project stories you have already told, so a second telling does not quietly contradict the first.

Semi-Technical or Behavioral Round

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

• Whether you can state the other side's argument accurately before you explain why you disagreed
• What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
• Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

• Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
• For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
• Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Where candidates lose points

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Comparing cohort retention curves of different maturities, or building the curve from users who are still present

A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.

Defining the cohort on a post-treatment condition

Ask how rows entered the table. Filtering on something that treatment itself influences, such as users who finished onboarding or accounts still active at ninety days, breaks comparability between arms; define the population at an entry point that precedes exposure and keep everyone in it.

Accepting a metric definition without asking about the denominator

Pin down the denominator, the eligibility filter and the time window before computing anything: conversion rate per session, per user, per eligible user and per new user are four different numbers with different behaviour. Restate the definition in one sentence and get agreement before you analyse.

Questions & practice

How would you approach building a predictive model for seasonal inventory demand?

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Frame the prediction: the label, the moment of prediction, and the action it triggers.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• What would you monitor after launch to know the model is still valid?
• How would you choose the decision threshold, and who owns that choice?

Explain the mathematical difference between Ridge and Lasso regression. When would you use each?

Medium · machine learning and modelling

Approach

1. Set a baseline first, so any model has something honest to beat.
2. Frame the prediction: the label, the moment of prediction, and the action it triggers.
3. Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Rebuild per-visitor ordering without groupby convenience methods

Easy · pandas · vectorisation · window logic

You have a DataFrame of 2 million fct_event rows with visitor_id, occurred_at_utc and event_id, unsorted and containing duplicate timestamps within a visitor. Produce three new columns: event_rank, the 1-based position of the event within its visitor ordered by occurred_at_utc; seconds_since_prev, the gap to that visitor's previous event, NULL for the first; and is_first_for_visitor. You may use sort_values, shift, cumsum, numpy and boolean masking. You may not use groupby.transform, groupby.apply, groupby.cumcount, groupby.rank or merge_asof. Break timestamp ties on event_id.

Approach

1. Sort once by ['visitor_id', 'occurred_at_utc', 'event_id'] and reset the index. The whole exercise reduces to row arithmetic on a sorted frame, and the tiebreak on event_id is what makes the result reproducible across runs.
2. Mark visitor boundaries with is_first = df['visitor_id'].ne(df['visitor_id'].shift()). This is the single fact every other column derives from.
3. Compute seconds_since_prev as the diff of the timestamp column, then overwrite it with NaT/NaN wherever is_first is True. The shift crosses the boundary between visitors and will otherwise hand the first row of each visitor the last event of the previous one.
4. Build event_rank from a running counter that resets at boundaries: take a global cumulative position (np.arange(len(df))) and subtract, per row, the global position at which that visitor started. Get the start position by forward-filling the positions where is_first is True, which is a cumsum-free reset and is O(n).
5. Verify against the forbidden method once, as a test rather than as the implementation, and confirm the two agree on every row.

Worked solution (20 min)

1. Sort on the three-key tuple and reset_index(drop=True).
2. Compute is_first via .ne(.shift()), which is True for row 0 because the shifted value is NaN.
3. pos = np.arange(len(df)); start = pd.Series(np.where(is_first, pos, np.nan)).ffill(); event_rank = (pos - start + 1).astype(int).
4. gap = df['occurred_at_utc'].diff().dt.total_seconds(); gap[is_first] = np.nan.
5. Assert event_rank equals df.groupby('visitor_id').cumcount() + 1 on the sorted frame.

Expected result: Three columns on the sorted frame: event_rank starting at 1 for every visitor and increasing by 1 with no gaps, seconds_since_prev null exactly where is_first_for_visitor is True, and is_first_for_visitor summing to df['visitor_id'].nunique().

Verify it yourself

• is_first_for_visitor.sum() equals the number of distinct visitor_id values.
• seconds_since_prev.isna().sum() equals that same number.
• event_rank.max() per visitor equals that visitor's row count, and event_rank is non-negative everywhere.
• A visitor with two events at the identical timestamp gets ranks 1 and 2 with a gap of 0.0, ordered by event_id.

Follow-up

• The frame does not fit in memory. How does your approach change if you can only process one visitor-partitioned chunk at a time?
• occurred_at_utc is client-supplied and sometimes runs backwards within a visitor. Does your seconds_since_prev go negative, and should it?
• How would you extend this to reset the counter at every change of surface as well as visitor?

How would you handle a situation where your Python code is throwing a memory error while processing a large dataset?

Medium · sql and data manipulation

Approach

1. State the window function and its partition and ordering out loud before writing it.
2. Say which table is the grain you start from, and join outward from it.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

How do you optimize a slow-running SQL query if you cannot create new indexes or use temporary tables?

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. Say which table is the grain you start from, and join outward from it.

Follow-up

• How would you verify this result without re-running the same query?
• What breaks if events arrive late or out of order?

Write a SQL query to find the second highest purchasing customer in each region without using the WITH clause or window functions.

Medium · sql and data manipulation

Approach

1. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
2. State the window function and its partition and ordering out loud before writing it.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How would you verify this result without re-running the same query?
• How does the query change if the join becomes one-to-many?

Attribute account MRR to first-touch channel without seat fan-out

Medium · fan-out · joins · revenue · attribution

dim_user holds user_id, account_id, first_touch_channel, account_created_at_utc. dim_account holds account_id, lifecycle_status. fct_subscription_period holds account_id, period_start_utc, period_end_utc, period_status, mrr_cents_constant_fx. For a given as-of date, report active MRR by the first-touch channel of the account's earliest-created user. An account has many users and many periods. The channel totals must sum exactly to total active MRR on that date, and accounts whose label is NULL must appear as their own bucket.

Approach

1. Reduce the label side to one row per account before any join: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY account_created_at_utc, user_id) = 1 over dim_user, carrying first_touch_channel. Joining dim_user to the period table directly repeats each revenue row once per seat.
2. Reduce the money side to one row per account too: the period where period_start_utc <= :as_of AND period_end_utc > :as_of AND period_status IN ('active','past_due'). If more than one row survives that filter the account holds overlapping subscriptions, which must be deduped or deliberately summed, and either way the choice is stated rather than left to the join.
3. Join the two one-row-per-account CTEs and SUM(mrr_cents_constant_fx). Use the constant-FX column, since the same query run in two quarters otherwise reports currency moves as channel performance.
4. Bucket NULL first_touch_channel as 'unattributed' with COALESCE instead of letting it drop. Roughly the share of users with no observed touch is not small, and dropping it understates the total and inflates every named channel's share.
5. Reconcile: the SUM over the joined result must equal the SUM over the period CTE alone. That single equality check catches fan-out immediately, which is why it is worth writing before presenting the number.

Worked solution (30 min)

1. Compute total active MRR on the as-of date from the period table alone and write the number down as the target.
2. Build the label CTE and assert COUNT(*) = COUNT(DISTINCT account_id).
3. Build the as-of period CTE and assert the same, investigating any account with more than one surviving row.
4. Join, COALESCE the channel, aggregate, and compare the total to the target.
5. Deliberately run the naive dim_user join and record the inflation factor, which is the MRR-weighted mean seat count.

Expected result: One row per first_touch_channel plus an 'unattributed' bucket, whose MRR values sum exactly to the independently computed total active MRR for the as-of date.

Verify it yourself

• Channel total equals the standalone total from the period CTE, to the cent.
• Both CTEs satisfy COUNT(*) = COUNT(DISTINCT account_id).
• The unattributed bucket is present and non-zero rather than silently absent.
• The naive join version produces a strictly larger total, and you can state the ratio.

Follow-up

• Which user should own the account's label: earliest created, the 'owner' role, or the one who converted? What does each choice systematically favour?
• What does this table entitle you to say about the value of a channel, and what does it not?
• Some accounts have a parent_account_id. How does rolling those up change both the numerator and the channel mix?

What metrics would you use to evaluate a product recommendation engine on our website?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Name one primary metric, then the guardrail that stops it being gamed.
3. Fix the population and the time window before naming any metric.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

Choose one success number for a homepage redesign

Easy · conversion · guardrails · mix effects

A homepage redesign is ready to test. Marketing wants visit-to-signup conversion as the single success metric: distinct fct_session.visitor_id with a 'signup_completed' event in fct_event, over distinct visitor_id with a session started in the window where is_bot_flagged = FALSE and consent_state <> 'denied'. Sessions carry referrer_channel and device_type. Argue for or against that metric, name what you would decide on instead, and give the one guardrail you would refuse to ship without. Deliverable: the metric, the guardrail, and the failure mode in writing.

Approach

1. List everything that moves this rate without the page changing: referrer_channel mix, device_type mix, cookie lifetime and bot-rule changes, then pre-register the segments you will decompose on so the decomposition is not chosen after seeing the result.
2. Argue that signup is an intermediate outcome the homepage can inflate by over-promising, and propose activated signups per 1,000 eligible visitors — the same numerator further restricted to users clearing the seven-day activation bar — as the number the decision actually rests on.
3. Explain the consent exclusion rather than copying it: a 'denied' session can never be joined forward to a user, so leaving it in the denominator puts visitors there who have no path into the numerator and depresses the level permanently.
4. Separate what the metric can and cannot be used for: inside a randomised comparison over one window it is fine, but as a trend line across a platform privacy change it will step down because visitor_id resets more often, with no behaviour change behind it.
5. State the trade you would accept in advance: a smaller conversion gain with flat activation beats a larger conversion gain with activation down, and write the threshold before the readout.

Worked solution (15 min)

1. Restate the proposed metric and list every input that can move it with the page unchanged.
2. Compute the mix-held-fixed counterfactual: reweight this period's per-segment rates by last period's segment volumes, and report the difference against the pooled rate as the mix effect.
3. Define the decision metric: activated signups per 1,000 eligible visitors, with the seven-day activation predicate spelled out.
4. Write the guardrail and the explicit trade-off sentence you would put in the ship decision.
5. Record the denominator exclusions with the reason for each, not just the predicate.

Expected result: Conversion kept as a diagnostic, activated signups per 1,000 visitors named as the decision metric, a specified mix decomposition, and one guardrail carrying a written trade-off.

Verify it yourself

• Weekly rates rolled up by re-summing numerator and denominator differ from the average of the weekly rates whenever weekly traffic is uneven. If the two agree, a rate-of-rates has been computed somewhere.
• Per-segment numerators must sum to the pooled numerator. If they do not, a visitor is appearing in two segments and the segmentation is not a partition.
• Activation can only be read eight days after the last signup in the window, so the earliest honest decision date is eight days after the test stops.

Follow-up

• The redesign wins overall but the entire gain sits in paid social. What do you do, and what would change your mind?
• How do you roll four weekly conversion rates up to a month, and why does the obvious way give a different answer?
• Where does holding the segment mix fixed stop working as a correction?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

1. Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
2. Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
3. Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
4. Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
5. Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

• How do you choose the fixed cohort age, and what do you lose by choosing it too long?
• If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
• The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

Roughly 90 minutes a night on weekdays with one longer weekend block. The plan deliberately cuts scope rather than compressing everything, on the assumption that finishing one thing a night beats half-starting four.

Day 1 — Fix the scope and set a baseline

• Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list for everything else and keep it visible all week.
• Take one 20-minute SQL prompt and one 10-minute metric question cold, and write the single sentence that says what blocked each attempt, since that sentence is what decides which two topics get the most evenings.
• Set the week's one rule: one problem finished to completion every night, including the night you only have 40 minutes.

Deliverable: A one-page scope with an explicit not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One query pattern, written three times

• Choose the single pattern most likely to appear (a cohort retention grid, or a funnel counted by user) and write it three times from a blank file rather than editing the previous attempt.
• On the third attempt, write the grain of every CTE as a comment before writing its body.
• Stop at 90 minutes even if the third version is imperfect, and write the one thing you would fix with another hour.

Deliverable: Three independent versions of the same query plus a note on what changed between them.

Day 3 — Only the statistics you will be asked to defend

• Write, in under 200 words, how you would decide whether a difference between two groups is real: the test, its assumptions, and what you would switch to when an assumption fails.
• Compute a 95 percent confidence interval for a difference in proportions by hand on realistic numbers, then write in one sentence what changes if the two samples are paired rather than independent.
• Write your answer to "what does a p-value mean", check it against a definition, and delete the version that describes it as the probability the hypothesis is true.

Deliverable: A 200-word written answer and one hand-computed interval you can reproduce under pressure.

Day 4 — One case, and the assumptions holding it up

• Answer one product case aloud in 20 minutes with a recording running, then listen back with a pen and mark every claim you asserted without saying what it rested on: an assumed user behaviour, an assumed data source, an assumed baseline rate, an assumed grain.
• Pick the three assumptions the recommendation actually depends on, write how you would check each one against data, and say which one being wrong would flip the recommendation rather than merely weaken it.
• Write the four-step structure you used onto a card small enough to hold in working memory when you are nervous.

Deliverable: One recording, three load-bearing assumptions each with a written check, and a four-step structure card.

Day 5 — Your own work, timed

• Write a 90-second version and a four-minute version of your main project, and time both out loud rather than reading them.
• Prepare answers to the two follow-ups that always come: what you would do differently, and how you knew it worked.
• Put one number in the first sentence and be able to say exactly where that number came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in a longer weekend block

• Run a 60-minute mock covering query work, a case and a behavioural question in a single sitting with no breaks, because sustained attention is the thing evenings have not trained.
• Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
• Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

• Write the 20-minute warm-up you will actually do on the morning of the interview: one query you can already write from a blank file, one metric you can define out loud, and nothing you have never seen before.
• Re-read only your own notes from this week, and open no new material.
• Write down the logistics: the tool you will be asked to work in, whether lookups are allowed, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the case structure, the project numbers, and the logistics.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Walk through an analysis you got wrong and what changed

Easy · postmortem · data quality · self-assessment

Describe an analysis of yours that turned out to be wrong after somebody had already acted on it. You have four minutes. The account must name the defect mechanically, the join, the filter, the window or the identity key, rather than describing it as a communication problem. It must also say who did what because of the wrong number, how the error surfaced, how long it stood, and what control you put in place so that class of error cannot reach a decision again. Do not pick an error nobody acted on.

Approach

1. Recognise what is being probed: whether you can be specific about your own failure without minimising it or performing contrition. The discriminator is whether the defect has a mechanism the listener could reproduce in their own warehouse.
2. Choose the case by blast radius rather than by comfort. An error nobody acted on tests nothing, and picking one signals that you are managing the interview instead of answering it.
3. Structure the account in six beats: the number, the decision it drove, the defect, the detection, the correction, the control. Keep the defect to one reproducible sentence, for example an inner join to fct_subscription_period that dropped accounts with no subscription row and so computed retention over payers only.
4. State the direction of the bias, not only its existence. A filter or join that removes rows usually moves a metric predictably, and knowing which way shows you diagnosed the mechanism rather than patched the symptom.
5. Be exact about detection and elapsed time. 'A colleague noticed' and 'the row-count assertion failed before publication' are different answers about the same organisation, and the second one is the one your control is supposed to produce next time.
6. End on the control, its cost, whether it has fired since, and one thing it does not cover.

Follow-up

• What did the control cost, and has it fired since? If it never has, how do you know it works?
• How long did the wrong number stand before anyone questioned it, and what does that say about the review path it went through?
• What is the equivalent mistake you are most likely to make in this role, given the tables you would be working in?

Choose between three teams' requests with one analyst-week

Medium · prioritisation · stakeholder · judgement

You have one analyst-week. Three requests land the same morning. A growth team wants a paid-channel readout before a Friday spend decision. A billing team wants gross monthly revenue churn rebuilt, because the current figure recognises cancellation at canceled_at_utc rather than period_end_utc and is therefore wrong. A product team wants a dashboard for a feature launching in six weeks. All three sponsors are peers of your manager. Deliver your ranking, the explicit rule that produced it, and the message you send to the two teams you defer.

Approach

1. Recognise what is being probed: whether you prioritise on decision value and reversibility or on who asked most recently and most loudly. The generic answer sorts by importance; the strong one states a rule, applies it, and accepts the ranking it produces even where that is uncomfortable.
2. Score each request on three statable things: the decision it unblocks and the date that decision is made, the cost of being wrong in the meantime, and whether the work is one-off or compounding. A wrong published churn figure compounds, because it is quoted downstream and enters forecasts; the channel readout has a fixed date that cannot move; the dashboard has six weeks of slack.
3. Notice the tension between value and urgency rather than resolving it by feel. The churn defect is the most valuable item and the least urgent one, which is exactly the shape of work that never gets done.
4. Break the churn item in two. A one-hour severity check, sizing the gap between the two recognition points in MRR, is cheap enough to do before ranking anything and may promote the item outright. Do that first, then rank.
5. Make the deferrals concrete. Each deferred team gets a date, a reason expressed as another team's decision deadline rather than as relative importance, and the smallest thing you can hand them immediately.

Follow-up

• The dashboard team escalates to your manager. What do you say in that conversation?
• Your severity check shows churn is overstated by 15%. Does the ranking change, and does anybody need to be told today regardless of the ranking?
• A fourth request arrives Wednesday with a Thursday deadline. What comes off the list, and who do you tell first?

Defend a flat experiment readout against a post-hoc segment

Medium · experiments · segmentation · pushback

A feature you evaluated is flat on seven-day activation: +0.05pp with a 95% interval of [-0.47pp, +0.57pp], from 61,000 exposed users per arm in fct_experiment_exposure joined to dim_user and fct_event. Baseline activation is 32%. The launch team asks you to drop every surface except mobile_web, where the point estimate is +1.1pp, and re-run. You have ten minutes in their planning meeting. Deliver a spoken position: what you will and will not do, and the decision you recommend.

Approach

1. Recognise what is being probed: whether you hold a statistical position under social pressure without becoming either rigid or apologetic. A generic answer says the segment is not significant; a strong one separates the request into a question that is answerable (is the mobile_web number real?) and one that is not (can we ship on it?), and answers both.
2. Price the multiplicity out loud. The slice was chosen after seeing the results, so its estimate is selected on favourable noise and is biased away from zero. With k independent looks at a nominal 5% level, the chance of at least one false positive is 1 - 0.95^k: 26% at six segments, 64% at twenty. Quote the k you actually inspected, not the k you reported.
3. Use the arithmetic already in front of you. On the point estimates, a +1.1pp mobile_web effect combined with a pooled +0.05pp implies the remaining surfaces average negative in proportion to mobile_web's share of exposures. State that as a testable implication of their story rather than as a rebuttal of it.
4. Ask the one question that settles the category: was mobile_web named in the analysis plan before launch? If it was, it is a planned comparison and gets a corrected reading. If it was not, it is a hypothesis, and the honest move is to size the test that would confirm it.
5. Convert the refusal into a cost. Size a mobile_web-only confirmatory test at the claimed effect, state the weeks of mobile_web traffic it needs, and close with the recommendation: do not ship this as a lift, and note that the interval already rules out anything at or above +0.6pp, which is itself a useful input to the roadmap.

Follow-up

• The confirmatory test you sized needs nine weeks of mobile_web traffic and the team has three. What do you recommend instead?
• Suppose mobile_web was pre-registered. How does your reading change, and what correction do you apply?
• Your interval excludes +0.6pp. Is that the same as saying the feature does nothing?