ByteDance Data Scientist Interview Questions & Guide 2026

Interviewing at ByteDance

• Practice bank: 26+ questions
• Rounds: 3
• Typical prep: 2–4 weeks
• Interview reports: 4

Interview rounds

Round	Questions
Online Assessment	1
Technical Screen	22
Onsite	3

How hard is the ByteDance Data Scientist interview?

Difficulty	Share	Questions
Easy	12%	3
Medium	65%	17
Hard	23%	6

What ByteDance actually tests for

Topic	Share	Questions
Machine Learning	35%	9
Coding & Algorithms	23%	6
Data Manipulation (SQL/Python)	15%	4
Statistics & Math	12%	3
Analytics & Experimentation	8%	2
Behavioral & Leadership	8%	2

The 60-second brief

At ByteDance, the Data Scientist role is at the center of products that reach over a billion monthly active users globally, including TikTok, TikTok Shop, and a vast ecosystem of enterprise and content platforms. Unlike traditional analytics roles that focus solely on reporting, a Data Scientist at ByteDance is a core driver of product strategy, algorithm optimization, and business growth. You will operate at the intersection of product intuition, advanced statistical modeling, and large-scale data engineering.

ByteDance candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

What a Data Scientist does at ByteDance

The scale and operational pace of ByteDance create an environment where data insights directly dictate product decisions within days—or even hours. Whether you are analyzing user engagement algorithms for short-form video, optimizing supply chain logistics and merchant growth for TikTok E-Commerce, detecting risk and fraud within Trust & Safety, or designing evaluation frameworks for large language models (LLMs), your work directly impacts user retention and multi-billion-dollar revenue streams.

To succeed as a Data Scientist at ByteDance, you must balance rigorous quantitative methodology with ruthless prioritization. You will be expected to tackle complex, highly ambiguous problems, move rapidly through iteration cycles, and communicate technical insights clearly to cross-functional stakeholders across global engineering and product hubs.

The ByteDance interview process

ByteDance candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

HR Screen

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What it evaluates

• Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
• Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
• The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

• Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
• Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
• Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Technical Rounds

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What it evaluates

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Take-home Case Study

A take-home is graded as an argument, not as a notebook. Somebody reads the submission without you in the room, so every choice has to survive on the page: why the question was framed this way, and what was deliberately left out. The gap between a strong and a weak submission is almost never model quality. It is whether the writeup names the specific question it answers and commits to a recommendation, including what evidence would overturn it. A high-accuracy model attached to no conclusion reads as effort that stopped before the decision.

What it evaluates

• Whether the question you answered is stated outright, and whether it is the question the prompt posed rather than an easier neighbour of it
• Whether the recommendation is specific enough to act on, with the uncertainty attached to it instead of parked in a caveats section at the end
• Whether analytical choices such as the metric definition, the population filter and the time window are justified in the prose, not merely visible in code

How to prepare

• Take a dataset you have already worked with, write the one-paragraph conclusion first, then check whether the analysis you were planning actually supports it and cut whatever does not
• Practise stating a metric in one sentence that fixes the population, the time window and the denominator, then confirm your query computes exactly that sentence and nothing adjacent to it
• Hand a draft to someone outside the problem and ask them to tell you back what you recommended and why; anything they cannot recover is not on the page yet

Live Coding Challenge

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What it evaluates

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Final Round

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What it evaluates

• Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
• Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
• Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

• Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
• Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
• Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
• Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Questions ByteDance candidates report

Product sense and metrics

Choosing a primary metric is a commitment, and the interesting part is what you give up. The measure that best captures long-run value usually moves slowly, in few users, with high variance, so it cannot settle the question inside the window you have. The measure that moves quickly is usually a step on the way to the thing you care about, and it can rise while the real outcome falls. Say which trade you are making and why. Then choose guardrails against your own metric: for each way the primary could rise without the product getting better, name the number that would catch it.

• What metrics would you track to evaluate the success of a new search filtering feature in TikTok Shop?
• How do you define and measure "user engagement quality" beyond simple daily active users (DAU) or total session duration?
• How would you measure the long-term impact of increasing ad load on creator retention and user engagement?
• How would you estimate TikTok's daily advertising revenue from scratch without access to internal metrics or company data?

SQL and data manipulation

A window function earns its place when a question needs a per-row value that depends on other rows without giving up the rows themselves: rank inside a group, the previous value, a running total, a share of the group total. A grouped aggregate collapses, a window keeps. The detail most answers miss is the frame. Adding ORDER BY inside OVER switches the default frame to everything from the start of the partition through the current row, and that default is range-based in standard SQL, so rows tied on the ordering value all enter together. If ties matter, write ROWS yourself.

• Write an efficient query to determine the retention rate of first-time buyers on TikTok Shop day-1, day-7, and day-30 post-registration.
• How do you construct a query to deduplicate user interactions and identify the first and last touchpoints in a conversion funnel?

A/B testing and experimentation

Start every answer by naming the unit you randomise and the unit you analyse, and make sure they are the same thing. A test assigned by user but analysed per page view has many correlated rows inside each arm, so treating those rows as independent makes the standard error too small and the result looks more certain than it is. Splitting the other way fails differently: randomise by page view and one person lands in both arms, which contaminates the comparison and pulls the measured effect toward zero rather than flattering it. If the metric is a ratio whose denominator varies by user, say so and reach for the delta method or a bootstrap at the user level. Weak answers jump straight to the statistical test. Strong ones fix the unit, and the test follows from it.

• How do you determine the required sample size and duration for an A/B test on a highly active user feed with strong day-of-week seasonality?
• If an A/B test shows a 3% increase in video views but a 1.5% decrease in user comments, how do you decide whether to roll out the feature?

Statistics and probability

Nearly every question in this category is about a quantity you cannot observe, estimated from a sample that could have come out differently. So the first move is to name the estimand out loud: a mean, a proportion, a difference of two means, a ratio. Then say what its sampling distribution looks like and where that shape comes from. The split between strong and weak answers is almost always here. A weak answer describes how the data are spread. A strong one describes how the estimate would vary across repeated samples, and never lets those two standard deviations blur together.

• What mathematical approach would you take to predict spatio-temporal trajectories for express shipping package delivery?

Machine learning and modelling

A model that clearly improves offline can move nothing online, and the useful question is why. The candidate set feeding it may already exclude the cases it would have caught. The score may enter a surface where a cap, a fallback or a business rule blunts it. Training labels may describe a population the live traffic no longer matches, because they were collected under an earlier policy. Treat the offline number as a filter rather than as proof, plan the launch as a measured rollout with a guardrail and a rollback, and say which of those gaps you would check first.

• How would you design a seller tiering and outreach algorithm to improve merchant growth on an e-commerce platform?

Behavioural and stakeholder questions

Ownership questions usually arrive disguised as something else: a project that slipped, a pipeline that failed quietly, a number people had been reading wrong for months. What is being tested is what you did in the window before anyone told you it was yours. Be concrete about the first hour. Who you told, and what you checked before you knew how bad it was. The answer that falls short is the one that jumps from discovery straight to resolution and skips the part where you sized the damage, because sizing it is the decision everything else depended on.

• Describe a project where your analytical insights directly challenged a product manager's hypothesis. How did you resolve the disagreement?

Practice exercises with worked solutions

The prompts above tell you what gets asked.

Compute new-member activation using member-local calendar days

Using dim_member(member_id, registered_at_utc, tz_offset_minutes, account_status, country_code) and qualified_session(session_id, member_id, started_at_utc), compute weekly activation: the share of a registration cohort with a qualified session on at least 3 distinct member-local calendar days within days 0 to 6 after registration. Days are member-local, not UTC. The denominator is every registration in the cohort week, including accounts later suspended or deleted. Return cohort week, registrations, activated and rate. Return the same rate computed on UTC days alongside it so the gap is visible.

How a strong candidate works it

• Build the cohort from dim_member with no account_status filter. Suspension and deletion happen after registration, so filtering on them conditions the denominator on a post-treatment outcome and removes exactly the accounts least likely to have activated, which raises the rate for a reason that has nothing to do with the product.
• Shift both registration and session timestamps by tz_offset_minutes before taking a date, then subtract. Doing the subtraction on UTC timestamps and shifting afterwards gives a different day index at the boundary for any member with a non-zero offset.
• Restrict to a local day index of 0 through 6, then COUNT(DISTINCT local session date) per member and test it against 3. Counting sessions instead of distinct days lets one long evening satisfy a rule that exists precisely to require repeat visits.
• Report only cohort weeks whose day-6 local window has closed for every member. The most recent cohort otherwise reads low for a purely mechanical reason, and someone will read that as a regression.
• Run the same pipeline with the offset forced to zero and report both rates. The gap is not noise: it is concentrated in members whose local evening straddles UTC midnight, so its size and even its sign vary by country and it cannot be applied afterwards as a constant correction.
• Print registrations and activated counts beside the rate. A cohort-week rate moves when acquisition mix moves, and the two counts are what let a reader tell those apart.

Expected result (40 min): One row per cohort week with registrations, activated count, local-day rate and UTC-day rate. The two rates differ, and the difference is larger for cohorts concentrated in large-offset countries; its sign depends on whether that country's evening usage straddles UTC midnight, so it is not a constant correction.

Check yourself

• Registrations per cohort week equals COUNT(*) of dim_member registrations in that week, with no account_status filter applied anywhere in the query.
• Activated never exceeds registrations for any cohort week.
• Constructing a member with sessions on exactly 2 distinct local days yields an activated flag of 0, and adding a third distinct day flips it to 1.
• Restricting to members with tz_offset_minutes = 0 makes the local-day and UTC-day rates identical; if they still differ, the day-index arithmetic is shifting one side of the subtraction and not the other.

Follow-ups

• Activation fell 2 points in one week while the paid social share of registrations rose 8 points. What do you run next, and what would convince you it is mix rather than product?
• A member registers at 23:50 local time. How many local days does your 0-to-6 window actually give them, and is that the behaviour you want?
• Would you rather define activation on 3 distinct days or on total qualified sessions in the window, and what evidence would change your mind?

Write a data-quality audit over impression and engagement logs

You are given impressions (impression_id, event_date, viewer_member_id, content_id, rank_position, dwell_ms, served_at_utc), engagements (event_id, actor_member_id, content_id, impression_id nullable, action_type, occurred_at_utc, undone_at_utc nullable) and content (content_id, author_member_id, created_at_utc, enforcement_state, enforcement_at_utc). Write audit(impressions, engagements, content) returning one row per check with the failing row count and share. Cover at least six checks across uniqueness, value range, referential integrity and timestamp ordering. Mark each check block-the-refresh or log-only, and say which NULLs are by design rather than defects.

How a strong candidate works it

• Group the checks by failure class first, then write one vectorised expression per check, so the function stays a table of (name, predicate) rather than a wall of conditionals. Every check returns a boolean mask over one frame and the report is built by summing masks.
• Uniqueness and range: duplicated impression_id; rank_position < 1; dwell_ms < 0 or null; served_at_utc in the future relative to the partition date.
• Referential integrity: engagement rows whose non-null impression_id is absent from impressions; impression rows whose content_id is absent from content. Then the sharper one, which is what catches real pipeline breaks: engagement rows joined to their impression where viewer_member_id does not equal actor_member_id, or the two content_id values disagree.
• Timestamp ordering: undone_at_utc earlier than occurred_at_utc; engagement occurred_at_utc earlier than its impression's served_at_utc; content created_at_utc later than an impression of it.
• Separate by-design NULLs from defects explicitly. impression_id is NULL for follow, unfollow and profile_click, and for any deep-linked action, so its null rate is a distribution to monitor, not a check to fail. session_id is NULL for server-side actions. duration_seconds is NULL for non-video media.
• Classify severity by what the number is used for: referential mismatches between actor and viewer corrupt every rate denominated in impressions and should block; a small orphan-FK share from late-arriving partitions is log-only with a threshold and a trend.

Expected result (25 min): A DataFrame with one row per check giving check name, severity, failing rows, total rows and failing share, plus an informational table of impression_id null rate by action_type showing near 100 percent for follow, unfollow and profile_click and a low, stable rate elsewhere.

Check yourself

• Inject 10 known-bad rows of each kind into a copy of the input and confirm each check reports exactly 10.
• The total row count reported per check matches the frame that check runs over, so a share is never computed against the wrong denominator.
• On clean input every check reports zero except the informational null-rate block, which is not a check.
• The actor-versus-viewer mismatch check runs only over rows with a non-null impression_id, so it cannot be inflated by by-design NULLs.

Follow-ups

• The orphan rate on engagement to impression is 0.4 percent and all of it is within two hours of the partition boundary. Block or log?
• enforcement_at_utc is later than impressions already served for that content. Is that a defect, and what does it do to a query you ran last week?
• How would you turn this audit into something that fails a pipeline rather than producing a report nobody opens?

Choose a randomisation unit when reposts travel along the graph

A one-tap repost affordance is proposed for home_feed. Treated members are expected to repost more, and fct_follow_edge shows the median member follows 180 active accounts. A balanced partition of the active follow graph yields clusters averaging 40 members, with an intra-cluster correlation of 0.02 on the primary metric. Deliver the randomisation unit you would use, the sign of the bias if you randomise individual members instead, and the power cost of your design stated as a change in minimum detectable effect at fixed traffic.

How a strong candidate works it

• Name the leakage channel explicitly: the treatment output is content, and content reaches followers through fct_follow_edge regardless of the follower's arm. A control member who follows treated members receives part of the treatment, so the control mean rises and the measured difference understates the true effect.
• Say when the sign flips. For a feature that adds supply, interference attenuates the estimate toward zero. For a feature that reallocates a fixed number of slots, treated members win attention that control members lose, the control arm is pushed down, and the estimate is inflated.
• Randomise clusters of the graph: a balanced partition or ego-network design, assigned whole, so that most of a member's followees share their arm.
• Price the design: design effect = 1 + (m - 1) x rho with m = 40 and rho = 0.02. With unequal cluster sizes the cost is higher, approximately 1 + ((CV^2 + 1) x m - 1) x rho, and social graph partitions always have a heavy size tail.
• Estimate variance at the randomisation unit: cluster-level means weighted by size, or a member-level model with cluster-robust standard errors. With fewer than roughly 40 clusters per arm, fall back on CR2 with G - 1 degrees of freedom or randomisation inference.
• Report containment rather than assuming it: the share of each member's active followees that sit in the same cluster, plus an exposure-response cut on fraction of neighbours treated, which bounds the residual leakage the partition did not remove.

Expected result (30 min): Design effect = 1 + 39 x 0.02 = 1.78. At fixed traffic the MDE grows by sqrt(1.78) = 1.33, so a 1.0 percent detectable effect becomes 1.33 percent. Holding the MDE fixed instead, you need 1.78 times the members: about 171,000 per arm, which is roughly 4,270 clusters of 40 per arm. Randomise clusters and accept that cost; individual randomisation here attenuates the estimate toward zero.

Check yourself

• Setting rho = 0 or m = 1 must return a design effect of exactly 1, which is the sanity check on the formula.
• The cluster-level analysis and a member-level regression with cluster-robust errors should return the same point estimate and similar intervals; a large gap means the weighting is inconsistent.
• The containment share should be well above 0.5 for the partition to be worth its power cost; if most followees sit outside the cluster, the design bought little.
• Recomputing the design effect with the observed cluster-size coefficient of variation should not move it by more than a few percent, or the balanced partition was not balanced.

Follow-ups

• When would a switchback beat a cluster design here, and what property of the repost feature breaks a switchback?
• Your partitioner can only produce 30 clusters per arm. What changes in the analysis?
• How would you estimate the spillover effect itself rather than just eliminating it from the direct estimate?

Hide rate doubled overnight: content regression or affordance change

Negative feedback per 1,000 home_feed impressions rose from 3.1 to 5.8 in one day and stayed there. A client release went out the same day on one platform only. Using fct_engagement_event (action_type, is_negative_feedback, impression_id, surface, event_date) and fct_feed_impression (event_date, surface, client_platform, app_version, dwell_ms, session_id), decide whether content quality changed or the measurement did. Deliverable: a verdict, the evidence that separates the two causes, and a statement of what this rate is now comparable to.

How a strong candidate works it

• Split the ratio before interpreting it. Plot the numerator (negative feedback events) and the denominator (impressions) as separate daily series by client_platform. A rate doubles just as easily because the denominator halved, and this costs one query.
• Cut both series by app_version rather than only by platform. A staged rollout produces a rate that tracks adoption share day by day, which is a fingerprint no content change leaves.
• Use the platform that did not ship as a control. It draws from the same content pool over the same days, so a flat series there is strong evidence the content did not change.
• Split the numerator by action_type. Relocating one control moves that action alone, while a genuine quality regression moves hide, report, not_interested and unfollow together, with report typically lagging rather than stepping.
• Interrogate the denominator definition. An impression counts once the item held half the viewport for 250 ms, so a render or scroll change can alter what qualifies. Compare impressions per session and the dwell_ms distribution across the release boundary.
• Close on comparability: this rate is a function of how reachable the control is, so post-release values can only be compared to post-release values. Re-baseline the alert threshold and record the discontinuity in the metric definition.

Follow-ups

• The UI change is permanent. How do you set a new alert threshold without waiting a full seasonal cycle?
• If hides really did rise because the control is easier to reach, is that extra signal good or bad for the ranker that consumes it?
• What would convince you that content quality also degraded underneath the affordance change?

Choose between three requests with one analyst week

You have one analyst week and three open requests. Ranking wants an off-policy estimate of a candidate ranker, but log_propensity is NULL on 92 percent of home-feed impressions. Growth wants a week-1 retention read on a registration cohort whose day 7 to 13 window closes in four days. Integrity wants the audit sample re-sized because the prevalence interval cannot detect a 20 percent change. Give the order, what each requester gets and when, and the request you refuse outright with the reason.

How a strong candidate works it

• Sort by identification before sorting by importance. A request that cannot be answered correctly with any amount of effort is not competing for the week; it is competing for a different conversation.
• Rule out the off-policy estimate on positivity grounds and say it in one line: log_propensity NULL means serving was deterministic top-k, so the unserved slots had zero logging probability, and no inverse-propensity or doubly-robust estimator is identified there. The fix is a randomised exploration slot or an online test, which is a platform change and not an analyst task.
• Rule out the retention read on timing: days 7 to 13 must have closed for every member in the cohort. Reading it four days early truncates the window for the late-registering half of the cohort and biases the estimate by registration day, which correlates with acquisition channel.
• Do the integrity re-size, because it is tractable in hours and unblocks a standing measurement. At a two-sided alpha of 0.05 and 80 percent power, detecting a move from 0.50 percent to 0.40 percent needs roughly 70,000 labelled impressions per comparison period under simple random sampling, before any design effect from unequal weights. That number reframes the request: the honest deliverable may be a stratified design plus a longer comparison window, not a bigger weekly sample.
• Convert each refusal into a dated deliverable so the week does not end in three unhappy teams: ranking gets the exploration-slot design and the randomised fraction it needs; growth gets the date the cohort closes plus the activation rate on days 0 to 6, which is already readable and is a leading indicator of the number they want.

Follow-ups

• Ranking says a biased estimate is better than none. Answer them in two sentences.
• How large an exploration fraction would you request, and what does it cost in served quality?
• If integrity cannot fund 70,000 labels a week, what do you offer instead?

Explain a prevalence interval to a non-technical executive

A weekly impression-weighted violating-content prevalence estimate came in at 0.42 percent, 95 percent interval 0.28 to 0.61, against 0.51 percent (0.35 to 0.72) the week before. The audit sample is 4,000 served impressions drawn with unequal, recorded selection probabilities across risk strata, labelled by humans against written policy. An executive asks whether the number went down and wants one figure for a board slide. In five minutes: answer the question, say what goes on the slide, and state what you would need to give a sharper answer next quarter.

How a strong candidate works it

• Answer the question in one sentence before explaining anything: the point estimate is lower, the intervals overlap across most of their range, and the week-over-week change is not distinguishable from zero.
• Show why with one arithmetic step rather than vocabulary. At n = 4,000 and p near 0.004 the simple-random-sampling standard error is sqrt(p(1-p)/n), about 0.10 percentage points, so an SRS interval would run roughly plus or minus 0.20 points and a 0.09 point move sits well inside it. Two facts about the reported interval belong in your head rather than on the slide. Its asymmetry comes from the construction, not from the weights: Wilson, Clopper-Pearson and logit intervals are built on a bounded scale, so near p = 0 the upper limit sits further from the point estimate than the lower one. The 1/p_i weights act on width only, through a design effect that multiplies the variance. Here the reported width of 0.33 points implies a standard error near 0.085 (0.33 divided by 3.92), so the design effect is about 0.7, which is what oversampling high-risk strata buys when selection probability correlates with the outcome. Uninformative weights would instead give a design effect of 1 + CV squared of the weights, above 1, and an interval wider than the SRS one rather than narrower.
• Replace the bare point estimate with a number that is stable at board cadence: the trailing four-week pooled estimate, formed by re-summing the weighted numerator and the weighted denominator across weeks. Averaging the four weekly rates gives a different and wrong number when weekly sample sizes differ.
• Price the precision the executive is implicitly asking for. Halving the interval width needs roughly four times the labelled sample, so 16,000 labels a week to go from a half-width near 0.17 points to one near 0.085. The cheaper lever is allocation rather than volume: the design already uses unequal, recorded, strictly positive selection probabilities and is already running a design effect near 0.7, so re-fitting the strata on current classifier scores and moving more of the 4,000 into the strata carrying the violating mass pushes that number down further without a fourfold labelling bill.
• State plainly what this number is not, because the executive will meet substitutes. Report volume and enforcement volume are member and operations behaviours; they can fall while prevalence rises if the ranker gets better at matching violating content to receptive audiences.

Follow-ups

• The executive wants a weekly trend line on the slide anyway. What do you draw, and what do you label the band?
• How long would it take to detect a 20 percent reduction in prevalence at the current sample size?
• Why not score every impression with the classifier instead of paying for human labels?

Where candidates lose points

Crediting notifications with the sessions that follow them

Members who open a push notification were already more likely to open the app, so attributing their session to the notification measures intent rather than causation and makes almost any send look profitable. The predictable result is a push-volume increase that shows a large modelled gain and a small real one, paid for later in opt-outs and uninstalls that no single experiment window captures. The only defensible estimate compares a stable send arm against a holdout arm assigned at the decision point, with the held-out decisions logged and suppressed at delivery, over a window long enough to include the opt-out response. Frequency effects are non-linear, so a per-notification incremental rate estimated at one volume does not extrapolate to a higher one.

Randomising individual members when the treatment travels along the social graph

If treated members post, comment or share more, their followers see the extra content whether or not those followers are treated, so the control group is partly treated and the measured difference understates the true effect. For features that redistribute a fixed amount of attention, the leakage runs the other way and the effect is overstated. The size of the bias scales with how dense the neighbourhood is, so it is largest exactly among the connected members whose behaviour the feature was built for. The fix is randomising clusters of the graph (ego networks or communities found by balanced partitioning), clustering the variance at that unit, and accepting that effective sample size falls by roughly the average cluster size.

Over-explaining the method and under-explaining the implication

Lead with the answer and what you would do about it, then give the approach when asked. Roughly one sentence of method per three of implication is the right ratio for a stakeholder-facing answer; the interviewer already knows what a regression is.

Ignoring interference between units in a marketplace experiment

Ask whether one unit's treatment can change another unit's outcome through shared inventory, a matching pool, a social graph or a common budget. Where it can, randomise at a level that contains the spillover, such as region or time slice, and say explicitly what that costs you in statistical power.

Experimentation heavy, for an analytics-leaning loop

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.