Adobe Data Scientist Interview Questions & Guide 2026

Interviewing at Adobe

• Practice bank: 5+ questions
• Rounds: 1
• Typical prep: 1–2 weeks

Interview rounds

Round	Questions
Onsite	5

How hard is the Adobe Data Scientist interview?

Difficulty	Share	Questions
Easy	40%	2
Medium	60%	3
Hard	0%	0

What Adobe actually tests for

Topic	Share	Questions
Analytics & Experimentation	40%	2
Behavioral & Leadership	20%	1
Data Manipulation (SQL/Python)	20%	1
Statistics & Math	20%	1

The 60-second brief

At Adobe, a Data Scientist occupies a strategic role at the intersection of advanced quantitative modeling, product strategy, and user experience engineering. Data scientists across Adobe Creative Cloud, Adobe Express, Adobe Firefly, and Adobe Experience Cloud transform petabytes of unstructured behavioral data into actionable insights and intelligent product features. Whether optimizing subscription conversion funnels, building machine learning models to power generative workflows, or sizing growth headroom for enterprise sales strategies, data scientists directly influence how millions of creators and enterprises interact with digital media daily.

Adobe candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

What a Data Scientist does at Adobe

The Data Scientist role at Adobe is heavily biased toward product analytics, behavioral statistical modeling, and experimental rigor. You will collaborate closely with product managers, growth marketers, engineering teams, and executive leadership to answer complex, unstructured questions. Candidates are evaluated not only on their technical ability to write clean code or construct predictive models but also on their commercial intuition, data storytelling capabilities, and ability to translate statistical output into high-impact product roadmap decisions.

Joining Adobe as a Data Scientist offers the opportunity to tackle multi-touch attribution, user retention modeling, generative model evaluation, and large-scale experimentation. You will design, evaluate, and scale algorithmic solutions that touch flagship applications like Photoshop, Illustrator, and Acrobat, driving tangible top-line monetization and shaping the future of digital creativity.

The Adobe interview process

Adobe candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Phone Screen

A screening call is a matching exercise run by someone who will not evaluate your statistics. They are checking that the work described on your resume is work you personally did, and that its scope matches the level the role is written for. Logistics get settled in the same half hour so nobody spends an interviewer's afternoon on a mismatch. The answer that fails is the one narrated in the plural. If every sentence is 'we built' and 'the team decided', there is nothing specific to write down about you. Name the piece that was yours, the decision you made inside it, and what changed after.

What it evaluates

• Whether the ownership implied by your resume survives one round of follow-up about who actually did which part
• Whether your described scope (data size, stakeholders, what shipped) matches the seniority the role is written at
• Whether timeline, location and compensation expectations make the rest of the loop worth scheduling

How to prepare

• Rewrite your top three resume bullets in the first person singular, each with the decision you made and what moved afterwards, then say them out loud once so the 'we' does not return under pressure
• Attach one number to each project: the baseline, the change, and the window it was measured over. Where impact was never measured, say that plainly rather than inventing a figure
• Settle your compensation range before the call and give it as a range with a reason behind it, such as current total comp or a competing timeline, instead of deflecting the question twice

Online Technical Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What it evaluates

• Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
• Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
• Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

• Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
• Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
• If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Team-Specific Technical Screens

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What it evaluates

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Formal Past-Work Presentation

An added round often puts you in front of someone outside the core hiring team: a partner engineer, a product owner, a domain expert, sometimes a more senior manager. The question they are really asking is not whether you can do the work but whether they would trust a number that came from you. That changes what a good answer looks like. Lead with what the decision cost and what it changed, keep the method available but not central, and be plain about the limits of your evidence. Overstating a result is the fastest way to lose this round.

What it evaluates

• Whether you can explain a technical choice to someone who will never read your code, without either flattening it into nothing or hiding inside jargon
• Honesty about evidence strength: what the analysis establishes, what it only suggests, and what it cannot say at all
• How you take disagreement, specifically whether you update on a good objection, hold your position with reasons, or fold on contact

How to prepare

• Write the two-sentence version of your most technical project for a non-specialist, then check that neither sentence needs a method name to make sense.
• For one result you are proud of, write the strongest objection someone could raise and a response that concedes the part of it that is correct.
• Prepare one decision that turned out to be wrong: how you found out, what it cost, and what you changed afterwards. A senior cross-functional interviewer asks for this more often than a technical one does.

Final Loop Rounds

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What it evaluates

• Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
• Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
• Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
• Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

• Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
• Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
• Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Questions Adobe candidates report

Product sense and metrics

Every question in this group starts from the same move: name the outcome the product is supposed to produce, then name the number that would show it happening. Work forward from the mechanism, not backward from a metric you already like. Ask what behaviour changes, for whom, and how soon, and only then pick the measure that sits closest to that behaviour without being trivially satisfied by it. What usually separates answers is the counterfactual. A candidate who says what result would make them call the change a failure has defined success; one who only describes what going well looks like has not.

• Imagine weekly active users (WAU) for Adobe Acrobat drop by 8% over a two-week period. Walk through your systematic root-cause diagnosis.
• How would you evaluate the success of a newly introduced generative AI editing feature in Adobe Express?
• How would you measure feature engagement and long-term user retention for trial subscribers converting to paid Creative Cloud plans?
• Define a comprehensive framework to segment Adobe Photoshop power users versus casual creators, and explain how you would use this segmentation to drive feature recommendations.

SQL and data manipulation

Funnels and retention read as counting problems and behave as time problems. Step two has to follow step one for the same user, inside a window someone has to define, and a cohort clock usually starts at each user's own first event rather than at a calendar boundary. Before writing, pin down which timestamp is event time rather than load time, whether your interval is half-open so a midnight row lands in exactly one bucket, and what time zone the reporting day belongs to. The usual failure is a query that looks right and whose day boundaries quietly belong to a different zone than the number it is compared against.

• Given a table of daily user login timestamps and action logs, write a query to compute the time elapsed between a user's first trial sign-up and their first paid transaction.
• Write an efficient SQL query using DENSE_RANK and LAG to detect continuous 5-day active usage streaks among Adobe Express creators.

A/B testing and experimentation

Start every answer by naming the unit you randomise and the unit you analyse, and make sure they are the same thing. A test assigned by user but analysed per page view has many correlated rows inside each arm, so treating those rows as independent makes the standard error too small and the result looks more certain than it is. Splitting the other way fails differently: randomise by page view and one person lands in both arms, which contaminates the comparison and pulls the measured effect toward zero rather than flattering it. If the metric is a ratio whose denominator varies by user, say so and reach for the delta method or a bootstrap at the user level. Weak answers jump straight to the statistical test. Strong ones fix the unit, and the test follows from it.

• How do you determine the required sample size and exposure duration for an experiment where the baseline conversion rate is very low?
• What primary experimentation pitfalls (such as novelty effects, network interference, or sample ratio mismatch) would you monitor when testing a new AI workflow in Adobe Express, and how would you mitigate them?

Statistics and probability

Nearly every question in this category is about a quantity you cannot observe, estimated from a sample that could have come out differently. So the first move is to name the estimand out loud: a mean, a proportion, a difference of two means, a ratio. Then say what its sampling distribution looks like and where that shape comes from. The split between strong and weak answers is almost always here. A weak answer describes how the data are spread. A strong one describes how the estimate would vary across repeated samples, and never lets those two standard deviations blur together.

• How would you explain the difference between Type I and Type II errors in the context of fraud detection for Adobe digital subscriptions?

Machine learning and modelling

A model that clearly improves offline can move nothing online, and the useful question is why. The candidate set feeding it may already exclude the cases it would have caught. The score may enter a surface where a cap, a fallback or a business rule blunts it. Training labels may describe a population the live traffic no longer matches, because they were collected under an earlier policy. Treat the offline number as a filter rather than as proof, plan the launch as a measured rollout with a guardrail and a rollback, and say which of those gaps you would check first.

• How would you design a propensity model to predict which Adobe trial users are most likely to churn before their first billing cycle?

Behavioural and stakeholder questions

Ownership questions usually arrive disguised as something else: a project that slipped, a pipeline that failed quietly, a number people had been reading wrong for months. What is being tested is what you did in the window before anyone told you it was yours. Be concrete about the first hour. Who you told, and what you checked before you knew how bad it was. The answer that falls short is the one that jumps from discovery straight to resolution and skips the part where you sized the damage, because sizing it is the decision everything else depended on.

• Explain statistical significance and $p$-values to a non-technical product stakeholder. What does a 95% confidence interval actually communicate?

Practice exercises with worked solutions

The prompts above tell you what gets asked.

Find paid seats that never called the API

dim_user_membership carries user_id, account_id, seat_type, is_service_account, deactivated_at and last_seen_at. fct_api_request carries user_id, which is null whenever the caller is a service account or an unattended key, plus account_id, request_at and http_status. For a single account, list every licensed_paid seat with is_service_account = false and deactivated_at null that issued no request at all in the trailing 90 days, returning user_id and last_seen_at. A first draft writes NOT IN against a subquery over fct_api_request.user_id. State exactly what that draft returns and why, then write the correct query.

How a strong candidate works it

• Read the column comment first: user_id is nullable on the fact, so the subquery almost certainly contains at least one NULL for any account that runs a service account or an unattended key.
• Work the three-valued logic out loud. x NOT IN (a, NULL) expands to NOT (x = a OR x = NULL); the second disjunct is UNKNOWN, so for any x not equal to a the whole predicate is UNKNOWN and the row is filtered out. The draft returns zero rows, which reads as full seat utilisation.
• Rewrite as NOT EXISTS with the 90-day and status predicates inside the correlated subquery. Putting them in the outer WHERE instead turns the anti-join into a different question and silently changes the answer.
• Correlate on both user_id and account_id. A membership is (user_id, account_id) and one person can hold memberships in several accounts, so correlating on user_id alone marks a seat as active because that human was busy somewhere else.
• Filter the seat side to seat_type = 'licensed_paid', is_service_account = false and deactivated_at IS NULL, and say what the resulting count means next to contracted_seats on the current subscription row.

Expected result (15 min): A list of user_id and last_seen_at for licensed, non-service, active memberships on that account with no request in 90 days. The NOT IN draft returns zero rows whenever the account has any service-account or unattended-key traffic, and returns the right answer only by accident on accounts that have none.

Check yourself

• The NOT EXISTS result and the LEFT JOIN ... IS NULL result return identical user_id sets.
• Idle seats plus seats with at least one request in 90 days equals the total licensed non-service active membership count for the account.
• Forcing a NULL into the subquery (union a SELECT NULL) leaves NOT EXISTS unchanged and empties the NOT IN version, which demonstrates the failure on demand.

Follow-ups

• Adding AND user_id IS NOT NULL to the subquery also fixes NOT IN. Why is NOT EXISTS still the form you would leave in the repository?
• last_seen_at looks like a shortcut for the whole query. What does it actually record, and where does it disagree with API activity?
• This is a seat-reduction risk list. What threshold would you attach before handing it to an account team, and what happens to seats below it?

Implement seven-day activation rate from its written definition

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

How a strong candidate works it

• Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
• Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
• Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
• Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
• Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Expected result (20 min): A dict whose denominator is the count of distinct non-internal accounts created in that ISO week, numerator <= denominator, and rate = numerator / denominator; accounts with no qualifying request sit in the denominator with NaT and evaluate to False.

Check yourself

• len(m) == cohort.account_id.nunique(); the merge must not add rows.
• Dropping the synthetic_monitor filter can only raise the numerator, never lower it.
• Widening the window from 168h to 336h is monotone: the numerator must not decrease.
• An account whose only qualifying request falls at exactly created_at + 168h counts as activated, since the comparison is inclusive.

Follow-ups

• Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
• How would you decide whether 168 hours is the right window rather than 72 or 336?
• An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Pick the randomisation unit when workspaces and queues are shared

A collaboration feature was tested by randomising 18,000 users 50/50. dim_user_membership shows a median of 9 activated, non-service users per paying account, and weekly successful interactive requests per user carries an intra-class correlation of 0.25 within an account. The reported lift is 4% at p = 0.05. Separately, the same team wants to evaluate a change to the admission-control queue, which is shared by every account in a region. Give the correct randomisation unit for each test, the effective sample size and corrected p-value for the first, and the design for the second.

How a strong candidate works it

• Separate the two failures of user-level randomisation, because they need different fixes and only one of them is a variance problem. Interference: a treated colleague changes a control colleague's behaviour inside a shared workspace, which biases the estimate toward zero. Correlated outcomes: users inside an account are not independent draws, which understates the variance. Clustering the standard errors fixes the second and leaves the first entirely intact.
• Compute the design effect: 1 + (m - 1) * rho = 1 + 8 * 0.25 = 3.0. Effective sample is 18,000 / 3 = 6,000 users, corresponding to roughly 2,000 accounts. Standard errors are understated by sqrt(3) = 1.73, so a reported t of 1.96 is really 1.13 and the honest two-sided p is about 0.26, not 0.05.
• Re-run the first test randomised on account_id with inference clustered on account_id, or equivalently as a regression on account-level means. The account-means version is the conservative one and is usually easier to defend to a non-specialist audience, at the cost of some efficiency when cluster sizes are very unequal.
• For the queue change, account randomisation fails too, and for a different reason: treated and control accounts contend for the same finite capacity, so the control arm is mechanically affected by the treated arm and the contrast estimates a property of a mixed system rather than the effect of the policy. Randomise time instead, with a switchback alternating the admission policy at the region-by-30-minute-slot level.
• Specify the switchback so it is actually valid. Impose a washout at the start of each slot at least as long as the p99 queue drain time and discard requests enqueued before the switch; randomise slot order rather than strictly alternating, because a fixed alternation aliases with the hourly and weekday traffic cycle; block on hour-of-day so both policies see peak and trough; and cluster inference on the slot, which is the unit that was randomised.
• State the power consequence plainly: the cluster count is slots, not requests. Fourteen days of 30-minute slots gives 672 slots, and the variance to plan against is between-slot variance in the outcome, which is far larger than between-request variance and is what makes switchbacks expensive.

Expected result (30 min): Design effect 3.0, effective sample about 6,000 users or 2,000 accounts, corrected two-sided p about 0.26. The 4% lift is not significant once clustering is respected and is additionally biased toward zero by within-workspace spillover. The queue change is run as a randomised-order switchback with slot-level clustering and a drain-time washout, not as an account-level split.

Check yourself

• The design effect depends only on cluster size and intra-class correlation: it must collapse to 1 when rho = 0 and when m = 1. If your expression does not, it is not the design effect.
• An account-means regression and a user-level regression with cluster-robust standard errors should give similar standard errors. A large gap indicates very unequal cluster sizes, in which case the means version needs weighting.
• The switchback's sample size is slots, not requests. If your power calculation cites a sample in the millions, you have counted requests and will badly overstate precision.

Follow-ups

• The intra-class correlation was estimated at 0.25 from a 300-account pilot. If it is really 0.40 the design effect becomes 4.2. How do you plan under that uncertainty rather than betting on the point estimate?
• Under what conditions is user-level randomisation still the right choice even though users share accounts?
• The queue change is expected to leave a four-hour retry backlog. What does carryover of that length do to a 30-minute switchback, and what would you run instead?

Blended gross margin fell while every segment improved

Blended gross margin across paying accounts fell from 71 percent to 66 percent over two quarters, yet margin improved inside every plan_tier and every deployment_model. You have fct_usage_daily (account_id, sku_code, usage_date, net_amount_cents, cogs_cents), fct_subscription_period (account_id, plan_tier, term_start_date, term_end_date) and dim_account (account_id, deployment_model, is_internal, effective_from, effective_to). Produce an exact decomposition of the five-point move into mix, rate and interaction terms, and name the segment shift that carries it.

How a strong candidate works it

• Write blended margin as a revenue-weighted average of segment margins, m = sum over i of w_i * m_i, where w_i is the segment's share of net_amount_cents. Margin is a ratio of sums, so only revenue weights reproduce the blended figure.
• Apply the exact three-term decomposition: delta_m = sum(delta_w_i * m_i0) + sum(w_i0 * delta_m_i) + sum(delta_w_i * delta_m_i), read as mix, rate and interaction. It is exact by construction, so the three terms must sum to the observed change with no residual.
• Build segments from as-of joins: take the fct_subscription_period version whose term brackets each usage_date and the dim_account version whose effective_from and effective_to bracket it, rather than joining on is_current and backdating today's attributes over last year's usage.
• Run the decomposition on more than one segmentation, at minimum plan_tier, deployment_model and sku_code, since the mix that actually moved may not be the one anybody already suspected.
• Once the driving segment is identified, determine whether the shift came from acquisition landing new accounts in a lower-margin segment or from existing accounts migrating, because those have different owners and different remedies.

Follow-ups

• The interaction term is sometimes large. What does it mean operationally, and when would you prefer a log-mean decomposition that distributes it across the other two terms?
• Revenue-weighted average margin is one construction. What changes if leadership wants margin weighted by account count, and which question does each version answer?

Defend a churn number twelve times the one in the board deck

You recompute logo churn on the renewal-eligible base from fct_subscription_period, counting only accounts whose term_end_date fell in the month and allowing a 45-day grace for late paperwork. Annualised, about 14 percent of accounts that reach a renewal date do not renew. A revenue leader has been quoting 1.2 percent to the board for two quarters, computed by dividing non-renewed accounts by the entire customer base in each month and printing that monthly figure with no period attached. You have 20 minutes with that leader and the finance lead. Decide which number is reported from now on, and what happens to the two quarters already published.

How a strong candidate works it

• The interviewer is probing whether you can hold a correct definition under social pressure without turning it into a competence dispute. Open by reproducing their 1.2 percent exactly, with their denominator and their months, so the disagreement is arithmetic both sides can see rather than a claim about who was careless.
• Separate the two defects, because they are different in kind. The denominator is wrong: on annual contracts only about one twelfth of the base reaches a renewal date in any month, so an account eleven months from renewal sits in the denominator while being structurally incapable of entering the numerator, which suppresses the rate by a factor near twelve. The period is merely unstated: a monthly figure printed beside annual revenue targets gets read as an annual rate.
• Say out loud that those two defects nearly cancel in the level, before the leader finds it. Twelve times 1.2 percent is about 14 percent, which is your number. That is the strongest thing you can say in the room, because it proves both figures rest on the same non-renewal count and moves the meeting onto which denominator and which period get published rather than onto whose query is right.
• The level is recoverable; the series is not. Non-renewals in a month are the eligible base for that month times the churn rate, so dividing by a fixed whole base makes the published line proportional to how many contracts happen to come up that month. Where signings cluster at quarter ends, the eligible base in a quarter-end month can be several times a quiet month's, and the month-over-month moves the board has been reading as satisfaction are the signing calendar.
• Separate the measurement change from a business change. Nothing got worse this week; the loss rate was always this. Bring net revenue retention over the same period as a ratio of sums on a cohort frozen twelve months earlier, because logo churn concentrated in small accounts can sit beside healthy revenue retention, and that combination is the actual story.
• Offer a migration path rather than a correction. Report both rates for one quarter with a written bridge, restate the prior two quarters in an appendix instead of silently, and pin the definition, including the period it is stated over, somewhere finance and product both read it. Concede the limits of your own number: the 45-day grace means the most recent 45 days are not reportable, and churn must be dated on term_end_date rather than on updated_at. A strong answer volunteers this; a generic one only defends.

Follow-ups

• The leader multiplies their monthly figure by twelve, lands on your annual number, and concludes nothing was ever wrong. What do you say?
• The leader says publishing the corrected rate costs the team its credibility with the board this quarter. What do you do?
• Gross logo retention worsened while net revenue retention improved. Which do you lead with, and what does the combination tell you about who is leaving?

Announce a metric fix that cuts the headline number

Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.

How a strong candidate works it

• The interviewer is probing whether you can land a correction as an operational change with a plan attached, rather than as an announcement other people then have to clean up after.
• Quantify each exclusion separately before telling anyone: internal accounts, synthetic monitors, load tests. Three known quantities are a discussion; one alarming total is an argument.
• Be precise about which side of the metric each exclusion touches, because one team's goal is on a count and the other's is on a ratio. The traffic-class filters remove requests, so they shrink the numerator only. Dropping internal accounts removes them from the ratio's denominator as well, since internal accounts carry ordinary account_status values and therefore sit in that denominator. Internal accounts are active in almost every week while the real base is not, so the numerator loses a larger share than the denominator and the ratio falls by less than the count does. Compute both and say which one the 11 percent is before anybody assumes.
• Check whether the trend changes, not only the level. A constant 11 percent shift is a rebasing and nothing more. A shift that widens over time means the reported growth was partly internal or synthetic, which makes the existing goals unachievable as written and changes what you are asking teams to do.
• Sequence the disclosure: the metric owner and the two teams whose goals move first and privately, then the board channel with a written bridge, then the dashboard. The dashboard is last because a number that changes without explanation is read as instability rather than as a fix.
• Run both series for one reporting period with the bridge visible, restate history rather than letting the series break at a date, and set the date the old series is removed.
• Propose the goal treatment yourself: rebase each target by the shift measured on the metric that target is written against, rather than leaving each team to negotiate individually, which is where corrections of this kind usually die.

Follow-ups

• One team's quarterly goal is now unreachable. Rebase the target or let it miss, and what does each choice teach the organisation?
• How would this have been caught when the metric was first defined?
• What else on that dashboard shares this failure mode, and how would you find out this week?

Where candidates lose points

Reporting a mean over accounts when account revenue is heavy-tailed

When a small number of accounts hold most of the revenue, the sample mean is dominated by whichever of them happens to be in the sample, and the sample variance keeps growing as more data arrives instead of stabilising. In that regime the usual central-limit-based confidence interval understates uncertainty, and a single renewal or a single large account's batch job can flip the sign of a measured effect. The fixes are to pre-register a winsorisation or capping rule before looking at the outcome, to report account counts crossing a threshold alongside the revenue figure, or to define the estimand on a bounded transform. Choosing the cap after seeing the result is a separate and worse problem, because the cap then encodes the answer.

Reading consumption metrics before the metering lag window has closed

Usage pipelines land late and correct themselves, which is exactly what is_restated and restated_at record. A dashboard queried on day T sees a partially populated tail for the last several days, so the most recent points always slope downward and always look like a regression. Analysts then explain the artefact, and sometimes ship a change to fix it. Establish the empirical settling time by measuring how much a given usage_date's total moves between first_written_at and its final value, exclude that many trailing days from every reportable figure, and never compare a fresh period against a settled one.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Product sense first, for an engineer moving into data science

For someone who can already write the query and train the model but stalls when asked what to measure or whether a change is worth making. Metric definition and case structure come first; the technical work is kept as maintenance rather than the centre of the week.

Day 1 — Metric anatomy

• For three products you use daily, write one primary metric, two input metrics that plausibly move it, and one guardrail that would catch a cheap way of moving the primary at the cost of the product.
• For one of them, specify the metric precisely enough that two analysts would return the same number: numerator, denominator, unit of observation, time window, and how returning and deleted accounts are treated.
• Pick a ratio metric and write what happens to it when the denominator shrinks for reasons unrelated to the numerator, with a concrete example of that happening.

Deliverable: A one-page metric tree for one product, with the primary metric written as an unambiguous spec.

Day 2 — Diagnosing a drop without guessing

• Take the prompt "weekly active users fell 8 percent week over week" and write the segmentation plan before proposing any cause: platform, region, tenure cohort, acquisition channel, and whether the movement sits in the numerator or in a changed denominator.
• List the instrumentation failures that manufacture fake drops (a client release that stopped firing an event, a bot filter change, a shifted date boundary or timezone) and write the query that rules out each one.
• Rehearse stating the boring explanations first, seasonality and day-of-week composition, before reaching for a product cause.

Deliverable: A drop-diagnosis checklist short enough to recite from memory in under a minute.

Day 3 — Should we build it

• Take a feature idea and write it as a bet: what you believe is true, what would have to be true for it to pay off, the metric that would confirm it, and the effect size that would justify the engineering cost.
• Size the opportunity top-down and bottom-up, then reconcile the two numbers in writing instead of quoting whichever is friendlier.
• Write the counter-metric that would make you kill the feature even if it wins on the primary metric.

Deliverable: A one-page product memo ending in a decision rather than a list of considerations.

Day 4 — The places aggregate numbers lie

• Construct a Simpson's paradox numerically: two segments where the treatment wins within each segment yet loses overall, and identify the shift in segment weights that causes it.
• Take a heavy right-tailed quantity such as revenue per user and write why the mean is the wrong summary, which percentile you would report instead, and what a moving mean with a stable median tells you.
• Write your definition of a session for the product from day one, then name two real behaviours it misclassifies.

Deliverable: One page holding a worked Simpson's paradox table and a session definition with its two known failure cases.

Day 5 — Technical maintenance, aimed at metrics

• Solve four timed SQL prompts that all end in a ratio metric, so the question of grain stays live in every answer.
• Compute a 95 percent confidence interval for a proportion on a small sample, and state why the normal approximation is unreliable when either np or n(1 minus p) falls below roughly 10, along with which interval you would use instead.
• Take one metric from your day-one tree, write the query that computes it correctly, then write the query that computes it wrong in the most plausible way and explain how you would notice.

Deliverable: Four solved prompts plus a matched correct and plausible-wrong query for one metric.

Day 6 — Turning engineering work into data science stories

• Write three project stories as situation, decision, trade-off, outcome, each carrying one number and one thing you got wrong.
• For the story you will lead with, prepare an answer to "what would you do differently" that names a decision you made, not a constraint you were handed.
• Practise the sentence that reframes a systems project as a question project: the question the work answered, ahead of the pipeline it shipped.

Deliverable: Three written stories with the lead story delivered aloud and timed under four minutes.

Day 7 — Mock case and gap list

• Run a 40-minute mock case with someone playing a product manager who pushes back on your metric choice, and record it.
• Listen back and mark every moment you proposed a solution before the success metric existed.
• Rewrite those moments as the question you should have asked, and rehearse the first 90 seconds of the case until scoping comes before solving.

Deliverable: A recorded case plus a rewritten opening 90 seconds.