Interview guides · Hexaware Technologies

Hexaware Technologies Data Scientist Interview Questions & Guide 2026

Diagnose a metric move for Hexaware Technologies data scientist prep: split by segment, rule out a logging change, separate mix shift from behaviour.

The 60-second brief

A Data Scientist at Hexaware Technologies plays a pivotal role in leveraging data to drive strategic decisions and enhance operational efficiencies. This position is crucial as it directly influences product development, user engagement, and business outcomes. As a member of the data science team, you will work on complex problems that involve analyzing vast datasets, building predictive models, and providing actionable insights that contribute to the company's competitive edge in the technology sector.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Analyse at the account grain, cluster errors
Power experiments for heavy-tailed account revenue
Separate contracted seats from actively used seats

Overview

In this role, you will engage with various teams, including engineering, product management, and business analysis, to develop solutions that meet client needs and drive innovation. You will be involved in projects that span across multiple domains, from finance to healthcare, applying machine learning algorithms and statistical techniques to deliver robust data-driven solutions. This position is both challenging and rewarding, providing opportunities to work on high-impact projects that shape the future of Hexaware Technologies.

The Hexaware Technologies interview process

Hexaware Technologies candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Telephonic Screening

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Technical Interviews

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Where candidates lose points

Treating raw request or usage volume as engagement

Most traffic in this domain is emitted by machines. Continuous-integration pipelines, scheduled batch jobs, synthetic monitors, backfills and client retries can all grow by an order of magnitude from one configuration change made by one engineer, and none of it represents a new decision to use the product. The inversion is what makes it dangerous: when the platform degrades, clients retry, so error-driven retry volume rises at the exact moment the customer is most likely to leave, and an engagement dashboard built on raw counts shows growth immediately before a churn. Filter on traffic_class and on successful status before anything else, and keep failed-request volume as its own separate series.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Accepting a metric definition without asking about the denominator

Pin down the denominator, the eligibility filter and the time window before computing anything: conversion rate per session, per user, per eligible user and per new user are four different numbers with different behaviour. Restate the definition in one sentence and get agreement before you analyse.

Averaging per-user rates to produce a population rate

Decide which quantity you want: the mean of per-user ratios and the ratio of summed numerator to summed denominator are different estimands, and heavy users dominate one but not the other. For a ratio metric, aggregate numerator and denominator separately and use the delta method for its variance.

Questions & practice

Write a function that calculates the mean and standard deviation of a list of numbers in Python.

Medium · statistics and probability

Approach

Say what the estimate is of, and over what population it generalises.
Sanity-check the answer against a simple bound or a simulated case.
Quantify uncertainty explicitly rather than reporting a point estimate alone.

Follow-up

Which assumption here is most likely to be violated in practice?
What sample size would you need to detect an effect half this size?

What is overfitting, and how can you prevent it?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Describe how you would validate a machine learning model before deployment.

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Sessionise an API event stream with a 30-minute inactivity gap

Medium · sessionisation · event-streams · pandas

fct_api_request arrives as a DataFrame with account_id, user_id, request_at (tz-aware UTC), traffic_class and http_status, roughly 5 million rows. Assign a session_id to every human-attributable request: drop rows where user_id is null or traffic_class is in ('ci','synthetic_monitor','load_test'), then open a new session whenever the gap since that user's previous remaining request exceeds 30 minutes. Return the filtered frame plus session_id, and a per-session summary with user_id, account_id, session start, session end and request count. Do not loop over rows.

Approach

Settle the filter-then-gap ordering before writing code. Removing CI and synthetic rows changes the gaps, so sessionising the raw stream and filtering afterwards is a different answer; the definition given filters first, and the two diverge most for accounts whose CI runs every ten minutes.
Sort once by (user_id, request_at) with a stable kind, then gap = df.groupby('user_id', sort=False).request_at.diff(). The first row of each user yields NaT, which is exactly the boundary condition you want rather than a special case to patch.
new_session = gap.isna() | (gap > Timedelta(minutes=30)); session_id = new_session.cumsum(). The cumsum runs over the whole sorted frame and therefore produces globally unique ids in one pass; a per-user cumcount collides across users and forces a composite key on every downstream join.
Build the summary with a single groupby('session_id').agg(...). user_id and account_id can be carried with 'first' only because the sort key groups them — state that dependency, since it silently breaks if someone later re-sorts the frame.
Decide explicitly what a session means when one user_id holds memberships in several accounts: either add account_id to the sort and group keys, or document that sessions may cross accounts. Leaving it undecided produces sessions whose account_id is whichever row sorted first.

Worked solution (30 min)

human = df[df.user_id.notna() & ~df.traffic_class.isin(['ci','synthetic_monitor','load_test'])].sort_values(['user_id','request_at'], kind='mergesort').reset_index(drop=True)
gap = human.groupby('user_id', sort=False).request_at.diff(); human['session_id'] = (gap.isna() | (gap > pd.Timedelta(minutes=30))).cumsum()
summary = human.groupby('session_id').agg(user_id=('user_id','first'), account_id=('account_id','first'), start=('request_at','min'), end=('request_at','max'), n_requests=('request_at','size')).reset_index()
assert summary.n_requests.sum() == len(human) and human.groupby('session_id').user_id.nunique().max() == 1

Expected result: human gains a dense session_id increasing in (user_id, request_at); summary has exactly one row per session, n_requests sums to len(human), and every within-session consecutive gap is at most 30 minutes while every same-user between-session gap exceeds it.

Verify it yourself

summary.n_requests.sum() == len(human); no request is lost or double-counted.
human.groupby('session_id').user_id.nunique().max() == 1.
human.groupby('session_id').request_at.diff().max() <= pd.Timedelta(minutes=30).
Shifting every timestamp by a constant offset leaves the session count unchanged.

Follow-up

Where does 30 minutes come from, and how would you pick it from this data instead of from convention?
An engineer reused their personal key for a nightly batch job, so machine traffic carries a human user_id. How would you detect that, and should those requests form sessions?
How much does the session count change if you sessionise before dropping CI traffic rather than after?

What data structures are most commonly used in data science applications?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
State the window function and its partition and ordering out loud before writing it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Collapse retried requests into logical operations per account

Medium · window functions · deduplication · null semantics

fct_api_request carries request_id, account_id, api_key_id, idempotency_key (null when the caller supplied none), is_retry, request_at and http_status. For one ISO week, count logical operations rather than HTTP calls: requests sharing (account_id, api_key_id, idempotency_key) collapse to the earliest one, while every request with a null idempotency_key is its own operation. Return per account the raw request count, the logical operation count and the ratio between them, and state which direction a rising 5xx rate pushes that ratio.

Approach

Split the stream before deduplicating. GROUP BY and PARTITION BY treat NULLs as equal to one another, while the = operator does not, so a single grouping pass folds every null-key request in an account into one 'operation' and can cut the count by orders of magnitude.
Deduplicate the keyed rows with ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) and keep rn = 1. Including request_id in the ORDER BY makes the result deterministic when two rows share a millisecond, which they will.
UNION ALL the null-key rows back in unchanged; they need no dedup and must not pass through the partition.
Aggregate per account: count() over the raw week, count() over the union, and the ratio of the two. Report the ratio, not the difference, so accounts of very different sizes are comparable.
Cross-check against is_retry, which flags a resend of the same idempotency key: raw minus logical should sit close to the count of is_retry rows, and a large gap means clients are resending without a key and the dedup is under-counting duplicates.
Name the inversion explicitly: 5xx responses provoke client retries, so amplification rises exactly when reliability falls, and any engagement metric built on raw request counts will show growth during an outage.

Worked solution (25 min)

Filter to the ISO week with request_at >= :week_start AND request_at < :week_start + interval '7 days', and record the raw count per account as the baseline.
Build keyed_dedup: SELECT ... ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) AS rn FROM the week WHERE idempotency_key IS NOT NULL, then keep rn = 1.
Build unkeyed: the same week's rows WHERE idempotency_key IS NULL, taken as-is.
UNION ALL the two, then GROUP BY account_id selecting the logical count; join back to the raw counts and compute raw::numeric / logical.
Validate on one account by comparing raw - logical against COUNT(*) FILTER (WHERE is_retry) for that account and explaining any gap.

Expected result: One row per account with raw_requests, logical_operations and an amplification ratio at or above 1.0, where logical_operations counts each distinct (account_id, api_key_id, idempotency_key) once and each null-key request once.

Verify it yourself

logical_operations is never greater than raw_requests, and the two are equal for accounts whose keys are all distinct.
For accounts with no null idempotency_key, raw_requests - logical_operations is close to the count of is_retry rows.
Deliberately grouping on the key without the null split reproduces a visibly lower count, confirming the NULL collapse is the mechanism.
Re-running the dedup returns the same surviving request_id set, proving the ORDER BY tiebreak made it deterministic.

Follow-up

Which of these two counts belongs in the billable-units metric, and which in an engagement metric?
An account's amplification ratio jumps from 1.05 to 3.0 in a day. Name three causes and the query that separates them.

Given a dataset with customer feedback, how would you derive actionable insights?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

Walk us through your approach to solving a data quality issue in a dataset.

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize tasks when working on multiple projects?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Name one primary metric, then the guardrail that stops it being gamed.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

How would you approach a project aimed at predicting customer churn?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

Which segment would you cut first, and what would that rule out?
How would you detect that the metric is being gamed rather than genuinely improving?

Describe the process of feature selection.

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer would contain.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Measure switching cost when switching cost is not observable

Medium · proxy metrics · bias statement · integration depth · stratification

Leadership wants switching cost tracked as a leading indicator of renewal. Nothing in the warehouse records it. Available: fct_api_request (account_id, workspace_id, environment, api_key_id, sdk_name, sdk_version, endpoint, traffic_class, http_status, request_at) and dim_account (account_id, employee_band, deployment_model, is_internal). Propose a proxy, state the direction and likely size of its bias, and name one decision the proxy is good enough for and one it is not. Deliver the proxy definition, the written bias statement, and the validation you would run against observed renewal outcomes.

Approach

Say first that switching cost is unobservable in this data and that the deliverable is a biased correlate with its bias written down, not a measurement. Anything presented as a direct measure of switching cost is already wrong before the SQL starts.
Define the proxy as production integration breadth per account over a trailing 28 days: distinct endpoint route templates, distinct api_key_id, and distinct workspace_id with environment = 'production', all restricted to http_status < 400 and traffic_class in ('interactive','batch'). Breadth, not volume, because request volume is one CI configuration change away from an order of magnitude.
State the biases with their mechanisms and their sign. Upward with account size, because breadth correlates with employee_band, so the proxy ranks large accounts as sticky whether or not they depend on anything. Downward for deployment_model = 'self_hosted', whose traffic does not all cross the managed gateway, so their breadth is systematically understated. Blind to criticality: one endpoint carrying a production billing path is a larger switching cost than twenty endpoints behind a read-only dashboard, and nothing in this data distinguishes them.
Handle the bias by stratifying rather than by pretending it is gone. Report the proxy within employee_band and deployment_model strata and state explicitly that cross-stratum comparisons are not supported by the construction.
Validate against the only outcome that is actually observed: renewal on the renewal-eligible base. Within strata, report renewal rate by proxy quintile for accounts whose term_end_date has passed plus a 45-day grace, and report discrimination at the operating point a capacity-bound team can work rather than a global AUC.

Worked solution (30 min)

Compute the trailing-28-day proxy per account: distinct endpoint, distinct api_key_id and distinct production workspace_id under successful, interactive-or-batch filters.
Join dim_account for employee_band and deployment_model and drop is_internal = true.
Build the renewal-eligible cohort from fct_subscription_period term_end_date in the target months with a 45-day grace, labelling each account renewed or not.
Within each employee_band stratum, report renewal rate by proxy quintile, and report the proxy distribution for self_hosted accounts separately so its understatement is visible.

Expected result: A proxy defined as production integration breadth over a fixed 28 days, a written bias statement naming upward bias with account size, downward bias for self-hosted deployment and total blindness to criticality, plus a within-stratum validation showing renewal rate by proxy quintile. The answer names one supported decision, such as ordering a capacity-bound outreach list inside a stratum, and one unsupported decision, such as setting price or contract terms by switching cost.

Verify it yourself

Every account's proxy must be computed over the same 28-day window; a variable window makes the quintiles incomparable.
Self-hosted accounts should show systematically lower breadth; if they do not, gateway coverage of that deployment model differs from what was assumed and the bias statement is wrong.
The renewal-eligible cohort should be roughly one twelfth of the paying base per month on annual contracts; a much larger share means the term_end_date filter is not doing what you think.

Follow-up

An account's endpoint breadth drops 40% in a week. Name three explanations that have nothing to do with reduced dependency.
Which decision would you refuse to make on this proxy, and what evidence would you need before making it?

Error rate halves while severe support tickets double

Hard · missing data · aggregation bias · reconciliation

The fleet-wide customer-visible error rate fell from 1.8 percent to 0.9 percent, while sev1 and sev2 tickets and reopened_count rose across the same fortnight. Using fct_api_request (account_id, environment, sdk_name, traffic_class, http_status, request_at, billable_units), fct_usage_daily (account_id, sku_code, usage_date, billable_quantity) and fct_support_ticket (account_id, severity, opened_at, reopened_count, linked_incident_id), determine whether reliability improved, and if not, identify precisely which rows are missing and from when. Deliverable: a diagnosis backed by an independent corroborating source.

Approach

Distrust an improvement that contradicts an independent operational signal. Two sources disagreeing is itself the finding; decide which one is more likely to be broken before explaining either.
Recompute the rate as the metric tree defines it, per account first and then as the share of accounts above the reliability target. A single global average is dominated by whichever account sends the most traffic, so a fleet number can fall while a quarter of accounts get worse.
Audit for missingness rather than for badness: count fct_api_request rows per hour split by environment, sdk_name and status class, indexed against the trailing same-hour baseline. A partial ingestion failure shows as a step drop confined to one slice, not a uniform decline.
Reconcile against a source the request pipeline does not feed, such as implied request volume from fct_usage_daily for the same accounts and dates. If the usage table is flat while request rows fell, rows are missing rather than traffic.
Test whether the missingness is differential by status, which is the mechanism that fakes an improvement: if 5xx rows are written on a path that stopped while 2xx rows were unaffected, the numerator falls faster than the denominator and the rate drops with nothing improving.
Close with the affected window, the affected slice and a restated series marked unreliable across that window, rather than a silently patched number.

Follow-up

The missing rows are unrecoverable. How do you present that fortnight in a series people compare week over week?
What monitor would have caught this within an hour, and what is its false-positive cost on a normal quiet weekend?

Suggested practice checklist

Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.

Day 1 — Breadth pass: query fluency

Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.

Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.

Day 2 — Breadth pass: statistics and inference

Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
Rewrite the two weakest answers the following morning from memory in full sentences.

Deliverable: Ten graded answers with an honest count of exact hits.

Day 3 — Breadth pass: modelling

Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.

Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.

Day 4 — Breadth pass: product judgement

Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.

Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.

Day 5 — Depth, first area

Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
Re-solve the two you failed the same evening with notes closed.

Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.

Day 6 — Depth, second area, and the seam between them

Repeat the depth protocol on the second-ranked area with the same six-problem structure.
Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
Solve your own combined problem end to end and note where the handoff between the two areas cost you time.

Deliverable: One combined problem, solved end to end, with the handoff failure written down.

Day 7 — Integration and re-measurement

Re-run the six prompts from day one under the same clock and compare both correctness and time.
Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.

Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

What steps do you take to ensure collaboration with cross-functional teams?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

State the measured impact of your own work honestly

Hard · impact · causal inference · self-assessment

You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.

Approach

The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.

Follow-up

Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
What would have had to be true for the naive covered versus uncovered comparison to be valid?
If the holdout costs the team real renewals, how do you justify asking for it, and to whom?

Report an underpowered consumption test to a non-technical executive

Medium · communication · uncertainty · experimentation

An account-randomised packaging change ran six weeks across 900 paying accounts. The effect on billable units per account per month is plus 4.1 percent, with a 95 percent interval from minus 3.2 to plus 11.8 after clustering standard errors at the account and applying the pre-registered winsorisation at the 99th percentile. An executive with no statistical background wants one number this week to decide a full rollout. Produce a three-sentence spoken answer, one chart, and an explicit recommendation of ship, stop or keep running, with the cost of each option stated.

Approach

The interviewer is probing whether you can be decision-useful without either hiding the uncertainty or hiding behind it. Start from the decision rather than the statistics: establish what the executive would do differently at plus 4 percent versus zero, because if the action is identical the interval does not matter.
Translate the interval into consequences in units the executive already reasons about. Multiply both endpoints by the cohort's baseline consumption and contracted rates to give an annualised revenue range, so the answer is a range of dollars rather than a range of percentages.
Price the option to wait. Using the observed variance, state roughly how many additional account-weeks halve the interval width, so keep running becomes a quantified choice instead of a stall.
Offer a cheaper path to the same decision: a lower-variance proximate outcome such as successful billable units on the new SKU, or CUPED using each account's pre-period consumption, quoting the expected variance reduction as one minus the squared pre-post correlation.
Give a recommendation and name the single observation that would reverse it. A strong answer commits; a generic one recites the interval and leaves the decision on the table.

Follow-up

The executive says it clearly works and is just not provable, so ship it. What is your answer?
How much of the interval width comes from clustering and how much from the revenue tail, and what would you do about each?
If you had to ship this week with no more data, which guardrail would you watch for the first fortnight and at what threshold would you roll back?