Interview guides · Exela Technologies

Exela Technologies Data Scientist Interview Questions & Guide 2026

Exela Technologies Data Scientist interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

A Data Scientist at Exela Technologies plays a pivotal role in transforming complex business information into actionable intelligence. By leveraging advanced analytics, machine learning, and automation, you will help optimize business processes and enhance the digital solutions Exela Technologies provides to its global clients. Your work directly impacts how the organization handles large-scale data, specifically in domains like document processing, workflow automation, and intelligent content management.

How much statistics you need depends on the flavour of the seat. Experiment-facing work wants you deep enough to notice that repeated looks at accumulating data inflate the false positive rate of a fixed-sample test; modelling-facing work wants estimation and honest uncertainty intervals.

Focus areas

Read NRR on a fixed account cohort
Analyse at the account grain, cluster errors
Measure churn only on renewal-eligible accounts

Overview

This role is both technically demanding and strategically significant. You will be expected to bridge the gap between raw, unstructured data and high-value business outcomes. Whether you are building predictive models, refining natural language processing (NLP) pipelines, or developing sophisticated video processing algorithms, your contributions will be central to maintaining the company’s competitive edge in the automation and technology services space.

Prepare to demonstrate not just your technical proficiency, but your ability to translate complex data findings into clear, business-oriented recommendations for non-technical stakeholders.

The Exela Technologies interview process

Exela Technologies candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Aptitude Assessment

A handful of shapes account for most of what gets asked in this format: a ranking or deduplication inside groups, a running or rolling total, a period-over-period comparison, and a cohort tracked forward over time. Recognising the shape quickly is most of the speed here; deriving it from scratch while a clock runs is where the time goes. Know that a window function keeps every row while a GROUP BY collapses them, and know which one the question needs. If the exercise is in Python instead of SQL, the same shapes arrive as groupby with transform, shift and merge, and the same grain mistakes are available.

What to demonstrate

Whether you reach the right construct without a detour, such as ROW_NUMBER over a partition to deduplicate instead of a self-join against a MAX subquery
Whether you know what your window frame actually is, since adding ORDER BY inside OVER changes the default frame and silently changes a running total
Whether the thing runs. A near-miss that throws an error scores below a plainer query that returns the right rows.

How to prepare

Write each of the four shapes once from memory against a small schema and keep the working version somewhere you will reread it: dedupe with ROW_NUMBER, a running total, a month-over-month change with LAG, and a retention table
Compute one running total twice on data with tied timestamps, once on the default frame and once with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and look at where the two disagree
If Python is on the table, rebuild the dedupe and the running total with groupby and cumsum, then assert the two implementations return identical rows

Technical Discussions

Much of what gets scored here happens out loud while you type. Nobody can see your reasoning inside a half-written query, so five silent minutes read as being stuck even when they are not. State the plan in plain language first: which tables, what grain you are aggregating to, and the one filter that defines the population. Then write it. The narration doubles as insurance, because a wrong plan gets caught early and cheaply while a wrong query gets caught at the end with no time left to redo it. A timed statistics section, where one exists, is a separate test with its own clock.

What to demonstrate

Whether the query you write matches the plan you just described
What you do with a hint, meaning whether the correction gets absorbed or the first approach gets defended
Whether you can debug your own wrong output by reading the result set and naming which part of the query produced the anomaly

How to prepare

Solve three problems while screen-sharing into a recording, then watch it back and mark every stretch longer than thirty seconds where you said nothing
Practise compressing the plan into one sentence before typing, then check afterwards whether the finished query actually matched it
Time yourself on statistics questions that carry a business reading, such as what a confidence interval does and does not claim, rather than re-reading notes without a clock

Manager Interview

Expect a live problem with pieces of it missing, closer to a conversation than an exam. A metric moved, or somebody wants to know whether a change worked, and you are asked how you would find out. The manager is watching the first ninety seconds, specifically whether you establish what decision hangs on the answer before you start proposing methods. Candidates who open with a technique get steered back. Once the decision is clear, describe what the data would look like if the story were true, and say what you would accept as evidence that it is not.

What to demonstrate

Whether you fix the decision the analysis serves before choosing an approach
How you continue when you are told the data you just asked for does not exist
Whether you state what would change your mind, not only what would confirm the hypothesis you started with
How you size an effect before you have measured it

How to prepare

Take a metric you know well and practise explaining in under two minutes the four things that could have moved it and how you would separate them
Pick a recent launch or experiment and write the single number you would ask for first, plus what you would conclude if it came back flat
Practise being interrupted: have someone remove a data source halfway through your answer and carry on without restarting

Where candidates lose points

Randomising an experiment at the user level when users share an account

Two problems fire at once. Colleagues in one workspace see each other's work and talk to each other, so a treated user changes the behaviour of a control user in the same account, which violates the no-interference assumption and biases the estimate toward zero. Separately, outcomes within an account are strongly correlated, so the effective sample size is roughly n / (1 + (m - 1) * rho) for m users per account and intra-class correlation rho, not n. With rho around 0.3 and twenty users per account that is a design effect near 6.7, meaning a user-level confidence interval is about two and a half times narrower than it should be and results cross significance thresholds on noise alone. Randomise the account and cluster the standard errors.

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Reporting a mean for a heavy-tailed metric without saying what it hides

For spend, session length or items per order, a small fraction of units carries most of the total, so the mean has a wide standard error and one account can move it. Fix the handling before you see the result: cap or winsorise at a pre-declared percentile, and report the median or the share above a threshold next to the mean. Capping changes the estimand, so say which question the capped number answers, and check how much of any difference comes from the top 0.1 percent of units.

Questions & practice

How do you handle a situation where a model performs well on training data but fails in production?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

Compare transformer-based architectures with traditional sequence models for specific text classification tasks.

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

How do you approach feature engineering for high-dimensional datasets?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Implement seven-day activation rate from its written definition

Easy · metrics · pandas · activation

Given dim_account (account_id, created_at, is_internal, is_current) and fct_api_request (account_id, request_at, http_status, api_key_id, traffic_class), implement activation_rate(accounts, requests, week_start). Definition: the numerator is accounts whose first request with http_status < 400, api_key_id not null and traffic_class != 'synthetic_monitor' occurs no later than 168 hours after created_at; the denominator is non-internal accounts created during the ISO week starting week_start. All timestamps are tz-aware UTC. Return the rate and both counts, and refuse to report a week until every account in it has had its full 168 hours.

Approach

Reduce dim_account to one row per account_id before joining anything. It is a type 2 dimension, so several versions of the same account exist; joining the versioned table to requests multiplies the denominator by the number of plan changes an account happened to make.
Build the denominator first and freeze it: is_internal == False and week_start <= created_at < week_start + 7 days. Everything after this is a filter on the numerator only, because an account that never sent a request must still sit in the bottom of the fraction.
Filter requests to qualifying rows and only then take groupby('account_id').request_at.min(). The first qualifying request is not the same object as the global first request filtered afterwards, and the two answers differ for every account whose first call was a 4xx.
Left-join the first qualifying timestamp onto the cohort and test (first_ok - created_at) <= Timedelta(hours=168). NaT propagates to False in that comparison, which is the behaviour you want, but assert it rather than assume it.
Guard reportability explicitly: if week_start + 7 days + 168 hours exceeds the maximum request_at in the data, the week is censored and will read as a drop, so return None or raise rather than emit a number.

Worked solution (20 min)

cohort = accounts.loc[~accounts.is_internal & accounts.created_at.between(week_start, week_start + pd.Timedelta(days=7), inclusive='left'), ['account_id','created_at']].drop_duplicates('account_id')
ok = requests[(requests.http_status < 400) & requests.api_key_id.notna() & (requests.traffic_class != 'synthetic_monitor')]
first_ok = ok.groupby('account_id', as_index=False).request_at.min(); m = cohort.merge(first_ok, on='account_id', how='left'); assert len(m) == len(cohort)
activated = (m.request_at - m.created_at) <= pd.Timedelta(hours=168); return {'rate': float(activated.mean()), 'numerator': int(activated.sum()), 'denominator': len(m)}

Expected result: A dict whose denominator is the count of distinct non-internal accounts created in that ISO week, numerator <= denominator, and rate = numerator / denominator; accounts with no qualifying request sit in the denominator with NaT and evaluate to False.

Verify it yourself

len(m) == cohort.account_id.nunique(); the merge must not add rows.
Dropping the synthetic_monitor filter can only raise the numerator, never lower it.
Widening the window from 168h to 336h is monotone: the numerator must not decrease.
An account whose only qualifying request falls at exactly created_at + 168h counts as activated, since the comparison is inclusive.

Follow-up

Median time-to-first-call is more informative. What breaks if you take the median over activated accounts only, and what estimator fixes it?
How would you decide whether 168 hours is the right window rather than 72 or 336?
An account signs up, does nothing for 20 days, then integrates heavily. Where does it land in this metric, and is that what you want?

Consecutive qualifying weeks before renewal, as a ranked worklist

Hard · gaps and islands · window functions · date boundaries

fct_api_request carries account_id, workspace_id, environment, request_at (timestamptz), http_status and traffic_class. fct_subscription_period carries account_id, term_end_date, auto_renew and is_current. A qualifying week for an account is an ISO week with at least 50 successful production requests in traffic_class ('interactive','batch'). Over the last 52 whole ISO weeks, and for accounts whose current term ends within 90 days, return the current qualifying-week streak length, the week it began, and the longest earlier streak. An account whose streak has broken must appear with a current length of zero.

Approach

Bucket weeks as date_trunc('week', request_at AT TIME ZONE 'UTC'). request_at is a timestamptz, so an unpinned date_trunc silently uses the session time zone, weeks start at a local midnight, and Monday-morning traffic lands in the previous week for part of the fleet. Pinning UTC also makes the seven-day arithmetic below exact across daylight-saving transitions.
Apply the exclusions before counting: environment = 'production', http_status < 400, traffic_class IN ('interactive','batch'). Then apply the volume floor and drop the current partial week, which can never meet a floor calibrated on whole weeks.
Build islands with the row-number anchor: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) as rn, then week_start - rn * interval '7 days' is constant inside a run of consecutive weeks. Group by that anchor to get each streak's start, end and length.
The current streak is the island whose end equals the last whole week; if none does, the account's current streak is zero and that is the interesting case. The longest earlier streak is the maximum length among the remaining islands.
Join the renewal filter from the current subscription row and LEFT JOIN the streak summary so an account with no qualifying week at all still appears, rather than vanishing from the risk list precisely because it went quiet.
Finish with an operating point. The list is worked by a team with finite capacity, so order it and cut it at that capacity, and say what happens to the accounts below the line.

Worked solution (40 min)

Build weekly_qualified: filter the fact on environment, status and traffic_class, group by account_id and date_trunc('week', request_at AT TIME ZONE 'UTC'), keep groups with count(*) >= 50, and exclude the in-progress week and anything older than 52 whole weeks.
Add rn = ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) and anchor = week_start - rn * interval '7 days'.
Group by (account_id, anchor) to get streak_start = min(week_start), streak_end = max(week_start), streak_len = count(*).
Per account, take current_len as the streak_len where streak_end = the last whole week else 0, current_start from the same island, and longest_prior as max(streak_len) over the other islands.
Join to fct_subscription_period on is_current with term_end_date <= current_date + 90, LEFT JOIN the streak summary, and order by current_len ascending then term_end_date ascending.

Expected result: One row per renewal-eligible account with current_streak_weeks, current_streak_start_week and longest_prior_streak_weeks, where an account with no qualifying week in the last whole week returns current_streak_weeks = 0 and still appears, ordered so the quietest imminent renewals sit at the top.

Verify it yourself

Per account, the sum of all island lengths equals the count of qualifying weeks for that account.
current_streak_weeks > 0 if and only if the last whole week is a qualifying week for that account.
No island contains a gap: re-deriving consecutive differences inside an island returns exactly 7 days everywhere.
An account qualifying in all 52 weeks returns a single island of length 52 and a null-free current streak start.

Follow-up

The floor of 50 requests was picked for this exercise. How would you calibrate it from data, and what would force a recalibration?
A regional holiday week drops several accounts below the floor at once. How do you keep that out of the risk list?
How would you evaluate whether contacting these accounts actually changed renewal, given that coverage is assigned deliberately?

Running commitment burn-down and the date consumption crosses it

Medium · window functions · running totals · committed consumption

fct_subscription_period gives committed_amount_cents, term_start_date and term_end_date for each account's current version where pricing_model = 'committed_consumption'. fct_usage_daily gives account_id, workspace_id, sku_code, usage_date and net_amount_cents, at one row per workspace and SKU per day. Inside each account's current term, return the running total of net_amount_cents by usage_date, the first usage_date on which that running total reaches committed_amount_cents, and the fraction of the term elapsed at that point. Accounts that have not reached their commitment must still appear, with a null crossing date.

Approach

Collapse usage to one row per (account_id, usage_date) first. The fact is grained by workspace and SKU, so a raw running total leaves several rows per date and the first-crossing date becomes dependent on the arbitrary order of rows inside that day.
Restrict to the term with usage_date BETWEEN term_start_date AND term_end_date on the account's current row, so no prior term's consumption leaks into this term's burn-down.
Compute SUM(net_cents) OVER (PARTITION BY account_id ORDER BY usage_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). Name ROWS explicitly: the default frame is RANGE, which includes all peer rows at the same ORDER BY value and would hand back the whole day's total on each of that day's rows.
Extract the crossing with MIN(usage_date) FILTER (WHERE running_cents >= committed_amount_cents) grouped per account, which returns NULL for accounts still under commitment instead of dropping them.
Elapsed fraction is (crossing_date - term_start_date)::numeric / NULLIF(term_end_date - term_start_date, 0); Postgres date subtraction yields whole days, so the guard matters for same-day terms.
Exclude the trailing days still inside the metering settling window, measured from first_written_at against restated_at, and say how many days you cut and why.

Follow-up

Turn this into an end-of-term overage forecast. What breaks if you extrapolate a linear run rate on a consumption product?
An account crosses its commitment at 40 percent of the term. Is that an expansion signal or a billing-surprise risk, and what would you check to tell them apart?

How do you optimize inference speed for real-time video analytics?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

Explain the trade-offs between different loss functions in deep learning models.

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

What techniques do you use to handle noise in unstructured text documents?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Stop metering retried requests: design the metric that decides it

Medium · guardrail conflict · billing integrity · retries · expected value

The platform meters accepted requests. A proposal carries three clauses: stop metering fct_api_request rows where is_retry = true, stop metering rows with a 4xx status, and stop metering rows with http_status >= 500. Its author has attached one figure to all three together, roughly 4% of requests_thousands volume. You have fct_api_request (account_id, is_retry, idempotency_key, http_status, traffic_class, billable_units, request_at) and fct_usage_daily (billable_quantity, net_amount_cents, cogs_cents). Size each clause separately before arguing about any of them, then define the primary metric, the guardrail that genuinely conflicts with it, and how you resolve that conflict for a decision that has to be made this quarter. Revenue falls this quarter with certainty; any benefit appears at renewals up to twelve months out.

Approach

Size the three clauses before accepting the headline 4%, because one of them is a no-op. billable_units is defined as zero for requests that failed with a 5xx, so the third clause removes no metered volume at all. Confirm that in the data rather than trusting the column comment: if sum(billable_units) over rows with http_status >= 500 is not zero, the metering pipeline contradicts its own definition and that is a billing defect to file before any pricing conversation happens. The two live clauses are 4xx failures, which are metered in full, and retries that did not themselves end in a 5xx.
Size the two live clauses as a union, not a sum. A retry can return 4xx, so the clauses overlap and adding their volumes counts that intersection twice. Partition the trailing 90 days into four mutually exclusive buckets instead: clean (is_retry = false, http_status < 400), non-retry 4xx, retry with http_status < 500, and http_status >= 500. Report the removable share per account as a distribution; if the mass sits in a handful of accounts this is a commercial conversation with those accounts rather than a platform-wide pricing change.
State the conflict rather than dissolving it. The primary metric, net metered revenue per paying account, and the integrity guardrail, the share of metered volume that is retried or failed traffic, move in opposite directions by construction. No redefinition removes that. The job is to price the trade-off, not to make it disappear.
Show the perverse coupling with data, and be exact about its mechanism. Because a 5xx already carries zero billable_units, the platform is not paid directly for its own failures; it is paid for the retries and the client-side 4xx traffic those failures provoke, which is one step removed and therefore easy to miss. Cross-tab each account's trailing 28-day 5xx rate against its metered volume in the same window. If metered volume rises with error rate, that indirect coupling is the actual argument for the change.
Resolve on expected value with the uncertainty stated. The revenue loss is computable and near-certain; the renewal benefit is not, so invert it and state the break-even: how many basis points of gross logo retention on the renewal-eligible base would offset the loss. That converts an argument about values into an argument about one number. Then propose the measurement that would settle it instead of claiming a readout you do not have: stage the rollout by renewal cohort so accounts whose terms end soonest are treated first, read out on gross logo retention on the renewal-eligible base, and say honestly whether the number of annual renewals in the window can support that estimate at all.

Worked solution (30 min)

Test the third clause first: over the trailing 90 days compute count(*) and sum(billable_units) from fct_api_request where http_status >= 500. The sum must be zero, because billable_units is defined as zero for 5xx failures. If it is zero the clause removes nothing and drops out of the analysis; if it is not, stop and raise a metering defect, because every volume figure downstream of that column is then suspect.
Compute trailing-90-day metered volume per account in the four mutually exclusive buckets: clean, non-retry 4xx, retry with http_status < 500, and http_status >= 500. Roll the total up to the requests_thousands SKU and reconcile it against fct_usage_daily billable_quantity for the same window.
Convert the two removable buckets to money using each account's realised rate, net_amount_cents / billable_quantity from fct_usage_daily, because list rate overstates revenue for every discounted account.
Annualise the revenue at risk and divide it by the ARR of the renewal-eligible base to express the break-even as an improvement in gross logo retention, in basis points.
Cross-tab account 28-day 5xx rate deciles against metered volume per account to establish whether the error-to-revenue coupling, which can only run through retries and 4xx rather than through the failed requests themselves, is real or a story.

Expected result: Three clause sizes rather than one blended figure: zero for the 5xx clause, since billable_units is already zero there, and separate volumes for 4xx failures and for retries that did not 5xx, combined as a union so the retry-and-4xx intersection is counted once. On top of that, a single break-even figure stated as the improvement in gross logo retention on the renewal-eligible base needed to offset the metered revenue actually removed, and the per-account concentration of that removable volume. The answer says that the primary metric and the integrity guardrail conflict by construction, and resolves it with the break-even number and a staged rollout rather than by moving a definition.

Verify it yourself

sum(billable_units) over rows with http_status >= 500 must be exactly zero; any other result contradicts the documented definition of the column and invalidates the volume figures derived from it.
The four buckets must be mutually exclusive and must sum to total billable_units, which must in turn reconcile to fct_usage_daily billable_quantity within the known restatement tolerance; a large gap means the request-to-SKU mapping is wrong.
The two removable buckets must be reported as a union; their arithmetic sum will exceed the union by exactly the retry-and-4xx intersection, and a write-up that quotes the sum has overstated the revenue at risk.
Requests sent with no idempotency_key cannot carry is_retry = true, so the measured retry share is a lower bound and the write-up must say so.

Follow-up

Suppose the two live clauses turn out to remove 2.6% of consumption revenue. How much improvement in gross logo retention on an annual-contract base pays that back, and over what horizon does the payback land?
A retry sent without an idempotency_key cannot be flagged as a retry. Which direction does that bias your estimate of the removable volume, and how can you bound it?

Blended gross margin fell while every segment improved

Medium · mix shift · simpsons paradox · as-of join

Blended gross margin across paying accounts fell from 71 percent to 66 percent over two quarters, yet margin improved inside every plan_tier and every deployment_model. You have fct_usage_daily (account_id, sku_code, usage_date, net_amount_cents, cogs_cents), fct_subscription_period (account_id, plan_tier, term_start_date, term_end_date) and dim_account (account_id, deployment_model, is_internal, effective_from, effective_to). Produce an exact decomposition of the five-point move into mix, rate and interaction terms, and name the segment shift that carries it.

Approach

Write blended margin as a revenue-weighted average of segment margins, m = sum over i of w_i * m_i, where w_i is the segment's share of net_amount_cents. Margin is a ratio of sums, so only revenue weights reproduce the blended figure.
Apply the exact three-term decomposition: delta_m = sum(delta_w_i * m_i0) + sum(w_i0 * delta_m_i) + sum(delta_w_i * delta_m_i), read as mix, rate and interaction. It is exact by construction, so the three terms must sum to the observed change with no residual.
Build segments from as-of joins: take the fct_subscription_period version whose term brackets each usage_date and the dim_account version whose effective_from and effective_to bracket it, rather than joining on is_current and backdating today's attributes over last year's usage.
Run the decomposition on more than one segmentation, at minimum plan_tier, deployment_model and sku_code, since the mix that actually moved may not be the one anybody already suspected.
Once the driving segment is identified, determine whether the shift came from acquisition landing new accounts in a lower-margin segment or from existing accounts migrating, because those have different owners and different remedies.

Follow-up

The interaction term is sometimes large. What does it mean operationally, and when would you prefer a log-mean decomposition that distributes it across the other two terms?
Revenue-weighted average margin is one construction. What changes if leadership wants margin weighted by account count, and which question does each version answer?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

What the behavioural round is actually scoring

Interviewers here are not checking whether you can describe a project. They want the decision you made, why you made it under the information you had, and what changed afterwards that someone else could measure. A story that ends at 'I built a model' has no ending. Say what the model caused, or what you stopped doing because of it.

Describe your experience with model deployment and monitoring in a production environment.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.
Quantify the outcome, including what you would not claim credit for.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Describe a time you had to clean a messy, real-world dataset to make it trainable.

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

State the measured impact of your own work honestly

Hard · impact · causal inference · self-assessment

You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.

Approach

The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.

Follow-up

Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
What would have had to be true for the naive covered versus uncovered comparison to be valid?
If the holdout costs the team real renewals, how do you justify asking for it, and to whom?