Interview guides · eXalt

eXalt Data Scientist Interview Questions & Guide 2026

Data scientist prep for eXalt candidates: drill A/B design, power and minimum detectable effect on original exercises.

The 60-second brief

A Data Scientist at eXalt operates at the intersection of technical rigor and business consulting. As a professional in an ESN (Entreprise de Services du Numérique) environment, you are not merely building models in isolation; you are a strategic partner responsible for delivering actionable data insights that solve specific client challenges. Your work directly affects how eXalt's clients optimize their operations and make data-driven decisions.

Nearly every loop contains a round whose deliverable is a recommendation to someone non-technical. Practise stating a conclusion, the confidence attached to it, and the cost of being wrong in each direction, because that triple is the artifact being graded.

Focus areas

Segment margin by pricing model before comparing
Compute utilisation against a defended availability denominator
Separate bookings, recognised revenue and collected cash

Overview

The role is inherently dynamic, requiring you to adapt quickly to different industry sectors and client infrastructures. You will bridge the gap between complex raw data and clear, value-added communication. Success in this role requires a blend of sharp analytical skills and a consultant’s mindset—the ability to identify the "why" behind the data and translate it into a compelling narrative for stakeholders who may not have a technical background.

The eXalt interview process

eXalt candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Screening Phase

Because the format is not fixed, prepare the reasoning rather than the ritual. Nearly every version of this round draws on the same underlying material: a design you can defend, a metric you can define exactly, an analysis whose assumptions you can state out loud. Only the wrapper changes, whether that is a take-home, a live case, a deep dive on past work, or a rough estimate on a whiteboard. Answers rehearsed to fit one shape stall the moment the shape differs. Practise naming the assumption behind a number, then saying how much the conclusion moves if that assumption is wrong.

What to demonstrate

Whether your justification for a method survives the question 'why not the simpler thing', including when the simpler thing would have worked
Precision under pressure: what exactly counts as an active user, a conversion or a success, over what window, with what exclusions
Whether you carry an argument through to a recommendation instead of stopping at a list of tradeoffs

How to prepare

For each project you plan to mention, write the metric definition in one sentence: numerator, denominator, time window, exclusions. Say it out loud once, because vagueness shows up in speech before it shows up on paper.
Rehearse the same project at three lengths: two minutes, ten minutes, and a deep dive on one technical decision. Cutting live is harder than it sounds.
For your headline result, write down what would have had to be true for it to be wrong, and how you ruled that out.

Technical Assessment

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Final Interview

Where a loop ends with a senior leader, that conversation is rarely another skills test. The technical signal already exists by then, so the questions tend to open up: what you would look at first, where a metric you have heard about could mislead, what you would push back on. The decision being made is scope, which in practice means level and how much you would be trusted to own unsupervised. Treating it as a formality is the usual mistake. An open question late in the day is still being scored, and a vague answer reads as someone who has not run anything themselves.

What to demonstrate

Whether your view of the business has anything specific behind it, given that you are working only from what is public and are expected to say so
Whether the scope of work you describe owning matches the scope of the role, instead of sitting a level below it
Whether you can disagree with something concrete and stay useful about it, rather than agreeing with everything said in the room
Whether your questions are ones only this person could answer, as opposed to ones the recruiter already covered

How to prepare

Build one view you could defend for two minutes using only public information: what the funnel probably looks like, which metric likely drives decisions, and where that metric could mislead. Being wrong for a stated reason survives this round; having no view does not
Write down the largest piece of work you have owned from question to decision, who else touched it, and what you decided alone, then check that it reads at the level you are interviewing for
Prepare one thing you would want changed if you joined and phrase it as a question rather than a verdict, so it opens a conversation instead of closing one

Where candidates lose points

Modelling win rate on proposals with a recorded outcome, using fields written after the decision

Two failures compound here. First, stage IN ('withdrawn','no_decision') is not missing at random: those are disproportionately deals that were going to be lost, so training on won-plus-lost only inflates apparent win rate and distorts the coefficients. Second, fields like engagement_id, final scope and revised pricing are populated after the outcome is known, so including them leaks the label and produces a model with excellent backtest accuracy and no forward value. Restrict features to values knowable at submitted_at.

Trending utilisation or revenue on work_date without accounting for timesheet backfill

Time entries are created days to weeks after the work happens, and the backfill tail often runs two to six weeks. A dashboard keyed on work_date therefore shows the most recent weeks as a decline that reverses on every refresh. The fix is either to hold the reporting window back past the observed backfill tail (measure the tail with the timesheet submission lag metric rather than guessing) or to report an as-of-entered_at snapshot so the series is internally consistent, and to state which one you used.

Ending an analysis without a recommendation or next step

Close with what you would do and what would change your mind, stated as a condition you can check later. If the evidence is genuinely inconclusive, recommend the specific next measurement and say what it costs in time or exposure.

Stopping an experiment the moment it crosses significance

Fix the sample size or duration before launch, or use a method built for continuous monitoring such as a sequential test, always-valid confidence intervals, or group-sequential boundaries. Repeatedly checking a fixed-horizon p-value against 0.05 pushes the real false-positive rate well above 5 percent.

Questions & practice

Build a dense consultant-week hours spine with zero weeks

Easy · calendar spine · groupby · missing rows

You are given time_entries (time_entry_id, consultant_id, engagement_id, charge_code, work_date as datetime64, hours, is_billable, status) and a list of consultant_ids in scope plus a start and end date. Produce a DataFrame with exactly one row per consultant per Monday-anchored week in that range, with columns consultant_id, week_start, billable_hours, total_hours. Weeks in which a consultant logged nothing must appear as zeros rather than be absent. Count only rows with status == 'approved'. Do not loop over consultants and do not use resample.

Approach

Anchor weeks arithmetically: week_start = work_date - pd.to_timedelta(work_date.dt.weekday, unit='D'). Avoid dt.isocalendar().week, whose numbers restart each January and do not sort across a year boundary.
Construct the spine from the inputs, not from the data: pd.date_range(start, end, freq='W-MON') crossed with the in-scope consultant_ids via pd.MultiIndex.from_product, so its size is fixed before any aggregation runs.
Aggregate approved entries once with a single groupby on [consultant_id, week_start], producing total hours and billable hours (hours.where(is_billable, 0).sum()) in the same pass.
Left-merge the aggregate onto the spine, fillna(0.0) on both hour columns, and keep float dtypes so later ratios do not integer-divide.
Assert the shape before returning: len(out) == n_consultants * n_weeks, and out.total_hours.sum() equals the sum of hours over the filtered input inside the window.

Follow-up

Utilisation gets computed on this spine. What breaks for a consultant hired mid-window or terminated mid-window, and where would you fix it?
How would you extend this to consultant x engagement x week without the spine exploding to a row count nobody can hold in memory?

Implement billable utilisation against a defended availability denominator

Medium · metric implementation · denominators · proration

Implement billable utilisation for one calendar month at consultant grain, then roll it up to practice_area. Inputs: time_entries (consultant_id, work_date, hours, is_billable, charge_code, status), consultants (consultant_id, practice_area, fte_fraction, home_region, is_billable_role, hire_date, termination_date) and holidays (region, holiday_date). Numerator is approved hours with is_billable true. Denominator is scheduled workdays in the month, clipped to [hire_date, termination_date] and net of that region's holidays, times 8 times fte_fraction, minus approved leave hours. Roll up as sum(numerator) / sum(denominator).

Approach

Build the workday set per region first: business days in the month minus that region's holiday_dates. State the Monday-to-Friday assumption out loud, because a region with a different working week makes np.busday_count wrong rather than approximate.
Prorate by clipping each consultant's interval to [max(month_start, hire_date), min(month_end, termination_date or month_end)] and counting workdays inside the clipped interval, so a mid-month start produces a smaller denominator instead of a fake dip in the ratio.
Subtract approved leave hours (charge_code in leave_paid, leave_unpaid) only for leave falling on days already counted as workdays; leave logged on a public holiday would otherwise be subtracted twice and can push a denominator negative.
Filter to is_billable_role, then drop consultant-months whose denominator is zero or negative (full-month leave, a hire on the last day) rather than emitting inf or NaN, and return the dropped count as part of the result.
Roll up by summing numerators and denominators separately. Averaging per-person ratios gives a 0.2 FTE consultant the same weight as a full-time one and produces a practice number that no individual reconciles to.

Follow-up

Utilisation rose two points this month while realisation fell. Reconcile those two movements with one mechanism.
A regional lead says their team looks four points below another region. What do you check before you answer, and in what order?

Collapse time entries into contiguous staffing spells

Medium · sessionisation · gap and island · vectorisation

From approved delivery time entries (consultant_id, engagement_id, work_date, hours, charge_code, status, with engagement_id not null), build staffing spells. Per consultant and engagement, collapse weeks containing any logged hours into contiguous runs, where three or more consecutive zero-hour weeks end a spell. Output one row per spell with consultant_id, engagement_id, start_week, end_week, active_weeks, gap_weeks and total_hours. Consultants sit on several engagements at once, so spells from different engagements may overlap in time and must not be merged. No Python loop over rows.

Approach

Aggregate to (consultant_id, engagement_id, week_start) with summed hours and keep only weeks with positive hours. The absent weeks are the signal, so materialising zeros here would destroy the thing you are detecting.
Convert week_start into an integer week index, ((week_start - epoch_monday).dt.days // 7), so gap detection is integer subtraction rather than calendar arithmetic that breaks over month and year boundaries.
Sort by [consultant_id, engagement_id, week_index], diff the week index inside each pair, and mark a spell start where the diff is null (first row of the pair) or greater than 3. A diff of 1 is adjacent weeks and a diff of 3 is two empty weeks, which the tolerance permits.
Take spell_id = the cumulative sum of that boolean over the whole frame so ids are globally unique, then a single groupby on [consultant_id, engagement_id, spell_id] yields min and max week, active week count and summed hours; gap_weeks = (end - start + 1) - active_weeks.
Do not deduplicate overlapping spells across engagements. A consultant on two engagements in the same week is the normal case, and that overlap is the fact any capacity or context-switching question needs.

Worked solution (30 min)

Filter to status == 'approved', charge_code == 'client_delivery' and engagement_id.notna(); derive week_start by subtracting the weekday offset from work_date.
wk = df.groupby(['consultant_id','engagement_id','week_start'], as_index=False).hours.sum(); wk = wk[wk.hours > 0]; wk['wi'] = (wk.week_start - pd.Timestamp('1970-01-05')).dt.days // 7.
wk = wk.sort_values(['consultant_id','engagement_id','wi']); d = wk.groupby(['consultant_id','engagement_id']).wi.diff(); wk['new_spell'] = d.isna() | (d > 3); wk['spell_id'] = wk.new_spell.cumsum().
spells = wk.groupby(['consultant_id','engagement_id','spell_id']).agg(start_week=('week_start','min'), end_week=('week_start','max'), active_weeks=('wi','size'), total_hours=('hours','sum'), span=('wi', lambda s: s.max() - s.min() + 1)).reset_index(); gap_weeks = span - active_weeks.
Run the conservation assertion (spell hours sum to input hours) and the toy case below before returning.

Expected result: One row per spell, where a single pair with active week indices [1, 2, 5, 9] produces two spells: weeks 1 to 5 with active_weeks 3 and gap_weeks 2, and week 9 alone with active_weeks 1 and gap_weeks 0.

Verify it yourself

Summed total_hours across spells equals the summed hours of the filtered input, exactly.
Within a pair, spells are disjoint, ordered, and separated by at least three empty weeks.
No spell contains an internal gap longer than two weeks, verified by re-expanding one spell against the week-level frame.
Two consultants with identical week patterns on the same engagement produce two separate spells, not one.

Follow-up

Re-run with a one-week and a four-week tolerance. What happens to the spell count, and which tolerance would you defend to a staffing lead?
Using these spells, how would you measure how many engagements a consultant is split across in a given week, and why is that not just a count of rows?

Longest consecutive bench spell per consultant from weekly activity

Hard · gaps and islands · window functions · run length

You have a dense series (consultant_id, week_start, billable_hours, full_week_leave BOOLEAN) covering 26 weeks for every billable-role consultant, with zero-hour weeks present as 0. Return every maximal run of consecutive weeks where billable_hours = 0, as consultant_id, spell_start_week, spell_end_week and week_count, then each consultant's longest run. Weeks flagged full_week_leave must neither break a run nor count toward its length. Mark runs still open at the last week of the series rather than reporting them as finished.

Approach

Remove full_week_leave weeks from the sequence first, then re-number what remains with ROW_NUMBER() OVER (PARTITION BY consultant_id ORDER BY week_start). Deleting before numbering is precisely what makes a leave week transparent to a run instead of a break in it; filtering afterwards would leave a gap in the sequence and split the spell.
Flag each surviving week as bench (billable_hours = 0) or active, then compute two row numbers per consultant: one over all surviving weeks, one partitioned additionally by the bench flag. Their difference is constant inside any run of adjacent same-flag weeks, which is the island key.
Group the bench weeks by (consultant_id, island key) and take MIN(week_start), MAX(week_start) and COUNT(*). Do not island on week_start minus an interval times a row number: that variant assumes the spine has no holes, and the leave removal has just deliberately put holes in it.
Pick the longest spell per consultant with ROW_NUMBER() OVER (PARTITION BY consultant_id ORDER BY week_count DESC, spell_start_week) so ties resolve deterministically, rather than a MAX that cannot carry the accompanying dates.
Handle both edges as censoring. A spell touching the last week of the series is still open and its length is a lower bound, so flag it; a spell touching the first week may have started earlier, so either flag it too or exclude it from any length distribution and say which.

Follow-up

A spell begins before your 26-week window. How do you detect that, and what do you report as its length?
Rebuild this at daily grain. What changes about weekends, holidays and the island key?
How would you separate true bench from unbilled delivery work on a live engagement, given only charge codes?

Dense consultant-week series with a trailing four-week average

Medium · window functions · calendar spine · running aggregates

fct_time_entry holds consultant_id, engagement_id, charge_code, work_date, hours, is_billable and status. dim_consultant holds consultant_id, is_billable_role, hire_date, termination_date and is_current. Produce one row per billable-role consultant per ISO week over the last 26 weeks with approved billable hours and the trailing four-week average including the current week. Weeks with no time entries must appear as 0, not disappear. No consultant may appear before hire_date or after termination_date. State where you put the right-hand edge of the window and why.

Approach

Build the week spine with generate_series over date_trunc('week', ...), which is Monday-aligned in Postgres, and CROSS JOIN it to the eligible consultant set, bounding each consultant by hire_date and termination_date. The spine is what converts an absent week into a zero week; no amount of COALESCE on a query driven by fct_time_entry can recover a row that was never there.
Aggregate time entries to (consultant_id, week_start) in a CTE first, then LEFT JOIN that onto the spine and COALESCE(hours, 0). Aggregating before the join keeps it one-to-one and removes any chance of fan-out.
Average with AVG(billable_hours) OVER (PARTITION BY consultant_id ORDER BY week_start ROWS BETWEEN 3 PRECEDING AND CURRENT ROW). Use ROWS rather than RANGE: on a dense spine they agree, but ROWS states the intent in row terms and does not change meaning if the spine ever develops a hole.
Decide what the first three weeks of each consultant's series mean. That window averages over fewer than four rows, so either label them as partial with a COUNT(*) OVER the same frame, or suppress them. Pick one and say so.
Hold the right edge of the window back past the measured timesheet backfill tail. Entries are created days to weeks after work_date, so the most recent weeks are systematically incomplete and the series will show a decline that reverses on the next refresh. Measure the tail rather than guessing a round number.

Worked solution (30 min)

Generate the 26 week_start values and confirm the count and the boundary dates before joining anything.
CROSS JOIN to billable-role consultants and apply the hire_date and termination_date bounds, then count rows per consultant.
Aggregate approved billable entries to consultant-week, LEFT JOIN onto the spine, COALESCE to 0.
Add the windowed average with an explicit ROWS frame, plus a COUNT over the same frame so partial windows are visible.
Hand-verify one consultant: print their 26 weeks and check that week 5's average equals the mean of weeks 2 through 5.

Expected result: 26 rows per eligible consultant, fewer only where hire_date or termination_date truncates the series. Zero-hour weeks are present with 0.00 hours. The trailing average on each row equals the arithmetic mean of that week and the three preceding weeks in the same consultant's series.

Verify it yourself

SUM(billable_hours) over the dense series equals SUM(hours) over approved billable entries in the same window, proving the LEFT JOIN added no rows and lost none.
Row count per consultant equals 26 minus any weeks outside their hire and termination bounds.
The hand-checked consultant's week-5 average matches the mean computed by hand from weeks 2 to 5.
No week_start in the output falls inside the held-back backfill tail.

Follow-up

Extend to a 13-week average. Does your choice between ROWS and RANGE change, and why?
Add the availability denominator so this becomes utilisation. What has to be subtracted from calendar hours, and what has to be prorated?
Reproduce this series as it would have looked 30 days ago. Which column do you filter on, and why do the two versions differ?

What metrics would you choose to evaluate a model for a churn prediction project?

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you manage your time when balancing multiple project priorities?

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Decompose the metric into the rates that drive it, and say which one you would check first.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you approach feature selection when dealing with high-dimensional data?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Say what you would check first and why it is the highest-information step.
Work from the decision backwards to the evidence you would need.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Can you explain the difference between supervised and unsupervised learning with a concrete example?

Medium

Approach

Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Design a bookings metric a partner incentive cannot game

Medium · gaming · bookings · guardrails · incentive design

Leadership wants to pay partners on bookings, defined today as SUM(contract_value_usd) from fct_engagement at sold_at. Using fct_engagement (engagement_id, client_id, prior_engagement_id, pricing_model, contract_value_usd, sold_at, status, close_reason) and fct_invoice_line (engagement_id, line_type, amount_usd, status), list three distinct ways a partner could raise the current metric without creating value for the firm, then propose a replacement primary metric plus two guardrails that close them. For each guardrail, name the lag in days or months before it can be read.

Approach

Enumerate the gaming paths concretely rather than abstractly. Discount to close: contract_value_usd rises while margin falls. Re-book the same scope as a change order, so a prior_engagement_id chain counts one body of work several times. Book an optimistic contract_value_usd on outcome_based or retainer work, where nothing pins the number until delivery. Book work with no delivery capacity behind it, which surfaces later as cancelled or closed_early.
Replace the primary with a value measure that only completes on delivery: chain-collapsed booked value, netted against the same chain's eventual credit_note and write_off lines, or booked value multiplied by a sold-margin estimate computed at signature.
Guardrail one, booking-to-realisation: SUM(amount_usd) on line_type IN ('fees','milestone') twelve months after sold_at, divided by contract_value_usd, reported per partner. This closes the discount path and the optimistic-value path. Lag: twelve months.
Guardrail two, early-termination rate: share of booked value on engagements reaching status IN ('cancelled','closed_early') within 90 days of start_date, plus a net-new versus expansion split built from prior_engagement_id. This closes the re-booking path and the no-capacity path. Lag: 90 days.
State the residual honestly. Every signal that detects gaming resolves after the bonus would be paid, so the workable design is a holdback, paying a share at signature and the balance against the trailing measures, not a cleverer single number.

Worked solution (30 min)

Build the extension chain with a recursive CTE over prior_engagement_id, collapse each chain to its root, and compute booked value per chain alongside the raw per-row sum.
For engagements sold at least twelve months ago, compute realised value as SUM(amount_usd) over line_type IN ('fees','milestone','credit_note','write_off') and take the ratio to contract_value_usd.
Break that ratio down by partner and by pricing_model; expect the widest spread on outcome_based and retainer rows.
Compute the 90-day early-termination share of booked value per partner.
Write the payout rule with the holdback split and each guardrail's lag stated in the rule itself.

Expected result: A primary metric of chain-collapsed booked value carrying a sold-margin condition, two guardrails (twelve-month booking-to-realisation per partner, and 90-day early-termination share), and an explicit holdback because both guardrails resolve after the signature date. Three named gaming paths, each mapped to the guardrail that closes it.

Verify it yourself

Chain-collapsed bookings are less than or equal to the row-level sum. If they are exactly equal, prior_engagement_id is unpopulated and the re-booking path is invisible in the data, which is itself the finding.
Booking-to-realisation has a visibly wider distribution on outcome_based and retainer engagements than on fixed_fee.
Every guardrail in the write-up carries a stated lag in days or months.

Follow-up

A partner argues the discount was necessary to win a strategic-tier account. What evidence would you accept for that, and is any of it auditable from these tables?
Your strongest guardrail lags twelve months. What do you pay on during the scheme's first year?

Fees per billable head fell nine percent; rank the causes

Hard · decomposition · seasonality · clustered inference

Recognised fees per billable head fell about 9% quarter over quarter and leadership wants a ranked cause by Friday. You have fct_time_entry, fct_invoice_line, fct_engagement and dim_consultant, plus a workday and regional holiday calendar. Decompose the move into available hours per head, billable utilisation and realised dollars per billable hour, separate calendar effects from behaviour, split within-segment change from mix, and deliver a ranked attribution with uncertainty. Name the one further cut you would run if the leading term is ambiguous.

Approach

Write the identity before computing anything: fees per head = (available hours per head) x (billable utilisation) x (realised dollars per billable hour), where the last term is recognised fees divided by approved billable hours. Verify the product reproduces the reported 9% exactly; if it does not, the scope of the three inputs disagrees and nothing downstream is trustworthy.
Decompose in logs: d ln(fees per head) = d ln(available per head) + d ln(utilisation) + d ln(realised rate). The log form is exactly additive for a product, so contributions sum to the total without an interaction residual and cannot attribute more than the whole move.
Strip the calendar first, because it is the term nobody can act on: available hours per head moves with scheduled workdays, regional public holidays, leave and any change in fte_fraction mix. Quantify how many of the nine points are calendar and quote the remainder as the behavioural move.
Split each surviving term by practice_area, level, region and pricing_model, and within each split separate within-segment change from mix using the weighted three-term decomposition, since a shift toward lower-rate levels reads as a rate decline that no individual rate produced.
Attach uncertainty at the unit that generates it: anything fee-side clusters at client_id because engagements in one account share a partner, a rate card and a team. With fewer than roughly forty clusters use a wild cluster bootstrap or a CR2 correction, since cluster-robust standard errors are downward-biased in that regime.
If realised rate leads and is ambiguous, the disambiguating cut is discount versus write-off versus fixed-fee overrun: compare invoiced fees to the standard-rate value of hours worked, and read written_off_hours and scope overrun by pricing_model.

Follow-up

Two of the nine points are calendar. How do you say that to an executive without sounding like you are explaining the number away?
Utilisation is up and realised rate is down. What single operational behaviour produces both, and how would you confirm it?
What would you need to see to claim this is a demand problem rather than a delivery or pricing one?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Stories where someone disagreed with you

Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.

Why do you want to work for a consulting firm like eXalt rather than a product-based company?

Medium · behavioural and stakeholder questions

Approach

Pick a story where you drove the decision, not one where you observed it.
Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

How do you handle feedback or criticism regarding your model results from a client?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What would you do differently if you ran that project again?
How did you know the outcome was caused by your change?

How do you handle missing or noisy data in a real-world production environment?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.
Quantify the outcome, including what you would not claim credit for.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?