Interview guides · Illumination Works

Illumination Works Data Scientist Interview Questions & Guide 2026

SQL practice for Illumination Works data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

As a Data Scientist at Illumination Works, you are at the forefront of solving complex data challenges for both commercial and government clients. This role isn't just about building models; it is about transforming raw, disparate data into actionable intelligence that drives strategic decisions. Because Illumination Works operates heavily in sectors like defense, aerospace, and supply chain, the impact of your work often scales to national or enterprise-level operations.

Ask early whether the loop includes an asynchronous take-home or a timed live case, because the two are graded on different things. A take-home is read as an artifact: the question you decided to answer, what you did about missing or malformed records, and a conclusion stated plainly enough for someone to act on. A reviewer who cannot rerun your notebook discounts the result whatever score is printed in it. Hold to the stated time box and write down what you would have done with more of it, since the follow-up round is usually a live defence of the same work.

Focus areas

Separate bookings, recognised revenue and collected cash
Correct late-entered timesheets before trending recent weeks
Cluster inference at the account, not the engagement

Overview

You will be embedded in cross-functional agile teams, working alongside data engineers, software developers, and domain experts. Whether you are developing predictive maintenance algorithms for aviation or optimizing logistics networks, your technical acumen will directly influence the success of critical missions and business objectives.

Expect a dynamic environment where adaptability is just as important as technical depth. You will be challenged to navigate strict data governance, work with unique datasets, and present complex findings to non-technical stakeholders who rely on your insights to lead effectively.

The Illumination Works interview process

Illumination Works candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Behavioral Interview

Rounds of this kind usually include one question about work that did not go well, and it is the part that carries the most information. Anyone can narrate a shipped win. What the interviewer learns from a project that stalled is how you behave without a result to hide behind: whether you noticed the problem yourself, how long it took, and who you told. Answers that route the failure onto a data pipeline or a reorganisation close the topic without answering it, and the follow-up comes back to your own part.

What to demonstrate

Whether you found the error yourself or someone else found it, and how long it sat before anyone knew
What you changed afterwards, stated as a check you now run rather than a lesson you now believe
Whether the mistake you choose has real cost attached, such as a quarter of misdirected roadmap or a metric that was reported upward, instead of one that flatters you

How to prepare

Choose a failure you caught yourself and be ready to say what tipped you off. A story where someone else caught it is still usable, but you will be asked why you missed it.
Write down the check you added afterwards and where it lives now, so the correction is a concrete artefact rather than a resolution.
Rehearse saying the cost out loud. Candidates shrink the number by instinct once the interviewer is in the room.

Technical Interview

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Case Study Evaluation

A case round is decided by whether you leave the interviewer with a recommendation, not by how much analysis you narrate on the way there. The prompt is open on purpose, so the first job is to convert it into a decision someone could act on: ask what would be done differently depending on the answer. From there name the quantity that would settle it, state the assumptions you need, and commit. Candidates who cover more ground than anyone expected and still end on "it depends" score below candidates who scoped narrowly and said what they would do.

What to demonstrate

Whether the version of the question you choose to answer is genuinely narrower than the prompt and still worth answering
Whether the recommendation arrives as an action with a number attached, rather than as a summary of what you looked at
Whether assumptions are stated at the moment you rely on them, instead of collected into a disclaimer at the end
Whether you notice when a branch you are exploring would not change the decision either way

How to prepare

Take six open prompts and write only the scoping move for each: the one-sentence question you would actually answer and the decision it feeds. Give yourself three minutes per prompt and stop there.
Put a five-minute warning into every practice case and force a closing statement that names the action, the result that would justify it, and the result that would reverse it.
Record one case and count how long you talked before naming a measurable quantity. Past roughly five minutes, what you are calling scoping is narration.

Final Round Interview

A day of back-to-back interviews samples your floor, not your ceiling. Four hours in, the habits that carry a good answer are the first to go: restating the question before solving it, asking what the data would have to look like, checking a number before quoting it. What the day decides is whether the tired version of you is still someone to leave alone with an ambiguous problem. The round that sinks a candidate is usually not the hardest one. It is the one immediately after the round that went badly.

What to demonstrate

Whether the late rounds get the same clarifying questions as the first one, or whether you start answering immediately to save effort
Whether a weak answer stays in the room it happened in, instead of following you into the next conversation as apology or distraction
Whether the quality of your questions holds up, since fatigue removes curiosity about the problem before it removes knowledge of the method

How to prepare

Rehearse the length, not just the content: book four mock interviews of different types in one afternoon with short gaps, because the one you need to observe is the fourth
Put the two or three questions you ask at the start of any problem on a card in front of you, so that under fatigue it is a habit you run rather than a decision you make
Decide in advance what the gap between rooms is for: water, one line of notes on anything you promised to follow up, and an explicit close on the round that just ended so it does not travel
Prepare a different closing question for each interviewer, so the end of a long day does not produce the same one four times

Where candidates lose points

Treating accounts as independent observations

Revenue is concentrated: a small number of client_ids typically carries a large share of fees, and engagements within one account share a partner, a rate card and a delivery team. Ordinary standard errors computed over engagements therefore understate uncertainty badly. Cluster at client_id, and with fewer than roughly 40 clusters use a wild cluster bootstrap or a CR2 correction, because cluster-robust standard errors are downward-biased in that regime and will manufacture significance that a replication will not reproduce.

Computing days-to-pay or proposal cycle time over completed records only

At any snapshot date, invoices that have already been paid are disproportionately the fast ones, and proposals that already have a decision are disproportionately the quick ones. Averaging over the completed set alone biases both numbers downward, and the bias grows exactly when the business is deteriorating, because the slow cases are the ones still open. Unpaid and undecided records are right-censored; use Kaplan-Meier or a restricted mean up to a fixed horizon, and never fill paid_at with a placeholder.

Extrapolating a first-week lift inflated by novelty effects

Plot the treatment effect by days since first exposure instead of quoting one pooled average. A lift that decays toward zero across the test window is behaviour that will not persist, and annualising it produces a forecast that misses by an order of magnitude.

Treating a non-significant result as proof of no effect

Say whether the confidence interval excludes the effect sizes you would have cared about. If it does not, the honest reading is that the test was underpowered, so report the minimum detectable effect the design could have found and what sample size would resolve it.

Questions & practice

What is the difference between L1 and L2 regularization, and when would you use each?

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Sanity-check the answer against a simple bound or a simulated case.
Say what the estimate is of, and over what population it generalises.

Follow-up

What sample size would you need to detect an effect half this size?
How would you explain this result to someone who does not know statistics?

Explain the concept of cross-validation and why it is important.

Medium · statistics and probability

Approach

Sanity-check the answer against a simple bound or a simulated case.
Say what the estimate is of, and over what population it generalises.
Translate the result into the decision it informs, in one plain sentence.

Follow-up

How would you explain this result to someone who does not know statistics?
What sample size would you need to detect an effect half this size?

A client asks for a machine learning model, but you realize a simple rule-based system would solve their problem faster and cheaper. How do you navigate this conversation?

Medium · machine learning and modelling

Approach

Say how the offline result would be validated online before it is trusted.
Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

Kaplan-Meier days to payment with unpaid invoices censored

Hard · survival analysis · censoring · kaplan-meier

invoice_lines has invoice_line_id, engagement_id, line_type, issued_at, due_date, paid_at (null when unpaid), amount_usd and status in draft, issued, partially_paid, paid, disputed, written_off. At a given snapshot_date, estimate the median days from issue to full payment. Implement Kaplan-Meier yourself; no lifelines or equivalent. Treat issued, partially_paid and disputed as right-censored at snapshot_date, and decide and justify what to do with written_off. Report the naive mean over paid lines alongside your estimate and state the sign of its bias.

Approach

Build the duration and event table explicitly. Drop draft lines, which have no clock. For status paid, duration = (paid_at - issued_at).days with event = 1. For issued, partially_paid and disputed, duration = (snapshot_date - issued_at).days with event = 0.
Handle written_off as a competing event rather than a censor. Censoring it makes the estimator answer 'time to payment if written-off invoices could still pay', which overstates collection. Either report a cumulative-incidence version alongside, or censor them and say plainly that the result is conditional on eventual collection.
Implement the estimator directly: sort unique event times, at each t take n_i as the count with duration >= t and d_i as the payments at exactly t, and accumulate S(t) = product of (1 - d_i / n_i). Censored rows leave the risk set without causing a drop, which is the whole mechanism and the reason the answer differs from any completed-case average.
Read the median as min{t : S(t) <= 0.5}. If S never reaches 0.5 within observed follow-up, report 'not reached'; interpolating past the last observation invents data that the snapshot does not contain.
Add Greenwood's formula for Var(S(t)) to put a band on the curve, then invert the band at 0.5 for an interval on the median rather than quoting a point estimate alone.
Compare against the mean over paid lines only and name the direction: at any snapshot the paid set over-represents fast payers, so the naive mean is biased low, and the bias widens exactly when collections deteriorate.

Worked solution (40 min)

Assemble a frame of (duration_days, event) per non-draft line, with snapshot-based durations for the three open statuses and an explicit, documented rule for written_off.
times = np.sort(df.loc[df.event == 1, 'duration_days'].unique()); for each t, n = (df.duration_days >= t).sum() and d = ((df.duration_days == t) & df.event).sum().
S = np.cumprod(1 - d / n) aligned to times; prepend S(0) = 1.0 so the curve starts where it should.
median = times[np.argmax(S <= 0.5)] if (S <= 0.5).any() else 'not reached'; compute Greenwood variance as S(t)^2 * cumsum(d / (n * (n - d))).
naive = df.loc[df.event == 1, 'duration_days'].mean(); report both, plus the censored share, since the gap between the two scales with it.

Expected result: A step function starting at S(0) = 1.0 that drops only at observed payment days, an integer median or an explicit 'not reached', a Greenwood-based interval, and a naive completed-case mean that sits below the Kaplan-Meier median on data with material censoring.

Verify it yourself

S is non-increasing, starts at 1.0, and never goes below 0; the number at risk at the first event time equals the count of all non-draft lines.
Re-running with every censored row deleted makes the Kaplan-Meier median equal the empirical median of paid durations, since with no censoring the estimator reduces to the empirical survival function.
A six-row toy example worked by hand reproduces the implementation's S values exactly.
The gap between the naive mean and the Kaplan-Meier median grows when you move snapshot_date earlier, which raises the censored share.

Follow-up

A large account moved to a monthly payment run. Is administrative censoring still independent of payment time, and what would you check?
Finance wants one DSO number against a target. What do you give them, and what do you refuse to give them?
Stratify by line_type. Do milestone lines behave like fees lines, and what would it mean for the firm if they do not?

Write a Python function to identify and remove duplicate records from a large dataset.

Medium · sql and data manipulation

Approach

Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
Compute rates by summing numerator and denominator separately, never by averaging rates.
State the window function and its partition and ordering out loud before writing it.

Follow-up

How does the query change if the join becomes one-to-many?
What breaks if events arrive late or out of order?

Write a script to parse a JSON file containing nested dictionaries and flatten it into a tabular format.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

How would you verify this result without re-running the same query?
What breaks if events arrive late or out of order?

Given a table of user transactions, write a SQL query to find the 7-day rolling average of transaction amounts.

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Compute rates by summing numerator and denominator separately, never by averaging rates.
Say which table is the grain you start from, and join outward from it.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Cost hours at the rate in force on the work date

Medium · type-2 scd · effective dating · date boundaries

dim_consultant is a type-2 dimension with consultant_sk, consultant_id, level, practice_area, standard_cost_rate_usd, effective_from TIMESTAMPTZ, effective_to TIMESTAMPTZ NULL and is_current. fct_time_entry holds consultant_id, work_date DATE, hours, status and a cost_rate_usd snapshotted when the row was written. Recompute last year's delivery cost per practice_area by joining each approved entry to the dimension row valid on its work_date, then reconcile against the snapshotted rate and list entries where the two differ by more than one cent. Say what a session timezone change does to your join predicate.

Approach

Join on a half-open interval: te.work_date >= d.effective_from AND (d.effective_to IS NULL OR te.work_date < d.effective_to). This assumes effective_to holds the next row's effective_from. Verify that on one consultant with three versions before committing, because if effective_to instead stores the last valid instant the upper bound must be inclusive, and getting it backwards either double counts or drops a boundary day.
Refuse is_current = TRUE as a join predicate. It prices last year's hours at today's rate, so a promotion or an annual rate change silently restates history, and the restatement is invisible in the output.
Be explicit about the DATE to TIMESTAMPTZ comparison. Postgres casts work_date to midnight in the session TimeZone while effective_from is an absolute instant, so a version that opens at 2026-03-01 00:00 UTC lands on one side of a 2026-03-01 work_date under a UTC session and the other side under a negative-offset session. Normalise both to one convention, for example comparing work_date against (effective_from AT TIME ZONE 'UTC')::date, and write down which convention the warehouse uses.
Validate one-to-one before trusting any sum: group the joined result by time_entry_id and assert no id appears twice. Overlapping validity intervals in the dimension are a real defect and present as inflated cost, not as an error message.
Reconcile the recomputed rate against the snapshotted cost_rate_usd. Differences are either late-corrected dimension rows or entries snapshotted at the wrong moment; both are findings with a named cause, not rounding noise to be smoothed over.

Worked solution (30 min)

Count approved time entries in the window; this is the exact row count the join must preserve.
Write the half-open interval join and immediately compare the post-join row count against that number.
Check for duplicates by grouping on time_entry_id and, if any appear, inspect that consultant's dimension rows for overlap.
Aggregate cost by practice_area using the dimension rate, then again using the snapshotted rate, and diff the two totals.
List the individual entries whose two rates differ by more than a cent, with both rates and the dimension row's validity interval.

Expected result: A cost total per practice_area plus an exception list. The joined row count equals the approved-entry count exactly, and the two cost totals agree except for the entries on the exception list.

Verify it yourself

COUNT() after the join equals COUNT() before it.
No time_entry_id appears more than once in the joined set.
Recomputed total cost is within the exception list's aggregate difference of the snapshot-based total.
Re-running after SET TIME ZONE to a non-UTC zone assigns every entry to the same dimension row, or the query documents exactly where it does not.

Follow-up

How would you detect overlapping or gapped validity intervals across the whole dimension in one query?
Which of the two rates belongs in a client-facing margin number, and which in an internal cost review?
A consultant changes practice_area mid-engagement. Which practice_area owns the hours, and what breaks either way?

How do you prioritize tasks when working on multiple client deliverables with competing deadlines?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Explain how you would optimize a Pandas script that is currently running out of memory.

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Define competitive win rate for a quarterly pipeline review

Easy · win rate · censoring · denominators

Pipeline reporting currently shows win rate as won proposals divided by all proposals created. Using fct_proposal (proposal_id, expected_value_usd, stage, loss_reason, is_competitive, created_at, submitted_at, decided_at), write a definition you would defend to a finance lead: numerator, denominator, window and every exclusion. Decide whether to weight by count or by expected_value_usd, and justify the choice. State what happens to proposals still in 'qualifying', 'scoping' or 'submitted' at the snapshot date, and why dropping them is not a neutral act.

Approach

Cohort on decided_at, not created_at. A created_at cohort mixes decided and undecided proposals, so the rate for a recent quarter is computed over whichever deals resolved fastest and drifts upward on every refresh.
Set the denominator to stage IN ('won','lost'). Excluding 'withdrawn' and 'no_decision' is defensible only if you report their count and value alongside, because they are not missing at random: they skew toward deals that were heading for a loss.
Filter is_competitive = TRUE. Sole-sourced follow-on work closes at near certainty and mechanically inflates the rate, so it belongs on its own line rather than inside the headline number.
Report count-weighted and value-weighted side by side. They diverge when large proposals lose more often than small ones, which is the diagnostically useful case; the value-weighted version is the one that ties to revenue.
Treat open proposals as right-censored rather than absent. Either restrict to cohorts old enough that most proposals have decided, picking that horizon off a Kaplan-Meier cycle-time curve, or publish the rate with the undecided count and value attached to it.

Worked solution (25 min)

Compute the naive rate (won / all created) and the proposed rate (won / won+lost, is_competitive = TRUE, cohorted on decided_at) for the same four quarters.
For each quarter, count and sum the value of proposals in 'withdrawn' and 'no_decision', and of proposals still open at the snapshot.
Compute the value-weighted version and compare it with the count-weighted one.
Re-run the most recent quarter as it would have looked 30 days ago and record how much each definition moved.

Expected result: Two numbers per trailing quarter, count-weighted and value-weighted competitive win rate, each with the denominator predicate written out in full, cohorted on decided_at, plus a reported count and value for withdrawn, no_decision and still-open proposals. The horizon is chosen so the undecided share is small rather than set to a round number of days.

Verify it yourself

The value-weighted and count-weighted rates differ. If they are identical, expected_value_usd is either constant or lost in a join.
Re-running a given quarter two weeks later moves the decided_at-cohorted number by much less than it moves the created_at version.
Withdrawn and no_decision appear as reported counts, not as a silent exclusion.

Follow-up

Win rate rose four points while proposal volume fell by a third. Which do you report first, and what single query separates a genuine quality shift from a volume shift?
How would you detect partners reclassifying likely losses as 'withdrawn' to protect the rate?

Realisation rate steps up overnight after a warehouse release

Medium · definition change · realisation · restatement

Realisation rate for one practice area steps from 0.93 to 1.02 on a single reporting date. Bill rates, hours worked and the client set are unchanged. The metric is SUM(amount_usd) from fct_invoice_line over SUM(hours * bill_rate_usd) from approved billable fct_time_entry rows for the same engagement and period. A warehouse release shipped that week. Determine what in the definition can produce a level shift with no operational change, and deliver a back-restated series plus the definition you would keep.

Approach

Treat the shape as evidence: a definition change is a discontinuity at one processing boundary, while a behaviour change ramps over weeks. Plot daily, not monthly, to see which one this is before forming any hypothesis.
Diff the metric's lineage across the release and enumerate the candidate predicates: which line_type values enter the numerator (dropping credit_note and write_off inflates it), whether the period join moved from period_start_date to issued_at, and whether the denominator counts status = 'approved' only or now includes 'submitted'.
Run both definitions over the same fixed pre-release month. If old and new differ by approximately the size of the step, the step is definitional; if they agree, look for a behaviour change coinciding with the release date.
Sanity-check the new level against the metric's physics: values above 1.0 are legitimate only on fixed_fee work delivered under its hour budget, so a practice dominated by time_and_materials sitting at 1.02 is itself a red flag.
Restate the full history under the chosen definition, publish old and new series together with the changeover marked, and add a reconciliation test that recomputes one known engagement-month from raw rows.

Follow-up

Which definition is correct, and what does excluding write-offs from the numerator do to the guardrail relationship with utilisation?
How do you prevent the next release from silently restating a headline metric?
If the release also changed the denominator's status filter, which direction does that push realisation and by roughly how much?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

Give an example of a time you disagreed with a team member about a technical approach. How did you resolve it?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
State the situation in two sentences and spend the rest on your reasoning.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

Answering a request to redefine the utilisation denominator

Hard · denominator integrity · restatement · escalation

Before a quarterly review, a practice lead asks for two changes to reported billable utilisation: drop consultants allocated to engagements with status = 'paused' out of the denominator, and count approved travel hours booked to client_delivery as billable. Together the changes move reported utilisation from 71 to 79 percent. You must answer in writing today. Give your answer, the principle you apply to each change separately, and what you do if you are overruled.

Approach

Name the probe: whether you can hold a measurement line by adjudicating each proposal on its merits, rather than treating the request as bad faith or accepting it to avoid a fight.
Adjudicate the paused-engagement change against what the denominator means. Billable utilisation measures the share of available consultant hours that were sold. A consultant whose engagement is paused is still available and still costs salary, which is precisely the situation the metric exists to reveal. Removing them defines away the signal, so this one is refused on definitional grounds, not on suspicion.
Adjudicate the travel change as an empirical question, because it is one. If those hours are billed to the client, is_billable and bill_rate_usd should already reflect it; if they are charged and then written off or never invoiced, counting them raises utilisation while realisation falls. Check the invoice lines for the same engagement and period rather than arguing about principle.
Quantify each change separately, and the interaction. An eight-point move made of one point and seven points is a different conversation from two four-point moves, and the lead cannot make a good decision without that split.
If a change is accepted, restate history on the new definition and publish the old series alongside it with the date of the change. A definition applied forward only makes every period-over-period comparison meaningless. If overruled, comply with the restatement and the footnote, and escalate only the part that is a misstatement of fact rather than a defensible choice.

Follow-up

Which of the two changes would you accept, and what evidence flips your answer on the other one?
The lead wants the new definition applied only from this quarter forward. Why is that worse than applying it to history, and how do you explain that in one sentence?

Disagreeing with a proposed utilisation target using realisation evidence

Medium · realisation · guardrail metrics · observational evidence

A delivery lead proposes raising the billable utilisation target for analyst through senior_consultant from 72 to 85 percent. You have fct_time_entry, including is_billable, bill_rate_usd and written_off_hours, and fct_invoice_line. You believe the target will raise reported utilisation and lower fees. Prepare the disagreement: the evidence you pull, the mechanism you name, the metric pair you propose instead, and the condition under which you would concede that the target is correct.

Approach

Name the probe: whether you disagree with a mechanism and a measurement, or with an opinion about a metric being bad.
State the substitution precisely. Utilisation counts approved hours with is_billable = TRUE. An hour that is charged to the client and later written off stays in that numerator, so utilisation is unaffected while realisation, fees divided by hours times bill_rate_usd, falls and margin falls with it. That is the exact channel by which a higher target can raise the reported number and lower revenue.
Pull the evidence at consultant-month grain: plot realisation and the write-off share, written_off_hours over billable hours, against utilisation decile. If the current top decile already shows lower realisation, the proposed target moves a large share of the staff into that regime.
Stratify before concluding. Fixed_fee teams can show high utilisation and high realisation for reasons that have nothing to do with the proposal, so run the comparison within pricing_model and report the mix.
Propose the pair rather than the veto: utilisation published with realisation and write-off rate as standing guardrails, with the threshold at which the combination is net positive stated in advance. Then name your concession condition: if the top utilisation decile shows no realisation penalty and bench hours are the binding constraint, the target is right and you will say so.

Follow-up

Utilisation and realisation are computed from overlapping hours. Does that make the relationship you found mechanical rather than behavioural?
How many consultant-months would you need to detect a three-point realisation move, and does the firm have them?