Interview guides · Avenue Code

Avenue Code Data Scientist Interview Questions & Guide 2026

SQL practice for Avenue Code data scientist candidates: window functions, funnel and retention queries, joins that quietly drop rows.

The 60-second brief

As a Data Scientist at Avenue Code, you are stepping into a highly dynamic, client-facing environment where your technical expertise directly shapes enterprise-level digital transformations. Avenue Code is a premier software consultancy, meaning our data teams do not just work on isolated internal tools; they build robust, scalable data solutions for top-tier global clients across various industries, including retail, finance, and automotive.

Scope your preparation by the data you would actually touch, because the title will not tell you. A seat that lives in event logs and weekly readouts rewards fluency in aggregation and metric definitions; a seat that owns a model in production rewards fluency in train/serve skew, retraining cadence and drift monitoring. The fastest way to find out which one you are interviewing for is to ask what the team shipped last quarter and what it gets paged about.

Focus areas

Compute utilisation against a defended availability denominator
Cluster inference at the account, not the engagement
Reconstruct pipeline stage history from mutable rows

Overview

This role requires a unique blend of deep technical rigor and exceptional business acumen. You will be tasked with translating ambiguous client problems into structured data pipelines, predictive models, and actionable insights. Because of the consulting nature of the business, the impact of this position is massive—you are often the bridge between complex machine learning concepts and tangible business value for our partners.

You can expect to work alongside cross-functional teams of engineers, product managers, and client stakeholders. The problems you solve will vary from project to project, meaning you will constantly be challenged to adapt your toolset, learn new domains, and scale solutions to meet enterprise demands. If you thrive on variety, strategic influence, and high-impact problem-solving, this is the perfect environment for you.

The Avenue Code interview process

Avenue Code candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Data Scientist covers at least four different jobs: experimentation, product analytics, causal work on observational data, and applied modelling that ships into a system. A screening call is the cheapest place to find out which of them is being hired for, and doing that diagnosis openly reads as senior rather than fussy. Ask what the last few pieces of work on the team actually were, and roughly how a week splits between querying, modelling and stakeholder time. Then say which parts of that you have done and which you have not. Claiming the whole range is the fastest way to be caught one round later.

What to demonstrate

Whether you can distinguish the flavours of the role and locate your own experience inside one of them honestly
Whether you name what you have not done instead of stretching to cover every line of the posting
Whether your hard constraints (notice period, location, work authorisation, level) surface now rather than at offer stage

How to prepare

Map the last two years of your time into rough percentages across query writing, experiment design, modelling and stakeholder work, so a question about scope has a real answer
Mark every responsibility in the posting as done, adjacent or new, and prepare one sentence for each adjacent item naming the closest thing you have actually built
Decide which logistics are non-negotiable before the call so you can state them in one sentence rather than negotiating live

Deep-Dive Technical Interview

Before anything else, this round is a reading test. You are given a small schema and a question phrased in business language, and most of the difficulty sits in the gap between them. Who counts as an active user, does a refunded order still count as an order, is that date column an event time or a load time. Weak answers start typing immediately and compute something precise about the wrong population. Strong ones pin the definition in one sentence, name the column that encodes it, then write the query. On a timed assessment with nobody to tell, write the definition in a comment anyway.

What to demonstrate

Whether an ambiguous term becomes a specific column and filter before any computation happens
Whether you read the schema for keys and cardinality rather than only for column names
Whether the result answers the question at the grain it was asked at, per user or per session or per day

How to prepare

Take three metrics you already use and write down the exact filter and exact grain behind each, then practise stating one of them in a single sentence out loud
On a schema you have never seen, spend the first minute writing what one row of each table means and which key it is unique on, then predict which joins can duplicate rows
Rehearse a version where the definition changes halfway through, and edit the query you have instead of starting over

Where candidates lose points

Treating accounts as independent observations

Revenue is concentrated: a small number of client_ids typically carries a large share of fees, and engagements within one account share a partner, a rate card and a delivery team. Ordinary standard errors computed over engagements therefore understate uncertainty badly. Cluster at client_id, and with fewer than roughly 40 clusters use a wild cluster bootstrap or a CR2 correction, because cluster-robust standard errors are downward-biased in that regime and will manufacture significance that a replication will not reproduce.

Modelling win rate on proposals with a recorded outcome, using fields written after the decision

Two failures compound here. First, stage IN ('withdrawn','no_decision') is not missing at random: those are disproportionately deals that were going to be lost, so training on won-plus-lost only inflates apparent win rate and distorts the coefficients. Second, fields like engagement_id, final scope and revised pricing are populated after the outcome is known, so including them leaks the label and produces a model with excellent backtest accuracy and no forward value. Restrict features to values knowable at submitted_at.

Analysing at a different unit than the one randomised

Say out loud what was randomised (user, device, account, cluster) and make the analysis unit match, or account for the clustering with cluster-robust standard errors, the delta method, or aggregation up to the randomised unit. Randomising users and then running a test over sessions understates variance and inflates the false-positive rate.

Writing SQL without stating NULL and tie-breaking behaviour

Before calling a query finished, say what it does with NULLs, ties and empty groups. NOT IN against a subquery containing a single NULL returns no rows at all, and RANK, DENSE_RANK and ROW_NUMBER differ precisely on ties, so name which one the question requires.

Questions & practice

This category ensures your theoretical foundation is strong enough to support your practical decisions.

Medium · statistics and probability

Approach

Quantify uncertainty explicitly rather than reporting a point estimate alone.
Sanity-check the answer against a simple bound or a simulated case.
Say what the estimate is of, and over what population it generalises.

Follow-up

Which assumption here is most likely to be violated in practice?
How would you explain this result to someone who does not know statistics?

You are given a dataset with 1000 features and only 500 rows. How do you build a predictive model without overfitting?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Pick an evaluation metric that matches the cost of each error type, not a default.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

We have a model that detects fraudulent transactions. The client complains it is blocking too many legitimate users. How do you approach fixing this?

Medium · machine learning and modelling

Approach

Check what information would not exist at prediction time, and exclude it.
Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
Where could label leakage enter this setup?

Kaplan-Meier days to payment with unpaid invoices censored

Hard · survival analysis · censoring · kaplan-meier

invoice_lines has invoice_line_id, engagement_id, line_type, issued_at, due_date, paid_at (null when unpaid), amount_usd and status in draft, issued, partially_paid, paid, disputed, written_off. At a given snapshot_date, estimate the median days from issue to full payment. Implement Kaplan-Meier yourself; no lifelines or equivalent. Treat issued, partially_paid and disputed as right-censored at snapshot_date, and decide and justify what to do with written_off. Report the naive mean over paid lines alongside your estimate and state the sign of its bias.

Approach

Build the duration and event table explicitly. Drop draft lines, which have no clock. For status paid, duration = (paid_at - issued_at).days with event = 1. For issued, partially_paid and disputed, duration = (snapshot_date - issued_at).days with event = 0.
Handle written_off as a competing event rather than a censor. Censoring it makes the estimator answer 'time to payment if written-off invoices could still pay', which overstates collection. Either report a cumulative-incidence version alongside, or censor them and say plainly that the result is conditional on eventual collection.
Implement the estimator directly: sort unique event times, at each t take n_i as the count with duration >= t and d_i as the payments at exactly t, and accumulate S(t) = product of (1 - d_i / n_i). Censored rows leave the risk set without causing a drop, which is the whole mechanism and the reason the answer differs from any completed-case average.
Read the median as min{t : S(t) <= 0.5}. If S never reaches 0.5 within observed follow-up, report 'not reached'; interpolating past the last observation invents data that the snapshot does not contain.
Add Greenwood's formula for Var(S(t)) to put a band on the curve, then invert the band at 0.5 for an interval on the median rather than quoting a point estimate alone.
Compare against the mean over paid lines only and name the direction: at any snapshot the paid set over-represents fast payers, so the naive mean is biased low, and the bias widens exactly when collections deteriorate.

Worked solution (40 min)

Assemble a frame of (duration_days, event) per non-draft line, with snapshot-based durations for the three open statuses and an explicit, documented rule for written_off.
times = np.sort(df.loc[df.event == 1, 'duration_days'].unique()); for each t, n = (df.duration_days >= t).sum() and d = ((df.duration_days == t) & df.event).sum().
S = np.cumprod(1 - d / n) aligned to times; prepend S(0) = 1.0 so the curve starts where it should.
median = times[np.argmax(S <= 0.5)] if (S <= 0.5).any() else 'not reached'; compute Greenwood variance as S(t)^2 * cumsum(d / (n * (n - d))).
naive = df.loc[df.event == 1, 'duration_days'].mean(); report both, plus the censored share, since the gap between the two scales with it.

Expected result: A step function starting at S(0) = 1.0 that drops only at observed payment days, an integer median or an explicit 'not reached', a Greenwood-based interval, and a naive completed-case mean that sits below the Kaplan-Meier median on data with material censoring.

Verify it yourself

S is non-increasing, starts at 1.0, and never goes below 0; the number at risk at the first event time equals the count of all non-draft lines.
Re-running with every censored row deleted makes the Kaplan-Meier median equal the empirical median of paid durations, since with no censoring the estimator reduces to the empirical survival function.
A six-row toy example worked by hand reproduces the implementation's S values exactly.
The gap between the naive mean and the Kaplan-Meier median grows when you move snapshot_date earlier, which raises the censored share.

Follow-up

A large account moved to a monthly payment run. Is administrative censoring still independent of payment time, and what would you check?
Finance wants one DSO number against a target. What do you give them, and what do you refuse to give them?
Stratify by line_type. Do milestone lines behave like fees lines, and what would it mean for the firm if they do not?

A client wants to use machine learning to solve a problem that could easily be solved with a simple rule-based SQL query. How do you handle this conversation?

Medium · sql and data manipulation

Approach

Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which table is the grain you start from, and join outward from it.
Compute rates by summing numerator and denominator separately, never by averaging rates.

Follow-up

What breaks if events arrive late or out of order?
How would you verify this result without re-running the same query?

Deduplicate to the latest issued invoice line per engagement

Easy · window functions · dedup · determinism

fct_invoice_line holds invoice_line_id, invoice_id, engagement_id, line_type, period_end_date, amount_usd, issued_at and status; draft lines have issued_at NULL. Return exactly one row per engagement_id, its most recently issued line, with invoice_line_id, line_type, amount_usd and issued_at. Two lines on the same engagement can share an issued_at value to the second, so the result must be identical on every run against unchanged data. Write it with a window function, then say in one sentence when DISTINCT ON would be the better choice.

Approach

Filter to issued lines (issued_at IS NOT NULL, status <> 'draft') before ranking. Ranking first and filtering afterwards can leave an engagement whose top-ranked row was a draft with no row at all in the output.
Rank with ROW_NUMBER() OVER (PARTITION BY engagement_id ORDER BY issued_at DESC, invoice_line_id DESC). The surrogate key as the final ORDER BY term is what makes ties resolve the same way on every execution; without it the planner is free to return either tied row.
Wrap the ranked query in a CTE or subquery and filter rn = 1, since a window function cannot appear in the WHERE clause of the query that computes it.
Name why RANK() is wrong here: on tied issued_at it assigns 1 to both rows, so the result carries two rows for one engagement and any downstream SUM double counts.
Compare against DISTINCT ON (engagement_id) ... ORDER BY engagement_id, issued_at DESC, invoice_line_id DESC: same answer, one sort, usually cheaper, but the window form generalises the moment you also want rn <= 3 or a second ranking in the same pass.

Worked solution (15 min)

Count distinct engagement_id over the filtered (issued, non-draft) line set. This is your target row count.
Write the ROW_NUMBER CTE with both ORDER BY terms and select rn = 1.
Assert the contract with GROUP BY engagement_id HAVING COUNT(*) > 1 over the result; it must return nothing.
Deliberately construct or find an engagement with two lines sharing issued_at and confirm exactly one survives.
Rewrite with DISTINCT ON and diff the two result sets row by row.

Expected result: One row per engagement that has at least one non-draft issued line. Row count equals COUNT(DISTINCT engagement_id) over that filtered set. Engagements holding only draft lines are absent, which is correct here; listing them is an anti-join, not a dedup.

Verify it yourself

Result row count equals COUNT(DISTINCT engagement_id) on the filtered input.
GROUP BY engagement_id HAVING COUNT(*) > 1 over the result returns zero rows.
Two consecutive runs produce byte-identical output when ordered by engagement_id.
The DISTINCT ON rewrite returns the same invoice_line_id for every engagement.

Follow-up

Now return the latest line per engagement per line_type. What changes in the PARTITION BY, and what does that do to the row count?
Return the latest and second-latest line together. Which of the two constructs survives that requirement?
issued_at is TIMESTAMPTZ and finance reports on a single local calendar. Where can 'latest' flip between the two conventions?

How do you prioritize tasks when working on multiple client deliverables with competing deadlines?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

These questions test your ability to structure a problem from scratch. Interviewers want to hear your thought process, the assumptions you make, and how you validate your approach.

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
Which segment would you cut first, and what would that rule out?

The client wants to predict inventory shortages for the upcoming holiday season. Walk me through how you would build this solution from data collection to deployment.

Medium · product sense and metrics

Approach

State what result would change your recommendation, so the answer is falsifiable.
Restate the decision this analysis has to support, and who acts on the answer.
Decompose the metric into the rates that drive it, and say which one you would check first.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
Which segment would you cut first, and what would that rule out?

Resolve a real conflict between margin and account retention

Medium · guardrails · leverage · trade-offs · decision rules

A leverage push will staff more analysts and fewer managers on delivery. Engagement gross margin is the primary metric and twelve-month repeat purchase on client_id is the guardrail. Using fct_time_entry (consultant_id, engagement_id, hours, cost_rate_usd, bill_rate_usd, written_off_hours), dim_consultant (consultant_id, level, standard_cost_rate_usd) and fct_invoice_line (engagement_id, line_type, amount_usd, status), explain why the two genuinely move in opposite directions here rather than one being a measurement artifact, then specify the decision rule you would write before rollout, including the threshold at which the guardrail stops the change.

Approach

Write the per-hour arithmetic before quoting any number. Replacing one manager hour with k analyst hours changes fixed_fee margin by c_m - k x c_a, which is positive only while k < c_m / c_a. On uncapped time_and_materials it changes contribution by k x (b_a - c_a) - (b_m - c_m). Both inequalities turn on k, the efficiency ratio, and none of these tables record it.
Bound k from the one place it leaks: hours per comparable deliverable on past engagements with different level mixes. Concede the confound immediately, since the engagements that got junior teams were selected for being simpler.
Establish the guardrail's mechanism rather than asserting a correlation. More analyst hours per manager hour raises review and rework load, which appears as written_off_hours and disputed invoice lines within weeks, and in repeat purchase only after twelve months.
Substitute a leading guardrail for the lagging one deliberately and say so. Gate on write-off rate and disputed-line share, and state the residual risk: a client who pays every invoice in full and quietly does not return is invisible to both.
Write the rule with a price attached. Proceed while the within-pricing-model margin gain exceeds X basis points and write-off rate rises by less than Y points on the same engagements; the guardrail stops the change regardless of the margin gain. Fix the unit as engagement within pricing_model, clustered at client_id, because a leverage change lands unevenly across accounts and engagements inside one account are not independent.

Worked solution (35 min)

Pull current bill and cost rates by level from recent fct_time_entry rows and dim_consultant, and compute the break-even k for fixed_fee (c_m / c_a) and the break-even k for uncapped time_and_materials from k x (b_a - c_a) = b_m - c_m.
Estimate a plausible range for k from historical hours per deliverable across differing level mixes, and state the selection confound in the same sentence as the estimate.
Compute write-off rate and disputed-line share by engagement for the trailing four quarters, stratified by pricing_model, to fix the pre-period baseline the threshold is measured against.
Write the decision rule with both numeric thresholds, the override direction, and the clustering unit named.
Evaluate the rule against one historical engagement cohort that already ran junior-heavy, and record whether it would have fired.

Expected result: A decision rule carrying two numeric thresholds and an explicit override direction, built on the per-hour inequalities (fixed_fee improves while k < c_m / c_a; uncapped time_and_materials improves while k x (b_a - c_a) > b_m - c_m), with k named as unmeasured in this schema and bounded rather than assumed to equal 1. Leading guardrails identified as substitutes for twelve-month repeat purchase, with the substitution risk stated.

Verify it yourself

Setting k = 1 makes the fixed_fee case trivially favourable. If your rule still produces a hard decision at k = 1, you have assumed away the quantity that decides it.
The two pricing models yield different break-even k on the same rate card, so a single firm-wide threshold is wrong by construction.
The stated threshold is expressed in the same units as the metric it gates, so a reader can confirm from the scorecard alone whether the rule fired correctly.

Follow-up

Margin is up 300 basis points and write-off rate is up 1.5 points. Does your rule fire? Show the arithmetic that decides it.
What would convince you the write-off rise is a change in concession policy rather than a change in delivery quality?

Realisation rate steps up overnight after a warehouse release

Medium · definition change · realisation · restatement

Realisation rate for one practice area steps from 0.93 to 1.02 on a single reporting date. Bill rates, hours worked and the client set are unchanged. The metric is SUM(amount_usd) from fct_invoice_line over SUM(hours * bill_rate_usd) from approved billable fct_time_entry rows for the same engagement and period. A warehouse release shipped that week. Determine what in the definition can produce a level shift with no operational change, and deliver a back-restated series plus the definition you would keep.

Approach

Treat the shape as evidence: a definition change is a discontinuity at one processing boundary, while a behaviour change ramps over weeks. Plot daily, not monthly, to see which one this is before forming any hypothesis.
Diff the metric's lineage across the release and enumerate the candidate predicates: which line_type values enter the numerator (dropping credit_note and write_off inflates it), whether the period join moved from period_start_date to issued_at, and whether the denominator counts status = 'approved' only or now includes 'submitted'.
Run both definitions over the same fixed pre-release month. If old and new differ by approximately the size of the step, the step is definitional; if they agree, look for a behaviour change coinciding with the release date.
Sanity-check the new level against the metric's physics: values above 1.0 are legitimate only on fixed_fee work delivered under its hour budget, so a practice dominated by time_and_materials sitting at 1.02 is itself a red flag.
Restate the full history under the chosen definition, publish old and new series together with the changeover marked, and add a reconciliation test that recomputes one known engagement-month from raw rows.

Follow-up

Which definition is correct, and what does excluding write-offs from the numerator do to the guardrail relationship with utilisation?
How do you prevent the next release from silently restating a headline metric?
If the release also changed the denominator's status filter, which direction does that push realisation and by roughly how much?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

The project you argued against

Saying no well is a senior skill and it is rarely rehearsed. Think of a time you told someone their analysis was not worth doing, or that the experiment could not answer their question at the sample size available. Explain what you offered instead. Refusal without an alternative reads as obstruction rather than judgement.

Describe a project where the requirements changed drastically halfway through. How did you adapt?

Medium · behavioural and stakeholder questions

Approach

Quantify the outcome, including what you would not claim credit for.
Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Defending your own impact claim without randomisation or clean units

Hard · self-selection · clustered inference · impact measurement

At your review you plan to claim that the realisation dashboard you built recovered 1.4 million dollars. The evidence is that engagement-month realisation rose four points over two quarters among engagements whose leads used it. Adoption was voluntary. There are about sixty client accounts and the top five carry most fees. A new rate card shipped in the same quarter. Write the claim you can defend, the estimate you would actually produce, and what you say when asked for a causal number you cannot get.

Approach

Name the probe: whether you can separate the number you want from the number the data supports, under review pressure, without either inflating it or retreating to saying nothing can be known.
State both identification problems concretely. Voluntary adoption means adopting leads are plausibly the ones who already manage realisation, so the comparison is confounded at the person level. The rate card changes bill_rate_usd, which sits in the realisation denominator, so part of the four-point move is arithmetic rather than behavioural.
Neutralise what you can. Recompute realisation with bill rates snapshotted on work_date, or hold the denominator at the old rate card, so the rate-card change cannot move the metric by construction. Then rerun the comparison.
Get the inference right for the unit count. Cluster at client_id, not engagement, because engagements in one account share a partner, a rate card and a team. With sixty accounts and five carrying most fees, the effective cluster count is far below sixty, so report a wild cluster bootstrap interval rather than plain cluster-robust standard errors, which are biased downward in that regime.
Report both weightings and explain the divergence: an account-weighted estimate describes the typical account, a value-weighted one describes the revenue, and if they disagree a small number of accounts is carrying the result. Then give the decision-relevant sentence: the defensible range, whether its lower bound still clears the build cost, and what a proper staggered rollout would have bought.

Follow-up

The pre-period trends for adopters and non-adopters are not parallel. What do you report then?
You get to design the next rollout. What do you change so the same question is answerable, without randomising individual accounts?

Defending a fixed-fee overrun finding against the selling partner

Easy · metric definition · pricing model mix · stakeholder pushback

Over the trailing four quarters, fixed_fee engagements in one practice area show a median scope overrun ratio of 1.34, computed as approved client_delivery hours in fct_time_entry divided by fct_engagement.contracted_hours, and inception-to-date gross margin nine points below the time_and_materials book. The partner who sold most of that work says the denominator is wrong because change orders were signed, and each change order is a separate row linked by prior_engagement_id. You have ten minutes with the practice lead. Present and defend the finding.

Approach

Say out loud what is being probed: whether a specific methodological objection makes you recompute or makes you repeat yourself. Restate the metric as numerator, denominator and window before defending anything.
Test the objection empirically instead of debating it. Rebuild the denominator by walking the prior_engagement_id chain with a recursive CTE, summing contracted_hours across the original SOW and every extension, and summing approved delivery hours over the same chain.
Report both numbers and say what each one answers: the original-SOW denominator answers whether the team held the scope that was signed, the chained denominator answers whether the firm estimated the total work correctly. Neither is a trick; they are different questions.
Keep the margin claim separate from the overrun claim, and keep it stratified. Fixed_fee margin falls with hours worked while uncapped time_and_materials margin does not, so the nine-point gap is only meaningful within pricing_model, and a shift in the fixed_fee share of fees is reported alongside it.
Close with the decision and a falsifier: if the chained overrun is near 1.0, the problem is scope control on the original SOW, and the check is whether change orders repriced at the standard rate card or at the original blended rate.

Follow-up

The chained denominator narrows the overrun to 1.11. Does your recommendation change, and which number goes in the practice review?
How would you distinguish an estimation problem at sale time from a delivery problem during execution, using only these tables?