Interview guides · Cotality

Cotality Data Scientist Interview Questions & Guide 2026

Take-home style practice for data scientist candidates: a messy dataset, a fixed time budget, and an analysis you write up and defend.

The 60-second brief

At Cotality, the Data Scientist role is a high-impact position designed to bridge the gap between raw data and strategic business decisions. You are not just a builder of models; you are a storyteller who translates complex datasets into actionable insights that drive our products forward. This role is central to our mission of optimizing internal processes and enhancing user experiences through evidence-based innovation.

The boundary with engineering varies enough to be worth asking about directly. Some seats end at the analysis and the writeup; others carry feature pipelines, scheduling and alerting, which decides whether production data plumbing belongs in your prep at all.

Focus areas

Decompose a metric move by segment and mix
Separate novelty effects from durable behaviour change
Size an experiment before anyone launches it

Overview

As a Data Scientist, you will work at the intersection of technology and business strategy. Your contributions will directly influence how Cotality scales its operations, manages its resources, and identifies new market opportunities. Whether you are refining predictive models or conducting deep-dive analyses on user behavior, your work ensures that our growth is powered by data-driven intelligence rather than intuition alone.

The environment at Cotality is one of curiosity and collaboration. You will find yourself working with diverse teams—ranging from Engineering to Product Management—to solve problems that are often ambiguous and multifaceted. This role is ideal for those who thrive on ownership and are eager to see their analytical work manifest as tangible improvements in a fast-paced corporate ecosystem.

The Cotality interview process

Cotality candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Call

Most candidates lose this call inside the first two minutes, during the walkthrough of their own background. The account runs chronologically, sits at the level of tools and titles, and never arrives at a decision anyone could have disagreed with. Anchor on a problem instead of a timeline: what the team could not answer, what you did about it, what happened next. Ninety seconds is enough, and stopping on time leaves room for the half of the call that belongs to you. What you ask about how work gets prioritised signals your level more reliably than the walkthrough does.

What to demonstrate

Whether your background summary has a shape (problem, decision, consequence) or is a chronological list of tools and employers
Whether you can account for gaps, short stints and the reason you are looking, unprompted and without hedging
The substance of the questions you ask back, which an experienced screener reads as a level signal

How to prepare

Time your opening walkthrough against a clock. If it runs past two minutes, compress the earliest role into a single clause and spend the recovered time on the most recent one
Write one honest sentence for every gap or short stint visible on your resume and offer it before being asked about it
Prepare questions about how work arrives and gets prioritised: who writes the request, how often priorities change, and what happens to an analysis after it is delivered

Hiring Manager Interview

Underneath the questions about your past work sits a resourcing question. Given four things worth doing and one of you, which gets done and what happens to the rest? Managers ask because that is the daily texture of the job, and because the answer shows whether you rank work by effort or by what it changes. The weak version sorts by personal interest or by whoever asked most insistently. The strong version ties each candidate piece of work to a decision somebody downstream is waiting on, and then names the one you would drop and who you would tell.

What to demonstrate

Whether you rank work by the decision it unblocks or by how interesting the method is
How you describe a request you declined, and whether you can say who you said it to
Whether your sense of how long something takes survives one follow-up question about the messy part
How you decide something is good enough to hand over unfinished

How to prepare

Write out your current queue and, next to each item, the decision that stays stalled until it lands. Anything with no waiting decision becomes your example of work you would cut
Rehearse turning down a plausible stakeholder request out loud, including the smaller alternative you offered instead
Have one case where you shipped a rough answer early and one where you refused to, with the reason that separated them

Panel Interview

Where a loop includes a partner from outside the data team, that conversation usually carries the same weight as the technical ones and gets the least preparation. The person opposite you will not follow a derivation and does not need to. They are working out whether having you involved would make their decisions better or slower. The failure mode is not being too technical. It is answering a question about a decision with a description of your method, leaving the translation to them. What they carry into the debrief is the sentence you handed them, not the analysis underneath it.

What to demonstrate

Whether a statistical result arrives as something the partner could act on, with the one caveat that would change their decision kept and the rest left out
Whether you can state what you need from their side, in their terms: instrumentation that does not exist yet, a definition they own, or a holdout they have to agree to
Whether uncertainty is given as a range someone can plan against, rather than as hedging that invites them to ignore the result
Whether you ask what decision is actually on the table before explaining anything

How to prepare

Take a result you know well and write the version for someone who stops reading after one sentence, then the three-minute version, and check the short one is not the long one with the qualifications stripped out
For a past project, list everything you asked a non-technical partner for and how you phrased it, then rewrite each ask so it names what goes unmeasured without it
Practise saying where a result does not apply, out loud, in one sentence that a partner could repeat accurately to someone else

Where candidates lose points

Reading a pooled rate that moved because the mix moved, not because any behaviour changed

A pooled conversion rate is a weighted average, and a shift in the weights can move it in the opposite direction to every one of its parts. A paid campaign that brings low-converting traffic drops overall signup conversion even if desktop, mobile web and app conversion each rose that week, which is Simpson's paradox and it is the single most common cause of an inexplicable dashboard move. The discipline is to decompose before explaining: recompute the rate holding last period's segment weights fixed, and compare that counterfactual to the actual, so the mix effect and the rate effect are separated numerically rather than argued about. Segment on the dimensions that actually reweight, which in this domain are almost always device_type, referrer_channel, country and new versus returning.

Counting on an identity key that changes underneath the metric

visitor_id is per browser and per device, and it resets on cookie clearance, private browsing and platform privacy changes, so the distinct-visitor count drifts upward for reasons unrelated to reach. Any rate with visitors in the denominator therefore decays over time even when behaviour is constant, and any rate with visitors in the numerator inflates. The stitching at signup makes it worse in both directions: a user who signed up on mobile and returns on desktop is two visitors and one user, while a shared device is one visitor and several users. Decide which key each metric is counted on, write it into the definition, and when comparing a period before and after a platform privacy change, expect a level shift in every visitor-keyed metric and do not attribute it to the product.

Answering a product-sense question with a list of features

Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.

Reading a dozen metrics with no multiplicity control

Nominate one primary metric before launch and treat the rest as guardrails or exploratory, with Bonferroni or Benjamini-Hochberg applied when you intend to make claims from them. Twenty independent tests at 0.05 under the null produce at least one false positive about 64 percent of the time.

Questions & practice

How do you handle imbalanced datasets when building a classification model?

Medium · machine learning and modelling

Approach

Set a baseline first, so any model has something honest to beat.
Say how the offline result would be validated online before it is trusted.
Pick an evaluation metric that matches the cost of each error type, not a default.

Follow-up

How would you choose the decision threshold, and who owns that choice?
What would you monitor after launch to know the model is still valid?

What are the assumptions of linear regression, and what happens if they are violated?

Medium · machine learning and modelling

Approach

Pick an evaluation metric that matches the cost of each error type, not a default.
Set a baseline first, so any model has something honest to beat.
Frame the prediction: the label, the moment of prediction, and the action it triggers.

Follow-up

Where could label leakage enter this setup?
What would you monitor after launch to know the model is still valid?

Split a pooled conversion drop into rate and mix

Medium · mix shift · decomposition · pandas

You have weekly visit-to-signup counts by segment: a DataFrame with week, device_type, referrer_channel, visitors and signups. The pooled rate fell 0.84 percentage points between two consecutive weeks while several individual segments rose. Write a function that, for a caller-supplied list of segment columns, splits the pooled change into a rate effect, a mix effect and an interaction term that sum exactly to the observed change. Return those three scalars plus a per-segment contribution table sorted by absolute contribution, so the largest single driver can be named.

Approach

State the algebra before coding: the pooled rate is r = sum over segments of w_s * r_s, with w_s the segment's share of the denominator. Then r1 - r0 decomposes exactly into sum(w_s0 * (r_s1 - r_s0)) for rate, sum((w_s1 - w_s0) * r_s0) for mix, and sum((w_s1 - w_s0) * (r_s1 - r_s0)) for interaction. The identity is per-segment, so it holds for any numbers you put in the four slots.
Pivot both weeks onto a common segment index with an outer join so a segment that appeared or vanished is kept rather than dropped, then decide what rate to give a segment with no visitors in one of the weeks, and document the choice. The identity stays exact either way because the missing week's weight is 0, but the attribution does not. Filling the missing rate with 0 sends an appearing segment's entire w_s1 * r_s1 into the interaction term, since w_s0 = 0 makes both the rate term and the mix term (w_s1 - w_s0) * r_s0 identically zero; a vanishing segment then splits as -w_s0 * r_s0 in rate, -w_s0 * r_s0 in mix and +w_s0 * r_s0 in interaction.
The convention used below instead imputes the missing week's rate as that week's pooled rate. A vanishing segment then lands wholly in mix at -w_s0 * r_s0, with rate and interaction cancelling; an appearing segment puts w_s1 * r_pooled0 in mix (volume arriving at the average rate) and only w_s1 * (r_s1 - r_pooled0) in interaction (its rate differing from that average). Impute by which week the segment is missing from, never by argument order, or the swap identities below stop holding.
Guard the division where visitors is 0 so no NaN enters the vectors, because a single NaN poisons every sum. A segment with zero visitors in both weeks contributes exactly 0 and can be dropped; a segment missing from only one week does not contribute 0, and where its contribution lands is settled by the convention above, not by the guard.
Compute the three components as vectors over segments, then sum. Keep the vectors, because the per-segment contribution table is what turns the decomposition into an explanation.
Assert that the three components sum to the observed pooled change within floating-point tolerance. This identity is exact, so a mismatch means an implementation bug, not a modelling judgement.

Worked solution (25 min)

Aggregate to one row per (week, segment tuple) with summed visitors and signups, then split into w0 and w1 frames and align with an outer join, filling missing visitors and signups with 0.
Compute w_s = visitors / visitors.sum() within each week, and r_s = signups / visitors only where visitors > 0. Where a week's visitors are 0, set that week's r_s to that week's pooled rate, the stated convention; never leave it NaN.
rate_effect = (w0 * (r1 - r0)).sum(); mix_effect = ((w1 - w0) * r0).sum(); interaction = ((w1 - w0) * (r1 - r0)).sum().
contribution = w0*(r1-r0) + (w1-w0)r0 + (w1-w0)(r1-r0) per segment, which reduces to w1r1 - w0r0; sort by abs and return the head.
assert abs(rate + mix + interaction - (pooled1 - pooled0)) < 1e-12.

Expected result: Three scalars summing exactly to the observed pooled change, plus a segment table whose contribution column also sums to that change. In the stated case rate and mix carry opposite signs: several segments converted better while volume moved toward a weaker-converting one, which is what makes the pooled move look inexplicable until it is split.

Verify it yourself

On synthetic data where every segment appears in both weeks, every segment rate is held constant and only the weights move, rate_effect and interaction are exactly 0.
On synthetic data where every segment appears in both weeks, weights are held constant and only rates move, mix_effect and interaction are exactly 0.
The per-segment contribution column sums to the pooled change, matching the sum of the three scalars.
Swapping the two weeks negates the total, as it must, but not the parts. Assert the three identities that actually hold: the interaction term is exactly invariant, because both of its factors change sign; the swapped rate effect equals -(rate_effect + interaction); and the swapped mix effect equals -(mix_effect + interaction). They still sum to the negated pooled change. Expecting all three to flip sign is the common wrong assertion.
Plant a segment present only in week 1, holding 3 percent of week-1 visitors at a rate well below pooled. Under the documented pooled-rate convention its mix contribution is w_s1 * r_pooled0 and its interaction contribution is w_s1 * (r_s1 - r_pooled0), with a rate contribution of exactly 0; under a zero-fill convention its mix contribution is exactly 0 and the whole w_s1 * r_s1 lands in interaction. Assert whichever one the function documents, and fail if mix is asserted non-zero while the code zero-fills.

Follow-up

The mix effect accounts for 0.71 of the 0.84 point drop, driven by paid_social volume. What is your recommendation, and what would change it?
Why is a two-way split into a counterfactual rate and a residual also exact, and when would you prefer it to the three-way version?
Segmenting on device and channel leaves a large interaction term. What does that tell you about the choice of segments?

Read an experiment from first exposure, not assignment

Hard · experiments · window-functions · srm

fct_experiment_exposure holds experiment_id, unit_type, unit_id, variant, user_id, assigned_at_utc, first_exposed_at_utc, is_in_analysis_population and planned_end_utc. fct_event holds user_id, occurred_at_utc, is_core_action, and carries events up to a known data cut, :data_cut_utc. For one experiment randomised on unit_type = 'user', return per variant: exposed units, units with at least one core action in the seven days after that unit's own first exposure, the rate, and the variant share of exposed units. Only units whose seven-day window has fully elapsed as of the data cut belong in the readout. Units appearing under more than one variant are excluded from both arms and counted separately.

Approach

Run the contamination pass as an aggregate, not a window: SELECT unit_id FROM fct_experiment_exposure WHERE experiment_id = :exp GROUP BY unit_id HAVING COUNT(DISTINCT variant) > 1, then anti-join it away. PostgreSQL rejects COUNT(DISTINCT variant) OVER (PARTITION BY unit_id) outright, since DISTINCT is not implemented for window functions; if you want the test inline, MIN(variant) OVER (PARTITION BY unit_id) <> MAX(variant) OVER (PARTITION BY unit_id) is the equivalent that does run.
Do not resolve contamination by keeping the earliest variant. A unit that saw both arms carries treatment from both, so assigning it to either one biases that arm.
Define the population as is_in_analysis_population = TRUE AND unit_type = 'user' AND first_exposed_at_utc < planned_end_utc AND first_exposed_at_utc + interval '7 days' <= :data_cut_utc. The horizon filter is what makes the readout reproducible next week instead of drifting with every re-run; the data-cut filter is the one that actually buys seven days of follow-up, since a unit exposed an hour before the horizon otherwise contributes an hour of observation to a seven-day rate.
Measure the outcome on a per-unit relative window: LEFT JOIN fct_event on user_id with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc, first_exposed_at_utc + interval '7 days'). LEFT JOIN so units with no outcome stay in the denominator at zero rather than being deleted by an inner join.
Check the sample ratio before reading the effect: variant share of exposed units against the intended split, tested as a binomial. Run it on the truncated population as well as on the full exposed set, because if one arm exposes later on average the data-cut filter removes more of that arm and can manufacture a ratio mismatch the randomisation did not have. A mismatch on the full set means the exposure data is not a valid randomisation and invalidates the readout rather than being a footnote under it.
Report the per-variant rate, the absolute difference, and the fact that the variance unit is unit_id. That is straightforward here only because the grain is already one row per user; a per-session outcome under user randomisation would need a delta-method or bootstrap standard error instead.

Worked solution (35 min)

Count units per variant and units with more than one variant; report the contaminated count before anything else.
Build the exposed population CTE with both the horizon filter and the seven-days-before-data-cut filter, assert one row per unit_id, and record per variant how many units the second filter removed.
LEFT JOIN the bounded outcome events and flag conversion with a boolean aggregate.
Aggregate per variant to exposed, converted, rate and share, and run the binomial check on the share, both before and after the truncation.
Re-run with a fixed calendar window from experiment_start_utc and record the difference as the size of the dilution you avoided.

Expected result: One row per variant with exposed, converted, rate and share, plus a scalar count of excluded contaminated units. Every retained unit was first exposed at least seven days before the data cut, so each one has a full seven days of observation and the rates are directly comparable.

Verify it yourself

Exposed counts across variants sum to the population count minus the contaminated units.
MAX(first_exposed_at_utc) among retained units is at least seven days before :data_cut_utc.
Variant shares sum to 1.
No unit_id contributes to two variant rows.
The truncation drops a similar share of each arm; if it does not, note the imbalance alongside the SRM result rather than reading the effect.
Swapping the relative window for a calendar one changes the rates; if it does not, exposure timing is balanced and you can say so.

Follow-up

Some units were assigned days before they were exposed. What does analysing the assigned set instead do to the estimated effect, and in which direction?
The split is 51/49 on 400,000 exposed units. Do you read the result?
The treatment arm exposes on average two days later than control. What does that do to a fixed calendar outcome window, and which arm does it favour?

Find reactivation gaps in account paid-period history

Hard · gaps-and-islands · window-functions · subscriptions

fct_subscription_period holds account_id, subscription_id, period_start_utc, period_end_utc, period_status and change_reason. A mid-period plan or seat change closes one row and opens another, so a single continuous paid tenure is often many rows, and an account may hold two overlapping subscriptions. Collapse rows with period_status in ('active','past_due') into continuous tenures per account, treating gaps of three days or less as continuous. Return account_id, tenure_start, tenure_end, and for every tenure after the first, the gap in days that preceded it.

Approach

Filter to paid rows only: period_status IN ('active','past_due'). Trialing periods are not tenure, and including them turns every trial that never converted into a one-period tenure followed by a fake churn.
Order by period_start_utc and take a running maximum of all prior ends: MAX(period_end_utc) OVER (PARTITION BY account_id ORDER BY period_start_utc, period_end_utc, subscription_id ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). Those three columns are the only stable ordering this schema exposes, so check first that they are unique within an account; if rows tie on all three, the island numbering is order-dependent between runs and you need a real row key before the result is reproducible.
LAG on its own is wrong here because with overlapping or nested periods the immediately preceding row by start date is not the one that ends latest, so the running maximum is the part that cannot be shortcut.
Flag a new island when prior_max_end IS NULL OR period_start_utc > prior_max_end + interval '3 days', then number islands with a running SUM of the flag over the same ordering and an explicit ROWS frame.
Group to (account_id, island) taking MIN(period_start_utc) and MAX(period_end_utc), then LAG(tenure_end) OVER (PARTITION BY account_id ORDER BY tenure_start) to compute the preceding gap in days for every tenure after the first.
Sanity-check with change_reason, which is the only lineage this schema carries: list its distinct values first, then confirm that rows recording a plan or seat change sit inside a tenure rather than opening one, and that every tenure after the first opens on a row whose reason records a restart rather than an ordinary renewal. Do not reconcile against a churn timestamp on dim_account, which this schema does not define; and where such a column does exist, a cancellation timestamp records when the request was made and routinely sits weeks before the period it ends.

Follow-up

Why three days of grace? What do 0 and 30 days each do to the count of accounts classed as reactivated?
An account runs two concurrent subscriptions for different teams. One tenure or two, and what does the revenue reader expect?
How would you turn these tenures into a monthly gross logo churn series without double-counting an account that churned and returned in the same month?

How do you prioritize your tasks when working on multiple high-priority projects?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

What metrics would you use to evaluate the performance of a recommendation system?

Medium · product sense and metrics

Approach

Name one primary metric, then the guardrail that stops it being gamed.
Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How would you design an A/B test for a new feature on our platform?

Medium · a/b testing and experimentation

Approach

State the primary metric and the minimum effect worth shipping, then size the test.
Decide the analysis before seeing data, including how long it runs and when you look.
Name the guardrails that would stop a launch even on a positive primary result.

Follow-up

What would you conclude if the result is positive but the test is underpowered?
What would you do if you could not randomise at all?

Explain the concept of the "Curse of Dimensionality" and how it affects data analysis.

Medium

Approach

Work from the decision backwards to the evidence you would need.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Halve the runtime of a flat test with pre-period data

Medium · cuped · variance reduction · stratification

A test on 28-day core actions per user is powered for six weeks and the team will not wait. You have fct_event (user_id, occurred_at_utc, is_core_action) covering the period before each unit's first_exposed_at_utc in fct_experiment_exposure, and the pre-period count correlates with the in-experiment count at r = 0.6. Apply CUPED, state precisely how much runtime it buys, and handle the units with no pre-period because they signed up after the experiment started.

Approach

Define the adjusted outcome: Y_cuped = Y - theta (X - Xbar), with theta = Cov(Y, X) / Var(X) estimated on the pooled sample so theta itself carries no treatment effect. It is unbiased because X is pre-exposure and therefore has equal expectation in both arms.
Quantify the gain exactly: Var(Y_cuped) = Var(Y)(1 - r^2) = Var(Y) x 0.64, so 36% of the variance is removed. Required sample scales with variance, so six weeks becomes 3.8 weeks, and at the original sample the MDE shrinks by sqrt(0.64) = 0.8.
State the precondition that actually bites: X must be measured strictly before first_exposed_at_utc, not before assigned_at_utc and not before the calendar start of the experiment. Any post-exposure information in the covariate lets the treatment effect leak in and biases the estimate rather than merely failing to help.
Handle missing pre-periods as a stratum, never by dropping rows. Dropping changes the population; setting X to the pooled mean leaves those units' adjustment at zero, which is unbiased but buys them no reduction, so report the covered share and expect the blended reduction to fall short of 36%.
Note that stratification is the same mechanism with a categorical covariate. Post-stratifying on device_type and referrer_channel removes the variance those stratum means explain, and it composes with CUPED provided the strata are also fixed before exposure.

Worked solution (30 min)

Build the covariate: for each unit_id in fct_experiment_exposure, count fct_event rows with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc - 28 days, first_exposed_at_utc).
Estimate theta on the pooled sample as Cov(Y, X) / Var(X), and confirm Xbar is statistically indistinguishable between arms before using it.
Form Y_cuped = Y - theta (X - Xbar) and run exactly the test you would have run on Y.
For units with no pre-period, set X to the pooled mean so their adjustment is zero, and report what share of the sample that is.
Report unadjusted and adjusted estimates side by side, with the interval narrower on the adjusted one.

Expected result: 1 - 0.6^2 = 0.36 of the outcome variance is removed, so the required sample falls to 64% of the original and the six-week plan becomes about 3.8 weeks. At the original sample the MDE shrinks by a factor of 0.8. The blended reduction is smaller in proportion to the share of units with no pre-period, and that share is stated alongside the result.

Verify it yourself

Adjusted and unadjusted point estimates should be close. CUPED reduces variance, it does not move the effect, so a large shift means X was imbalanced at baseline and needs investigating rather than celebrating.
The realised standard error ratio should be near sqrt(1 - r^2) using the r observed in this experiment, not the historical r.
Var(Y_cuped) must be no larger than Var(Y) for the fitted theta. A larger value means theta has the wrong sign or was estimated on the wrong window.

Follow-up

Here the covariate is the same metric as the outcome. When would you deliberately pick a different one, and what is the risk if that covariate is itself affected by treatment?
The point estimate moves by 0.4 standard errors when CUPED is applied. Is that reassuring or alarming, and what would you check?
How does CUPED combine with cluster randomisation on account_id, and at which grain is the covariate built?

Size every candidate cause of a trial-to-paid decline

Hard · cohort-maturity · experiment-readout · mix-shift · contribution-sizing

Trial-to-paid conversion on weekly trial-start cohorts from fct_subscription_period reads 3.1 points below the trailing eight-week mean for the three most recent cohorts. Three things happened in that window: a pricing experiment reached 50% of new trials, a payment processor migration added settlement delay, and paid_search spend tripled. Using fct_subscription_period, fct_experiment_exposure and dim_user, rank the causes by their contribution in points of the headline, state the remainder, and give the decision you would take on Monday.

Approach

Kill the immature cohorts first, because everything downstream is computed on them. The metric is lagged by the trial length plus a 14-day conversion window plus a settlement allowance, and a processor migration lengthens exactly that last term; recompute each cohort at a fixed cohort age rather than as of today, and confirm the newest cohort's value is still climbing day over day.
Hold the experiment analysis to the exposed population. Join fct_experiment_exposure on unit_id with is_in_analysis_population = TRUE rather than reading an assignment log, then check the variant split for a sample-ratio mismatch before believing any effect at all. Contribution to the headline is the variant effect multiplied by the exposed share, which is not the same number as the variant effect.
Decompose the cohort mix by dim_user.first_touch_channel using the same weight-times-rate arithmetic as any other mix question, so the paid_search increase is sized as a weight change at a measured conversion rate rather than asserted from the spend figure.
Convert all three to points of the headline, sum them, and print the residual against the historical week-to-week standard deviation of the metric. If the residual is inside that band, say so and stop looking; if it is outside, name what you would investigate next rather than leaving it implied.
Land the decision. Only one of the three is actionable on Monday, so state whether the experiment has accrued enough exposed units to stop at the pre-declared horizon, and state separately what the settlement-lag correction does to the published series and its lag rule.

Follow-up

How do you choose the fixed cohort age, and what do you lose by choosing it too long?
If the pricing variant is genuinely 1.2 points worse, does that settle whether to stop it? What else is on the other side of that decision?
The trailing eight-week mean spans the processor migration. What is the right baseline instead?

Suggested practice checklist

For someone who has spent the last year in notebooks, dashboards or modelling work and has not written raw SQL under time pressure. The first four days rebuild query fluency against a fixture you control and can verify by hand; the last three attach that fluency to the rest of the loop.

Day 1 — Build a fixture you can check answers against

Create a local Postgres or SQLite database with four tables (users, sessions, events, orders) holding roughly 200 rows you generated yourself, so you know the contents well enough to predict every result.
Deliberately seed the cases that break queries: a user with no sessions, a session with no events, two orders sharing a timestamp, a NULL in one join key, and one duplicated user row.
Before writing any SQL, hand-compute five answers on paper (how many users placed at least one order, median orders per ordering user, and three others) and save them as the ground truth for the week.

Deliverable: A one-command seed script plus a text file of five hand-computed answers to grade every later query against.

Day 2 — Joins, filters and NULL semantics

Answer "which users have no orders" three ways (LEFT JOIN with IS NULL, NOT EXISTS, NOT IN) and confirm that the NOT IN version returns zero rows once the subquery contains a NULL, because the comparison is never TRUE.
Reproduce the LEFT JOIN that silently collapses to an inner join by putting a right-table predicate in WHERE, then fix it by moving the predicate into the ON clause, and record both row counts.
Create a fan-out bug on purpose by joining orders to order_items and summing the order total, then correct it with a pre-aggregated subquery and explain in one line which table changed the grain.

Deliverable: One annotated .sql file holding the three join traps, each with the wrong result and the corrected result side by side.

Day 3 — Window functions and frames

Write three window queries against the fixture: a running order total per user, the rank of each order within its user by value, and the day gap to that user's previous order, then check each against the day-one ground truth.
Run ROW_NUMBER, RANK and DENSE_RANK over a column containing ties, print all three side by side, and write one sentence on when each is the correct choice.
Switch one query from the default frame (RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which is what you get when ORDER BY is present and no frame is written) to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, and explain why the output differs only when the ORDER BY column has duplicates.

Deliverable: Three verified window queries plus a short note explaining the RANGE versus ROWS difference in your own words.

Day 4 — The four analytical query patterns

Write a monthly retention grid: first order month per user, then months-since-first as the column, and verify that month zero equals the cohort size exactly.
Sessionize the events table under a 30-minute inactivity rule using LAG plus a cumulative sum over a new-session flag.
Build a four-step funnel that counts distinct users rather than events at each step, and state the rule you applied to a user who reaches step three without ever logging step two.

Deliverable: One file with the retention, sessionization and funnel patterns, each carrying a one-line note on the assumption it bakes in.

Day 5 — Write SQL the way you will have to write it live

Set a 12-minute timer and solve three medium prompts in a plain editor with no execution and no autocomplete, then run them and tally syntax errors separately from logic errors.
Narrate one solution aloud while writing it, stating the grain of each intermediate result (one row per user, one row per user-day) before you type its body.
Rewrite your slowest solution as a CTE chain where every CTE name states its grain, and time yourself re-solving it from blank.

Deliverable: A recording of one narrated solution plus an error tally that separates syntax from logic.

Day 6 — One day for everything that is not SQL

Write the preconditions of the two-sample t-test from memory, then check them: independent observations, and a difference in means whose sampling distribution is approximately normal, which at large sample sizes follows from the central limit theorem rather than from normality of the raw values.
Write the difference between an odds ratio from logistic regression and a relative risk, and state the condition under which the two are close (low outcome prevalence).
Prepare a 90-second answer to "how would you know this model is any good" that names the metric, the baseline you would beat, and the cost of the errors you care about.

Deliverable: One page of notes covering test preconditions, the odds-ratio caveat and the model-quality answer.

Day 7 — Full loop rehearsal

Run a 45-minute mock with someone willing to interrupt: 20 minutes of SQL, 15 minutes defining a metric, 10 minutes on a past project.
Re-solve from blank the two queries you were slowest on this week and compare the times against day five.
Write a five-line answer to "walk me through a project" that puts a number in the first sentence and names the decision the work changed.

Deliverable: Mock feedback notes plus a timed project narrative you can deliver without reading it.

Stories where someone disagreed with you

Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.

Describe a project where you faced a significant data-related challenge. How did you overcome it?

Medium · behavioural and stakeholder questions

Approach

Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.
State the situation in two sentences and spend the rest on your reasoning.

Follow-up

How did you know the outcome was caused by your change?
What would you do differently if you ran that project again?

Tell me about a time you had to explain a complex technical concept to a non-technical stakeholder.

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Pick a story where you drove the decision, not one where you observed it.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know the outcome was caused by your change?

Disagree with a product manager's roadmap claim using data

Medium · causal inference · selection bias · stakeholder

A product manager proposes building a feature on the argument that accounts connecting an integration in week one retain three times better at week four. The figure is correctly computed from dim_user and fct_event, and it has already been shown to leadership. You have one scheduled 1:1 before the roadmap locks. Deliver the specific analysis you would run to test whether the relationship is causal, the result that would change your own mind, and how you open the conversation so that the PM is not put in the position of defending the number in public.

Approach

Recognise what is being probed: whether you can separate a number being right from an inference being wrong, and do it without costing the PM face. The generic answer recites that correlation is not causation; the strong one names the specific confound and proposes the cheapest design that could distinguish the explanations.
State the alternative concretely. Accounts that connect an integration in week one are accounts that already have a workflow and a technical owner, so week-one intent plausibly drives both the connection and week-four retention. The selection is on intent, which no amount of post-hoc adjustment observes.
Order the discriminating analyses by cost. First, condition on pre-connection activity by comparing retention within strata of week-one core-action count, which removes the crude version of the confound but not unobserved intent. Second, look for variation in integration availability that was unrelated to intent, such as a staggered release or an outage window. Third, an encouragement design that randomises a prompt to connect and reads the intent-to-treat effect on week-four retention, which is the only version that identifies an effect.
Run the timing check, because it is nearly free and it is the most persuasive single piece of evidence. If the retention advantage among connectors is already visible before any of them connected, the causal story is largely finished.
Pre-commit to what would change your mind and say it before you show anything: if the gap survives stratification and the encouragement arm moves week-four retention at all, the feature has a case and you will say so.
Open the 1:1 by agreeing with the true part, that the correlation is real and worth chasing, then ask what effect size the roadmap plan assumes. That makes the size of the claim the topic instead of its authorship.

Follow-up

The encouragement test needs six weeks and the roadmap locks in two. What do you recommend in the interim?
Stratifying on week-one activity closes half the gap. What do you conclude, and what do you still not know?
How would you word this in the roadmap document so the PM's original number is reframed rather than deleted?