Cybage Software Data Scientist Interview Questions & Guide 2026

The 60-second brief

A Data Scientist at Cybage Software serves as a vital bridge between complex raw data and actionable business intelligence. In this role, you are responsible for designing, developing, and deploying analytical models that solve real-world problems for a diverse portfolio of clients. Your work directly impacts how products function, how user engagement is measured, and how stakeholders make data-driven decisions.

Most of the loop measures decision-making under uncertainty rather than recall. You are scored on whether you state your assumptions, commit to an estimate you can defend, and say explicitly what evidence would change it.

Focus areas

• Measure churn only on renewal-eligible accounts
• Separate contracted seats from actively used seats
• Read NRR on a fixed account cohort

Overview

The environment at Cybage Software is dynamic and requires a blend of technical rigor and product intuition. You will often find yourself operating at the intersection of statistical modeling and software engineering, requiring you to not only build algorithms but to ensure they are scalable and reliable within production systems. Success in this role demands a high degree of adaptability, as you will likely work across various domains, ranging from predictive analytics to optimization and experimentation.

The Cybage Software interview process

Cybage Software candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

• Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
• Whether your reason for wanting the role points at the work itself rather than the company's reputation
• Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

• Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
• Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
• Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Interview

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

• Whether your row counts survive each join, and whether you notice on your own when they do not
• Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
• Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
• Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

• Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
• Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
• Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Fit Interview

Most of the weight in this round sits on the disagreement questions. Data work routinely produces an answer someone senior did not want, and the interviewer is trying to learn what you do in that hour. Both failure modes are common: folding as soon as a director pushes back, and treating the pushback as ignorance to be corrected with a better chart. A strong answer usually contains a specific thing the other person knew that you did not, and describes how you found out whether it changed the conclusion.

What to demonstrate

• Whether you can state the other side's argument accurately before you explain why you disagreed
• What you treated as evidence during the disagreement, such as a rerun under their assumption or a holdout check, rather than persuasion technique
• Whether you distinguish being overruled from being wrong, and can give an example of each

How to prepare

• Write out one disagreement where you turned out to be wrong, and say what in the data misled you. Candidates prepare the story where they were right, and the follow-up asks for the other one.
• For your main disagreement story, be ready to say what result would have made you drop your position. If no such result exists, you were not arguing from the data.
• Practise stating the opposing position out loud in one sentence the stakeholder would accept, then continue the story.

Where candidates lose points

Computing monthly churn against the entire customer base when contracts are annual

An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Accepting a metric definition without asking about the denominator

Pin down the denominator, the eligibility filter and the time window before computing anything: conversion rate per session, per user, per eligible user and per new user are four different numbers with different behaviour. Restate the definition in one sentence and get agreement before you analyse.

Dropping rows with missing values without naming the mechanism

Say whether the values are missing at random, missing by a known process, or missing in a way that depends on the outcome, and handle them accordingly. Deleting incomplete rows silently redefines the population whenever missingness correlates with what you are measuring.

Questions & practice

What is the difference between ANOVA 1-way and ANOVA 2-way?

Medium · statistics and probability

Approach

1. Translate the result into the decision it informs, in one plain sentence.
2. Write down the assumption the method needs before you use the method.
3. Sanity-check the answer against a simple bound or a simulated case.

Follow-up

• How would you explain this result to someone who does not know statistics?
• Which assumption here is most likely to be violated in practice?

How would you handle missing values or outliers when preparing a dataset for model training?

Medium · machine learning and modelling

Approach

1. Check what information would not exist at prediction time, and exclude it.
2. Pick an evaluation metric that matches the cost of each error type, not a default.
3. Set a baseline first, so any model has something honest to beat.

Follow-up

• Where could label leakage enter this setup?
• How would you choose the decision threshold, and who owns that choice?

Permutation-test a consumption experiment randomised at account level

Hard · permutation-test · heavy-tails · cuped · experiment-readout

An experiment randomised 900 accounts into two arms. You have one row per account: account_id, arm, consumption_28d (billable units after launch) and consumption_pre (the 28 days before). Consumption is heavy-tailed and the largest account is several percent of the total. Write a permutation test from scratch: winsorise at the pooled 99th percentile as a pre-registered rule, use the difference in arm means of the winsorised outcome as the statistic, and obtain a two-sided p-value from 20,000 relabellings of the account-level arm vector. Report the observed effect, the p-value, and the same test on a CUPED-adjusted outcome.

Approach

1. Be precise about what the permutation test needs. Under the sharp null of no effect for any account, the outcomes are exchangeable across arm labels, and the test is valid for ANY statistic T(outcomes, labels) provided the identical function is applied to the observed labels and to all 20,000 relabellings. The pooled 99th percentile is a function of the outcome vector alone, so recomputing it inside the loop returns the same number 20,000 times: that is wasted CPU, not a bias, and hoisting it out is an optimisation rather than a correctness fix. Say plainly that capping at all changes the estimand from mean consumption to mean capped consumption; it is not a neutral cleaning step.
2. The mistake that does invalidate the test is an asymmetry between the observed statistic and the permuted ones, and the easiest way to create it is to derive the cleaning rule from the observed arm labels and then freeze it — winsorise each arm at its own observed 99th percentile, hold those two caps fixed, and permute. The observed value is then computed with caps matched to its own partition while every relabelling is scored with caps belonging to a different one, so the null distribution no longer answers the question the p-value claims to answer. A per-arm cap recomputed consistently inside every permutation is a valid test, but it estimates a contrast whose two sides are capped at different thresholds, so prefer the pooled cap on estimand grounds and pre-register it.
3. Permute the account-level arm vector, because the account is the randomisation unit. Relabelling anything finer — users, workspaces, requests — generates a null distribution narrower than the design actually supports and returns p-values that are anti-conservative.
4. Vectorise the null: tile the treatment indicator into a (B, n) matrix and permute along axis 1 with rng.permuted(..., out=...). The statistic is a difference of means, so the treated sum alone determines it and the whole null is one matrix-vector product. Use the two-sided p-value (1 + count(|stat_perm| >= |stat_obs|)) / (B + 1); the plus-one on each side is not cosmetic, it keeps the p-value away from exactly zero and keeps the test valid at finite B.
5. For CUPED, fit theta = cov(y, x) / var(x) on the pooled data and use that same theta for the observed statistic and every relabelling. Pooled theta, like the pooled cap, carries no label information, so where in the loop you compute it is again only a performance question; fitting theta within arms is what goes wrong, because the adjusted outcome then depends on the labels and an observed-label fit frozen across all 20,000 relabellings breaks the match between observed and permuted statistics. x must be measured entirely before launch, which consumption_pre is. Expected variance reduction is about 1 - corr(y, x)^2; measure the achieved reduction from the two null distributions rather than asserting it.

Worked solution (45 min)

1. cap_y = np.quantile(df.consumption_28d, 0.99); y = np.minimum(df.consumption_28d.to_numpy(float), cap_y); cap_x = np.quantile(df.consumption_pre, 0.99); x = np.minimum(df.consumption_pre.to_numpy(float), cap_x)
2. t = (df.arm == 'treatment').to_numpy(); n1 = int(t.sum()); n0 = len(t) - n1; obs = y[t].mean() - y[~t].mean()
3. rng = np.random.default_rng(11); L = np.tile(t.astype(np.int8), (20_000, 1)); rng.permuted(L, axis=1, out=L); s1 = L @ y; stats = s1/n1 - (y.sum() - s1)/n0
4. p = (1 + int(np.sum(np.abs(stats) >= abs(obs)))) / (20_000 + 1)
5. theta = np.cov(y, x, ddof=1)[0,1] / np.var(x, ddof=1); y_adj = y - theta*(x - x.mean()); repeat steps 2 to 4 on y_adj and compare stats.std(ddof=1) between the two runs.

Expected result: A two-sided p-value strictly between 1/20001 and 1, an observed effect expressed in capped billable units per account, and a CUPED null whose standard deviation is smaller than the unadjusted one by roughly sqrt(1 - corr(y, x)^2). The CUPED point estimate stays close to the unadjusted one, since the adjustment removes variance rather than shifting the effect.

Verify it yourself

• The permutation null is centred near zero: |mean(stats)| is a small fraction of std(stats).
• An A/A run, relabelling arm at random before the test and repeating over seeds, gives roughly uniform p-values with about 5% below 0.05.
• std(stats_cuped) < std(stats) whenever corr(y, x) is materially non-zero, and the ratio matches sqrt(1 - r^2) to within simulation noise.
• Moving the pooled cap computation inside the permutation loop reproduces the same p-value bit for bit at the same seed — the direct demonstration that a pooled cap carries no label information — while setting the cap to infinity reproduces the uncapped test exactly, so the winsorisation switch is independently verifiable.

Follow-up

• The p-value is 0.04 with the cap and 0.31 without it. What do you report, and what did you pre-register?
• Colleagues in a shared workspace can see the treated behaviour. How does that change the design and the estimate?
• How many accounts would you need to detect a 5% lift given this outcome's distribution?

Explain the difference between JOIN and UNION in the context of data aggregation.

Medium · sql and data manipulation

Approach

1. Compute rates by summing numerator and denominator separately, never by averaging rates.
2. State the window function and its partition and ordering out loud before writing it.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• How does the query change if the join becomes one-to-many?
• What breaks if events arrive late or out of order?

Describe how you would extract specific time-series data using date functions.

Medium · sql and data manipulation

Approach

1. Check whether any join is one-to-many before aggregating, or the sums inflate.
2. Compute rates by summing numerator and denominator separately, never by averaging rates.
3. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

• What breaks if events arrive late or out of order?
• How does the query change if the join becomes one-to-many?

Write a query using SQL window functions to calculate a rolling average of user activity.

Medium · sql and data manipulation

Approach

1. Say which table is the grain you start from, and join outward from it.
2. Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.
3. Check whether any join is one-to-many before aggregating, or the sums inflate.

Follow-up

• How does the query change if the join becomes one-to-many?
• How would you verify this result without re-running the same query?

Monthly account margin joined as-of the live contract version

Medium · as-of join · fan-out · scd2 · unit economics

fct_usage_daily carries account_id, usage_date, net_amount_cents and cogs_cents. fct_subscription_period carries account_id, plan_tier, term_start_date, term_end_date, booked_at and is_current, with one row per contract version, so several superseded versions can bracket the same usage_date. dim_account is a type 2 dimension keyed on account_id with employee_band, is_internal, effective_from, effective_to and is_current. Return monthly net revenue, allocated COGS and gross margin per account, labelled with the plan_tier in force during that month and the employee_band in force at month end. The reported figure excludes internal accounts, so it cannot equal the raw sum of net_amount_cents; reconcile additively instead, quantifying every bucket you drop so the reported and dropped amounts add back to that raw sum.

Approach

1. Aggregate fct_usage_daily to one row per (account_id, month) before touching any dimension. Aggregating after the join is what multiplies revenue, and no amount of DISTINCT afterwards recovers the right number.
2. Resolve the contract as-of the month rather than as-of now. Take fct_subscription_period rows whose term brackets the month, then rank with ROW_NUMBER() OVER (PARTITION BY account_id, month ORDER BY booked_at DESC, subscription_period_id DESC) and keep rank 1. Filtering on is_current instead backdates today's plan over last year's usage and quietly rewrites history.
3. Resolve dim_account with the half-open interval effective_from <= month_end AND (effective_to > month_end OR effective_to IS NULL). The NULL on the live version has to be spelled out or the current row drops out of every recent month.
4. Compute margin as (sum(net_amount_cents) - sum(cogs_cents)) / NULLIF(sum(net_amount_cents), 0) at the account-month grain, and report the distribution rather than a blended rate so margin-negative accounts stay visible.
5. State the month-straddling rule explicitly: either split the month at the amendment date or take the version in force at month end. Both are defensible; an unstated choice is not.
6. Reconcile additively, not by equality to the unfiltered total. The output deliberately drops two populations: account-months whose as-of dim_account version carries is_internal = true, and account-months with no fct_subscription_period version bracketing the month, which an inner join on the contract removes without saying so. The statement that ties out is reported_net + internal_net + unmatched_net = sum(net_amount_cents) over the same usage_date range. Plain equality to the raw sum would only hold if both dropped buckets were empty, and the internal one never is.
7. Compute the internal bucket with the identical as-of rule used for the output, evaluating is_internal on the dim_account version in force at that month end. Reading is_internal from the current version instead moves accounts between the two sides of the identity and it stops closing.

Worked solution (30 min)

1. Build usage_monthly: SELECT account_id, date_trunc('month', usage_date) AS month, sum(net_amount_cents) AS net_cents, sum(cogs_cents) AS cogs_cents FROM fct_usage_daily GROUP BY 1, 2.
2. Build contract_asof by joining usage_monthly to fct_subscription_period on account_id with the term bracketing the month, then applying the ROW_NUMBER ranking on booked_at DESC and keeping rn = 1.
3. Join dim_account with the half-open effective_from/effective_to predicate evaluated at month end, and filter is_internal = false on the version selected.
4. Select account_id, month, plan_tier, employee_band, net_cents, cogs_cents and (net_cents - cogs_cents)::numeric / NULLIF(net_cents, 0) AS gross_margin.
5. Run the reconciliation as a three-way identity over the same usage_date range: sum(net_cents) in the output, plus sum(net_cents) over account-months whose as-of dim_account version has is_internal = true, plus sum(net_cents) over account-months with no bracketing subscription version, must equal SELECT sum(net_amount_cents) FROM fct_usage_daily for that range, to the cent. Publish the two dropped amounts next to the total rather than leaving them implicit; a non-zero unmatched bucket is a contract-coverage bug to chase, not rounding.

Expected result: One row per account per month with net revenue, allocated COGS, gross margin, and the plan_tier and employee_band that were actually in force, with output row count equal to the number of distinct (account_id, month) pairs that survive both the internal-account filter and the contract-bracket join.

Verify it yourself

• Output row count equals COUNT(DISTINCT (account_id, date_trunc('month', usage_date))) on the source fact restricted to the same two filters; counted against the unrestricted fact it is strictly smaller, by exactly the internal and unmatched account-months.
• reported_net + internal_net + unmatched_net equals the raw sum from fct_usage_daily over the same usage_date range, to the cent, with all three terms computed from the same usage_monthly CTE.
• The reported figure alone is strictly below the raw sum whenever any internal account recorded usage in the window, and equal to it only in a window where none did.
• No account-month carries more than one plan_tier or employee_band value.
• Accounts with net_cents of zero return NULL margin rather than a division error, and margin-negative accounts appear with a value below zero rather than being dropped.

Follow-up

• An account amends mid-month from team to enterprise. Show what your query reports and argue for one attribution rule over the other.
• Your monthly total disagrees with the finance figure by a small amount. Where would you look first, and which number do you defend?

How do you prioritize tasks when working on multiple high-impact projects simultaneously?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. Restate the decision this analysis has to support, and who acts on the answer.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• What would you do if the primary metric and the guardrail moved in opposite directions?
• How would you detect that the metric is being gamed rather than genuinely improving?

How would you diagnose a sudden metric drop in a core product dashboard?

Medium · product sense and metrics

Approach

1. Name one primary metric, then the guardrail that stops it being gamed.
2. Fix the population and the time window before naming any metric.
3. State what result would change your recommendation, so the answer is falsifiable.

Follow-up

• Which segment would you cut first, and what would that rule out?
• What would you do if the primary metric and the guardrail moved in opposite directions?

If you were tasked with product metric design, what indicators would you choose to measure the success of a new search function?

Medium · product sense and metrics

Approach

1. Decompose the metric into the rates that drive it, and say which one you would check first.
2. State what result would change your recommendation, so the answer is falsifiable.
3. Name one primary metric, then the guardrail that stops it being gamed.

Follow-up

• How would you detect that the metric is being gamed rather than genuinely improving?
• Which segment would you cut first, and what would that rule out?

How would you design an A/B test to measure the impact of a new feature on user retention?

Medium · a/b testing and experimentation

Approach

1. Name the randomisation unit first; it decides the variance and what the test can detect.
2. Name the guardrails that would stop a launch even on a positive primary result.
3. Decide the analysis before seeing data, including how long it runs and when you look.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• What would you do if you could not randomise at all?

What are common experimentation pitfalls that could lead to biased results?

Medium · a/b testing and experimentation

Approach

1. Name the guardrails that would stop a launch even on a positive primary result.
2. Say whether units interfere with each other, and switch design if they do.
3. Name the randomisation unit first; it decides the variance and what the test can detect.

Follow-up

• What would you conclude if the result is positive but the test is underpowered?
• How would you handle interference between treated and control units?

Estimate a staggered overage-price change without a control arm

Hard · difference-in-differences · synthetic-control · staggered-adoption

On 2026-03-01 the overage rate for sku_code = 'ingest_gb' rose 18% for accounts whose dim_account.billing_country falls in six countries. Each account moves to the new rate at its own renewal, so treatment switches on across twelve months. Randomisation was never available. Using fct_usage_daily and fct_subscription_period, estimate the effect on net_amount_cents per account and on gross logo retention. Specify the estimator, the identifying assumption and how you test it, how you handle accounts that churn out of the panel, and how you do inference with six treated countries.

Approach

1. Fix the estimand before the estimator. Effect on revenue per account and effect on retention are different questions, and the second determines whether the first is worth having: an 18% rate rise that raises revenue while costing renewals is a loss. Define both on a cohort frozen before 2026-03-01, carrying churned accounts at zero revenue rather than dropping them, so attrition cannot masquerade as a revenue effect.
2. Do not run a single two-way fixed-effects regression. With treatment switching on at different renewal dates and effects that vary with time since treatment, the two-way fixed-effects coefficient is a variance-weighted average that includes comparisons of later-treated accounts against already-treated ones, and those comparisons can carry negative weights. Use a cohort-by-cohort estimator such as Callaway-Sant'Anna, or a stacked event study using only not-yet-treated controls, and report the event-study path rather than one number.
3. Treat parallel trends as a maintained assumption to be probed, not a result to be shown. Plot pre-treatment lead coefficients with intervals and state that flat leads are consistent with the assumption rather than proof of it. Handle anticipation explicitly: accounts notified of the increase before their renewal can pull ingest volume forward, so exclude a pre-renewal anticipation window and check whether the leads move when you do.
4. When the six treated countries are dominated by a few large accounts, the group mean is not a stable object and difference-in-differences on it will be noisy in a way its standard error will not reflect. Build a synthetic control at the country level from a donor pool of untreated countries, weighted to match the pre-period trajectory of net revenue per account, and report the placebo distribution across donor countries instead of a conventional standard error.
5. Do inference at the level treatment was assigned, which is the country. Six treated clusters is far too few for cluster-robust standard errors, which are badly downward-biased below roughly forty clusters. Use a wild cluster bootstrap, and report a randomisation-inference p-value from placebo assignments of treated status alongside it.
6. Apply the domain's data hygiene or the whole estimate is contaminated: exclude is_internal accounts, exclude the trailing metering settling window so the final months are not artificially low, and join as-of to the fct_subscription_period version live on each date rather than to is_current, which would price last year's usage at this year's contract.

Worked solution (45 min)

1. Build a balanced monthly account panel from fct_usage_daily joined as-of to fct_subscription_period, excluding is_internal accounts and carrying churned accounts forward at zero revenue.
2. Define treatment cohorts by renewal month, estimate group-time average treatment effects, and aggregate to an event-study path with leads from -6 to -1 and lags from 0 to +11.
3. Fit a country-level synthetic control on pre-period net revenue per account and generate the placebo distribution by refitting on each donor country in turn.
4. Compute a wild cluster bootstrap p-value clustered on country and report the randomisation-inference p-value from the placebo distribution beside it.
5. Repeat the full path for gross logo retention, restricted each month to the renewal-eligible base plus the 45-day grace window.

Expected result: An event-study path whose lead coefficients are statistically indistinguishable from zero and whose lags are reported by months since renewal, with country-level inference from a wild cluster bootstrap and a placebo-based p-value from the synthetic control. A single pooled two-way fixed-effects number is not reported, and the retention path is computed only on the renewal-eligible base.

Verify it yourself

• Treated and synthetic pre-period trajectories must track within the pre-period fit error. A poor pre-period fit cannot be rescued by a longer post-period and invalidates the comparison.
• Gross logo retention must be computed on accounts whose term_end_date fell in the month, not on the whole panel. On annual contracts only about a twelfth of accounts can churn in any month, so the whole-base rate understates churn by roughly an order of magnitude.
• At six treated clusters the wild cluster bootstrap p-value should be noticeably larger than the naive cluster-robust one. If they agree closely, the bootstrap was run on the wrong clustering variable.

Follow-up

• Your event study shows a significant lead coefficient two months before renewal. Is that anticipation, a violation of parallel trends, or a coding error, and what distinguishes them?
• Revenue per account rose 6% and gross logo retention fell 1.8 points. How do you combine those into a single recommendation, and over what horizon?
• One treated country contains an account holding 20% of that country's revenue. What does that do to the synthetic control fit, and what would you do about it?

Activation drops six points starting at a deploy hour

Medium · instrumentation · release analysis · metric inputs

Seven-day activation, defined as an account's first request with http_status < 400, api_key_id not null and traffic_class <> 'synthetic_monitor' within 168 hours of created_at, fell six points for sign-up cohorts after a Tuesday. A client SDK major version shipped that morning. From fct_api_request (account_id, api_key_id, sdk_name, sdk_version, http_status, traffic_class, request_at) and dim_account (account_id, created_at, is_internal), decide whether activation actually fell or the metric's inputs changed, and state in advance what evidence would convince you of each.

Approach

1. Decompose the definition and recompute activation under each relaxation: status only, status plus traffic_class, then the full definition. If the entire drop lives in the api_key_id clause, this is an instrumentation question rather than a behavioural one.
2. Measure the null rate of api_key_id by sdk_version and by hour. A stamping change appears as a step at the deploy boundary confined to the new version; a behavioural change appears as a ramp that grows with adoption and leaves old-version traffic untouched.
3. Hold the cohort's SDK mix fixed before comparing. New sign-ups adopt the newest version first, so a cohort-level drop can be pure composition even when no individual version changed at all.
4. Corroborate with a source the release did not touch: whether the same cohorts appear in fct_usage_daily with non-zero billable_quantity, and whether their fct_support_ticket rows with category = 'onboarding' rose.
5. Write the decision rule down before looking at the answer. An instrumentation artefact predicts unchanged downstream usage and a version-confined null step; a real regression predicts falling downstream usage and more onboarding tickets in the same cohorts.

Follow-up

• Old-version and new-version populations are not exchangeable, because new accounts adopt the new version first. How would you build a comparison that is not confounded by cohort age?
• What backfill or metric-versioning policy keeps the historical series interpretable once you fix the stamping?

Suggested practice checklist

For a candidate whose interviews will centre on A/B testing, metric movement and causal claims. Design comes before arithmetic, arithmetic before analysis, and the week ends by rehearsing the readout rather than the derivation.

Day 1 — Design one test end to end on paper

• Take a single feature change and write the full design: randomization unit, the exact point of exposure, the primary metric with its grain, guardrails, allocation, planned duration, and the decision rule committed before any data exists.
• Write why the randomization unit must sit at or above the level where treatment can spill over, and give one case where user-level randomization is still contaminated (shared accounts or devices, or two participants in the same marketplace).
• State in advance what you will do if the primary metric is flat while a secondary metric is significant.

Deliverable: A one-page test design with a decision rule written before launch.

Day 2 — Power arithmetic until it is automatic

• Compute required sample size per arm for a binary metric with the normal approximation, n is approximately 2 times (z for alpha/2 plus z for power) squared times p(1 minus p) divided by delta squared, for baselines of 2, 10 and 40 percent at a 5 percent relative lift, and note that for a fixed relative lift the requirement falls as the baseline rises because delta grows proportionally with p.
• Redo the calculation for a continuous metric using variance in place of p(1 minus p), and show why a heavy-tailed quantity such as revenue per user needs either far more traffic or a capped version with a stated cap.
• Convert one of the results into weeks given a weekly eligible traffic figure, then list the two honest ways to shorten it (accept a larger detectable effect, or reduce variance) and write why quietly lowering the power target is a decision to miss more real wins, not a speedup.

Deliverable: A small script or sheet that maps baseline, minimum detectable effect, alpha and power to sample size and weeks, cross-checked against a published calculator.

Day 3 — Variance and the unit-of-analysis problem

• Take a ratio metric whose denominator is not the randomization unit (clicks per session, randomized by user) and compute the standard error twice, once naively at session level and once by the delta method or a user-level bootstrap, then record how much the naive version understates it.
• Implement CUPED on simulated data: choose a pre-period covariate X measured before assignment, estimate theta as Cov(Y, X) divided by Var(X), and analyse Y minus theta times (X minus its mean) in place of Y. Confirm the variance of the adjusted outcome equals the raw variance multiplied by one minus the squared correlation between Y and X, so a correlation of 0.45 removes about 20 percent of the variance and not 80.
• Now run that simulation a few hundred times and confirm the adjusted effect estimate is unbiased for the same effect rather than numerically identical to the raw one. Within any single run the two differ, sometimes by a large fraction of the true effect, because the two arms' pre-period covariate means never coincide exactly in a finite sample; they agree in expectation, which is the property that matters and the one to state out loud.

Deliverable: A notebook showing the adjusted estimator with a measurably smaller variance than the raw one, plus a repeated-simulation table showing the two estimators agreeing on average while differing run by run.

Day 4 — Validity threats you can actually test for

• Run a sample ratio mismatch check as a chi-square goodness-of-fit test against the intended allocation, and write the three causes you would chase first (assignment logged before exposure, an arm-specific redirect or load failure, bot filtering applied asymmetrically).
• Simulate peeking: generate A/A data, test daily at alpha 0.05 across 14 looks, record the inflated false positive rate, then apply an alpha-spending boundary or commit to a fixed horizon and confirm the rate returns to nominal.
• Write how you would separate a novelty effect from a durable lift using the treatment effect plotted against days since first exposure, and what shape would change your recommendation.

Deliverable: One table showing the peeking false positive rate before and after correction, plus a written SRM triage list.

Day 5 — When randomization is not available

• Write the identifying assumption for difference-in-differences (parallel trends in the absence of treatment), then plot pre-period trends for two candidate control groups and justify rejecting one of them.
• Design a switchback test for a change where user-level randomization would leak across participants, choosing a time-block length against the carryover you expect and saying how you would detect carryover in the data.
• List what an interrupted time series or a synthetic control buys you and the one thing neither can rule out: an unobserved shock that coincides with the launch.

Deliverable: A one-page memo recommending a single quasi-experimental design and naming its weakest assumption explicitly.

Day 6 — The readout query

• Write the assignment-to-exposure join that returns exactly one row per unit per experiment, and handle units appearing in both arms by excluding and counting them rather than silently keeping one.
• Compute the per-arm metric, its variance and the relative lift with a confidence interval in SQL, then reproduce the identical numbers in a notebook as a cross-check.
• Add a segment breakdown and write the sentence that keeps it from being p-hacking: segments declared in advance, everything else reported as exploratory and corrected for multiplicity.

Deliverable: A single query that outputs the full readout table, matched to a notebook recomputation.

Day 7 — Present it to someone who will not read the appendix

• Give a 10-minute readout of a real or simulated experiment in the order decision, number, uncertainty, caveat.
• Have your listener ask "can we ship it" in the case where the primary is flat and a guardrail moved, and answer with a recommendation rather than a request for more data.
• Rewrite your opening line so the recommendation lands before any methodology.

Deliverable: A one-page readout whose first line is the recommendation.

Being wrong in public

A number you shipped turned out to be wrong, and someone had already acted on it. That is one of the most useful stories a data person can carry. What is being scored is how fast you noticed, who you told first, and what you changed in the process so the same class of error could not repeat quietly.

Describe a challenging project where you had to pivot your approach due to data limitations.

Medium · behavioural and stakeholder questions

Approach

1. State the situation in two sentences and spend the rest on your reasoning.
2. Quantify the outcome, including what you would not claim credit for.
3. Close with what you would do differently, concretely.

Follow-up

• What would you do differently if you ran that project again?
• How did you know the outcome was caused by your change?

Allocate one analyst week across three competing requests

Easy · prioritisation · stakeholders · scoping

Three requests arrive the same morning and you have one week. Finance wants per-account gross margin from fct_usage_daily for a pricing review in three weeks. Sales wants a renewal-risk list for accounts with term_end_date inside 60 days. A product manager wants an experiment readout for a decision being taken on Thursday. Produce your allocation with hours attached, what you say to whoever receives less, and one thing you refuse to do this week, with the reason each decision is defensible to the person it costs.

Approach

1. The interviewer is probing whether you prioritise on decision timing and reversibility or on who asked most forcefully. Sort by the date each decision is actually taken and by what the default outcome is if nothing arrives.
2. Apply that sort concretely. The Thursday readout has a hard irreversible deadline and no value afterwards. The pricing review has three weeks of slack. The renewal list has a rolling deadline set by term_end_date, so part of it is urgent this week and the rest is not, which means it can be split rather than deferred whole.
3. Find the cheapest sufficient version of each request rather than the full version. The readout goes in full. The renewal list ships as a filtered query over renewal-eligible accounts ranked by two inspectable signals rather than as a model. The margin work is scoped to the accounts that dominate the pricing decision, since revenue is heavily skewed and the tail will not change the conclusion.
4. Make the trade visible in one written note to all three at once, with dates. Telling each person separately that they are the priority is how an allocation becomes a credibility problem.
5. Refuse something explicitly and say why. The model version of the renewal list is the usual candidate, because it cannot be evaluated without a holdout nobody has agreed to yet, and building it this week forecloses that.
6. Leave slack. A plan with none is a plan to miss the one deadline that cannot move.

Follow-up

• The sales leader escalates to your manager. What did you already do that makes that a short conversation?
• Which of the three deadlines would you push back on, and what exactly would you ask for?
• What would you change about how these requests reach you so next week is not the same?

Report an underpowered consumption test to a non-technical executive

Medium · communication · uncertainty · experimentation

An account-randomised packaging change ran six weeks across 900 paying accounts. The effect on billable units per account per month is plus 4.1 percent, with a 95 percent interval from minus 3.2 to plus 11.8 after clustering standard errors at the account and applying the pre-registered winsorisation at the 99th percentile. An executive with no statistical background wants one number this week to decide a full rollout. Produce a three-sentence spoken answer, one chart, and an explicit recommendation of ship, stop or keep running, with the cost of each option stated.

Approach

1. The interviewer is probing whether you can be decision-useful without either hiding the uncertainty or hiding behind it. Start from the decision rather than the statistics: establish what the executive would do differently at plus 4 percent versus zero, because if the action is identical the interval does not matter.
2. Translate the interval into consequences in units the executive already reasons about. Multiply both endpoints by the cohort's baseline consumption and contracted rates to give an annualised revenue range, so the answer is a range of dollars rather than a range of percentages.
3. Price the option to wait. Using the observed variance, state roughly how many additional account-weeks halve the interval width, so keep running becomes a quantified choice instead of a stall.
4. Offer a cheaper path to the same decision: a lower-variance proximate outcome such as successful billable units on the new SKU, or CUPED using each account's pre-period consumption, quoting the expected variance reduction as one minus the squared pre-post correlation.
5. Give a recommendation and name the single observation that would reverse it. A strong answer commits; a generic one recites the interval and leaves the decision on the table.

Follow-up

• The executive says it clearly works and is just not provable, so ship it. What is your answer?
• How much of the interval width comes from clustering and how much from the revenue tail, and what would you do about each?
• If you had to ship this week with no more data, which guardrail would you watch for the first fortnight and at what threshold would you roll back?