Interview guides · CertiK

CertiK Data Scientist Interview Questions & Guide 2026

Timed data scientist practice for CertiK: short SQL and probability sets run end to end, then scored against a rubric.

The 60-second brief

A Data Scientist at CertiK plays a pivotal role in harnessing data to drive decision-making processes and improve product effectiveness. This position is crucial for analyzing and interpreting data related to blockchain security and smart contract audits, which are central to CertiK's mission of ensuring a safer and more secure blockchain ecosystem. By leveraging data science techniques, you will contribute to enhancing the security protocols of various blockchain applications and aid in the development of innovative solutions that protect users and stakeholders.

If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.

Focus areas

Measure churn only on renewal-eligible accounts
Strip CI, retry and synthetic traffic first
Separate contracted seats from actively used seats

Overview

The impact of this role extends beyond mere analysis; you will be involved in developing predictive models and algorithms that influence the design and functionality of CertiK's products. Your work will directly affect the reliability and trustworthiness of services offered to clients, making this role not only challenging but also incredibly rewarding. As part of an interdisciplinary team, you will collaborate with engineers and product managers to turn complex data into actionable insights, driving the future direction of CertiK's offerings.

The CertiK interview process

CertiK candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

Whoever runs this call is usually not a practitioner. They take notes, and a hiring manager skims those notes later, so the real question is whether your work survives being written down by someone outside the field. Test every project sentence against that: could a non-specialist repeat it correctly without knowing what a propensity score is? Carry a plain-language version of each project and one reason you want this particular role that you could not copy onto another application. Vagueness at this stage reads as inexperience, even when the underlying work was genuinely deep.

What to demonstrate

Whether a non-specialist can restate your projects accurately, since their paraphrase is what reaches the hiring manager
Whether your reason for wanting the role points at the work itself rather than the company's reputation
Whether your language signals the level being screened for: what you decided yourself versus what you were handed

How to prepare

Write a two-sentence, jargon-free version of each major project: the question nobody could answer, and the decision your work changed. Read it to someone outside data and have them repeat it back
Point your 'why this role' answer at something concrete in the job description or the product surface you would be working on, and keep it to two sentences
Have two questions ready about measurement: which metric the team is held to, and who acts on an analysis once it lands

Technical Assessments

This round decides whether someone can hand you a schema and a question and trust the number that comes back. Correctness under a clock is the bar, not clever syntax. The habit that separates strong from weak answers is checking the grain: after every join, know how many rows you expect and whether the count moved. Most wrong answers in this format are not wrong logic, they are a fan-out from a key that turned out not to be unique, or a filter applied before an aggregate when it belonged after. Say what you expect before you run it.

What to demonstrate

Whether your row counts survive each join, and whether you notice on your own when they do not
Deliberate handling of rows that fail to match, including whether the question needs an inner join or a left join with the non-matches kept and counted
Whether NULLs are treated on purpose, given that a NULL compares equal to nothing and that COUNT of a column skips it
Reaching a defensible answer inside the window instead of a refined one after it

How to prepare

Take a two-table schema, write a join that fans out on purpose, then fix it by collapsing the many-side to one row per key before joining. Repeat until the fix is reflex rather than recall.
Write a funnel as one query and print the distinct user count at each stage, then confirm each stage is a subset of the one above it rather than assuming it
Do a few timed runs in a plain text box with no autocomplete and no formatter, since assessment editors often have neither

Behavioral Interviews

Behavioural answers from data candidates get audited in a way that answers from other roles do not. When you say a model lifted retention, the next question is the denominator, the window, and how you knew the lift was not seasonal. So attach the measurement to each claim while you tell it: what the metric was before, over what period, and against what comparison. Numbers with no baseline read as rounded-up memory, and one unsupported figure tends to make the rest of the story sound rehearsed.

What to demonstrate

Whether each impact number arrives with a baseline, a window and a comparison, or as a bare percentage
Whether you can name the method that attributed the effect to your work (an experiment, a staged rollout, a seasonal control) or concede the link was correlational
Whether the magnitudes stay internally consistent when the interviewer multiplies them against the scale you described earlier

How to prepare

For each story, write the impact line as metric, value before, value after, window, and how attribution was established. Any line missing two of those five is a follow-up you will answer badly.
Re-derive one headline number from the source table rather than the deck that reported it. Resume numbers drift upward across retellings.
Decide in advance which figures you cannot share, and prepare the ratio or relative change you can give instead, so a confidentiality limit does not read as evasion.

Cultural Fit Discussion

Rounds outside the standard loop often open with something deliberately under-specified: a loose business problem, an open question about a product area, a dataset described in one sentence. The common failure is surveying, listing six plausible approaches and committing to none of them. The thing that separates a strong answer is scoping out loud. State what you are treating as the goal, name the metric you would move, say what you are choosing not to do and why, then take one path through to an actual answer. An interviewer can follow you down a narrow path. Nobody can grade a menu.

What to demonstrate

Whether you turn an ambiguous prompt into a stated question with a measurable outcome before doing any work
The judgement visible in what you cut, and whether you say why you cut it rather than silently dropping it
Whether you land on a concrete recommendation with its caveat attached, rather than an unranked set of options

How to prepare

Take three vague prompts, such as 'is this feature working', 'why did retention drop', and 'should we expand into a new segment'. For each, write one sentence of goal, one primary metric with its window, and two things you are explicitly not doing.
Practise giving the recommendation first and the reasoning second, in five minutes. Loosely defined rounds are usually time-boxed, and an answer that arrives last often does not arrive.
Keep a running assumption list as you talk, on paper or in the shared doc, so the interviewer can challenge one assumption instead of your whole answer.

Where candidates lose points

Randomising an experiment at the user level when users share an account

Two problems fire at once. Colleagues in one workspace see each other's work and talk to each other, so a treated user changes the behaviour of a control user in the same account, which violates the no-interference assumption and biases the estimate toward zero. Separately, outcomes within an account are strongly correlated, so the effective sample size is roughly n / (1 + (m - 1) * rho) for m users per account and intra-class correlation rho, not n. With rho around 0.3 and twenty users per account that is a design effect near 6.7, meaning a user-level confidence interval is about two and a half times narrower than it should be and results cross significance thresholds on noise alone. Randomise the account and cluster the standard errors.

Comparing accounts that received a sales or customer-success touch against those that did not

Assignment of coverage is deliberate and pulls in both directions at once: the largest accounts get a named owner because they are valuable, and the accounts showing distress get one because they are at risk. The comparison therefore mixes a strong positive selection with a strong negative one, and the naive estimate can come out with either sign depending on which assignment rule dominated during the period examined. Nothing about matching on observed size fixes this, because the risk signal that triggered coverage is usually the same signal that predicts the outcome. It needs either an actual randomised or staggered rollout of coverage, or a design built on a capacity constraint or territory boundary that assigns coverage for reasons unrelated to account health.

Explaining an aggregate move without decomposing the mix shift

Split the change in the aggregate into within-segment movement and movement in segment weights before you explain it. Every segment's rate can fall while the overall rate rises, purely because volume shifted toward segments that already had higher rates.

Reporting a p-value with no effect size or interval

Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.

Questions & practice

Discuss various evaluation methods used to assess model performance.

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.
Set a baseline first, so any model has something honest to beat.

Follow-up

What would you monitor after launch to know the model is still valid?
How would you choose the decision threshold, and who owns that choice?

Write a function to implement a specific algorithm (e.g., decision tree, clustering).

Medium · machine learning and modelling

Approach

Frame the prediction: the label, the moment of prediction, and the action it triggers.
Check what information would not exist at prediction time, and exclude it.
Say how the offline result would be validated online before it is trusted.

Follow-up

Where could label leakage enter this setup?
How would you choose the decision threshold, and who owns that choice?

Audit daily usage rows for grain and arithmetic violations

Easy · data-quality · pandas · grain

You are handed fct_usage_daily as a pandas DataFrame with account_id, workspace_id, sku_code, usage_date, billable_quantity, included_quantity_applied, overage_quantity, list_amount_cents, discount_amount_cents, net_amount_cents, cogs_cents, is_restated, first_written_at and restated_at. The declared grain is one row per (account_id, workspace_id, sku_code, usage_date). Write audit(df) returning a DataFrame with one row per failing check: check name, failing row count, and one example key. Cover at minimum grain duplication, negative quantities or amounts, the identity net = list - discount, billable = included + overage, and rows where is_restated is true but restated_at is null.

Approach

Check the grain before anything else with df.duplicated(subset=key, keep=False), and count rows rather than groups so a key appearing twice contributes 2 — if the grain is broken every arithmetic count below it is uninterpretable.
Express each invariant as a boolean Series over the whole frame. The cent columns are integers and compare exactly, so use !=; the numeric(18,6) quantity columns need np.isclose with atol=1e-6 because included + overage is a decimal sum.
Handle null as its own failure mode. Comparisons against NaN return False, so a check written as rows_that_pass = (a == b - c) silently files every null-amount row wherever the negation happens to land; build each check as violations = ~condition | column.isna().
Collect the checks as a list of (name, mask) pairs and assemble the output in one pass, so adding a check is one line and every check reports in the same shape.
Order the output with structural failures (grain, null keys) above arithmetic failures, and report zero-count checks too — a check that silently disappears when it passes is indistinguishable from a check that was never run.

Worked solution (20 min)

key = ['account_id','workspace_id','sku_code','usage_date']; dup = df.duplicated(key, keep=False); record dup.sum() and df.loc[dup, key].iloc[0].to_dict() as the example.
neg = (df[['billable_quantity','included_quantity_applied','overage_quantity','list_amount_cents','net_amount_cents','cogs_cents']] < 0).any(axis=1); net_bad = df.net_amount_cents.isna() | (df.net_amount_cents != df.list_amount_cents - df.discount_amount_cents).
qty_bad = ~np.isclose(df.billable_quantity, df.included_quantity_applied + df.overage_quantity, atol=1e-6) | df.billable_quantity.isna(); restated_bad = df.is_restated & df.restated_at.isna().
Assemble pd.DataFrame([{'check': n, 'failing_rows': int(m.sum()), 'example': first_key(m)} for n, m in checks]) with the grain and null checks listed first.

Expected result: One row per check with failing_rows as an integer count of rows rather than groups, and a JSON-serialisable example key for any check with a non-zero count; a single doubled key contributes 2 to the grain check.

Verify it yourself

df.groupby(key).size().max() == 1 if and only if the grain check reports zero.
When the net check reports zero, sum(net_amount_cents) == sum(list_amount_cents) - sum(discount_amount_cents) exactly.
Running audit on df.sample(frac=1, random_state=0) gives identical counts; the checks must not depend on row order.
Corrupting one net_amount_cents by a single cent raises exactly one check's count by exactly 1.

Follow-up

Which of these should block a dashboard refresh and which should only warn?
Rows with is_restated = true legitimately change value after first write. How do you make yesterday's audit result reproducible?
How would you extend this to catch a partition that is missing entirely rather than wrong?

How would you write a SQL query to extract specific data from a database?

Medium · sql and data manipulation

Approach

Compute rates by summing numerator and denominator separately, never by averaging rates.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Handle the rows that do not match: a LEFT JOIN with a NULL check is usually the question.

Follow-up

How does the query change if the join becomes one-to-many?
How would you verify this result without re-running the same query?

Collapse retried requests into logical operations per account

Medium · window functions · deduplication · null semantics

fct_api_request carries request_id, account_id, api_key_id, idempotency_key (null when the caller supplied none), is_retry, request_at and http_status. For one ISO week, count logical operations rather than HTTP calls: requests sharing (account_id, api_key_id, idempotency_key) collapse to the earliest one, while every request with a null idempotency_key is its own operation. Return per account the raw request count, the logical operation count and the ratio between them, and state which direction a rising 5xx rate pushes that ratio.

Approach

Split the stream before deduplicating. GROUP BY and PARTITION BY treat NULLs as equal to one another, while the = operator does not, so a single grouping pass folds every null-key request in an account into one 'operation' and can cut the count by orders of magnitude.
Deduplicate the keyed rows with ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) and keep rn = 1. Including request_id in the ORDER BY makes the result deterministic when two rows share a millisecond, which they will.
UNION ALL the null-key rows back in unchanged; they need no dedup and must not pass through the partition.
Aggregate per account: count() over the raw week, count() over the union, and the ratio of the two. Report the ratio, not the difference, so accounts of very different sizes are comparable.
Cross-check against is_retry, which flags a resend of the same idempotency key: raw minus logical should sit close to the count of is_retry rows, and a large gap means clients are resending without a key and the dedup is under-counting duplicates.
Name the inversion explicitly: 5xx responses provoke client retries, so amplification rises exactly when reliability falls, and any engagement metric built on raw request counts will show growth during an outage.

Worked solution (25 min)

Filter to the ISO week with request_at >= :week_start AND request_at < :week_start + interval '7 days', and record the raw count per account as the baseline.
Build keyed_dedup: SELECT ... ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) AS rn FROM the week WHERE idempotency_key IS NOT NULL, then keep rn = 1.
Build unkeyed: the same week's rows WHERE idempotency_key IS NULL, taken as-is.
UNION ALL the two, then GROUP BY account_id selecting the logical count; join back to the raw counts and compute raw::numeric / logical.
Validate on one account by comparing raw - logical against COUNT(*) FILTER (WHERE is_retry) for that account and explaining any gap.

Expected result: One row per account with raw_requests, logical_operations and an amplification ratio at or above 1.0, where logical_operations counts each distinct (account_id, api_key_id, idempotency_key) once and each null-key request once.

Verify it yourself

logical_operations is never greater than raw_requests, and the two are equal for accounts whose keys are all distinct.
For accounts with no null idempotency_key, raw_requests - logical_operations is close to the count of is_retry rows.
Deliberately grouping on the key without the null split reproduces a visibly lower count, confirming the NULL collapse is the mechanism.
Re-running the dedup returns the same surviving request_id set, proving the ORDER BY tiebreak made it deterministic.

Follow-up

Which of these two counts belongs in the billable-units metric, and which in an engagement metric?
An account's amplification ratio jumps from 1.05 to 3.0 in a day. Name three causes and the query that separates them.

How would you approach a situation where the data is ambiguous or incomplete?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Name one primary metric, then the guardrail that stops it being gamed.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

How do you prioritize multiple tasks with competing deadlines?

Medium · product sense and metrics

Approach

Decompose the metric into the rates that drive it, and say which one you would check first.
State what result would change your recommendation, so the answer is falsifiable.
Fix the population and the time window before naming any metric.

Follow-up

Which segment would you cut first, and what would that rule out?
What would you do if the primary metric and the guardrail moved in opposite directions?

How do you optimize code performance in data processing tasks?

Medium · product sense and metrics

Approach

Restate the decision this analysis has to support, and who acts on the answer.
Fix the population and the time window before naming any metric.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

What would you do if the primary metric and the guardrail moved in opposite directions?
How would you detect that the metric is being gamed rather than genuinely improving?

Walk me through your thought process for designing a data-driven solution for a specific problem.

Medium · product sense and metrics

Approach

Fix the population and the time window before naming any metric.
Restate the decision this analysis has to support, and who acts on the answer.
State what result would change your recommendation, so the answer is falsifiable.

Follow-up

How would you detect that the metric is being gamed rather than genuinely improving?
What would you do if the primary metric and the guardrail moved in opposite directions?

Can you describe the concept of a confusion matrix and its significance?

Medium

Approach

Clarify what is being asked and what a complete answer would contain.
Work from the decision backwards to the evidence you would need.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Measure a repackaging that moves usage between metered SKUs

Hard · mix shift · sku migration · metering lag · unit economics

A repackaging moves work previously billed as compute_hours into build_minutes at a different rate. Early readouts show compute_hours down 30% and total net_amount_cents flat. You have fct_usage_daily (account_id, sku_code, usage_date, billable_quantity, net_amount_cents, cogs_cents, is_restated, first_written_at, restated_at) and fct_api_request (account_id, environment, traffic_class, http_status, compute_ms, egress_bytes, request_at). Design the metric that says whether real consumption changed rather than merely moving between SKUs, state the settling window you will wait for and how you measured it, and name the unit-economics guardrail. Deliver the metric definition and the readout you would take to a pricing decision.

Approach

Refuse to compare SKU-level quantity series across the change. billable_quantity carries a different unit per sku_code, so a 30% fall in compute_hours cannot be added to or netted against a rise in build_minutes. Any metric that sums billable_quantity across sku_code produces a number with no unit and no meaning.
Build a SKU-invariant measure of delivered work from fct_api_request instead: successful, non-synthetic, non-load-test production requests with sum(compute_ms) and sum(egress_bytes) per account per week. That series is unaffected by which SKU the billing system attributed the work to, which is exactly the property this question needs.
Measure the settling window before reading anything. For a sample of historical usage_date values, compare the total as first written against the settled total after restatement, and exclude as many trailing days as it takes for the delta to fall under a stated tolerance. The tail of every usage series slopes downward for this reason alone and is routinely misread as a regression, then explained, then fixed.
Compare like for like at the account grain against each account's own pre-period. Split accounts by whether their workload was in scope of the repackaging, and read the change in delivered work per account relative to its own 8-week pre-period level, reporting the median and interquartile range. A fleet mean is dominated by whichever account happened to scale one batch job, because account consumption is heavy-tailed.
Guardrail the economics rather than the revenue line. Compute gross margin per account from net_amount_cents and cogs_cents and report it as a distribution. Flat revenue with identical work moved onto a SKU carrying a higher cogs_cents allocation is a real margin loss that a flat revenue line conceals completely.

Worked solution (40 min)

Measure the settling window: for 30 historical usage_date values, compute total net_amount_cents as first written against the current settled value, and take the number of trailing days after which the delta falls below a stated tolerance such as 0.5%.
Build weekly per-account delivered work from fct_api_request: successful request count, sum(compute_ms) and sum(egress_bytes), filtered to traffic_class in ('interactive','batch') and environment = 'production'.
Split accounts into in-scope and out-of-scope for the repackaging and compute the change in delivered work per account relative to its own 8-week pre-period, reporting median and interquartile range rather than a mean.
Compute per-account monthly gross margin before and after from net_amount_cents and cogs_cents, and report the count and revenue share of accounts whose margin crossed below zero.
Reconcile the two sources and state why request-derived work and fct_usage_daily quantities will not tie exactly, naming retries, failed requests and restatement.

Expected result: A three-part readout: the empirically measured settling window in days, the median per-account change in delivered work (successful production requests and compute_ms) for in-scope against out-of-scope accounts measured relative to each account's own pre-period, and the per-account margin distribution before and after. The conclusion states whether consumption actually changed or merely moved between SKUs, and at no point adds quantities across sku_code.

Verify it yourself

Out-of-scope accounts should show no change in delivered work; movement there means the scope split is wrong or something else shipped in the same window.
The settling window must be measured rather than assumed; re-running the readout a week later must not move the final week's figures by more than the stated tolerance.
Total net_amount_cents across all SKUs before and after must reconcile to the invoiced total; a gap means restated rows were read mid-settlement.

Follow-up

Revenue is flat and delivered work is flat. Name the one thing that could still have got worse, and exactly where you would see it.
Packaging changes are usually applied to everyone at once. How would you design this as an experiment rather than a before-and-after, and what would you give up?

Monthly logo churn triples with no change in satisfaction

Hard · renewal-eligible base · censoring · timestamp semantics

Monthly logo churn, computed as churned accounts divided by all paying accounts, tripled last month. Contracts are annual. From fct_subscription_period (account_id, term_start_date, term_end_date, amendment_type, auto_renew, booked_at, superseded_by_id, is_current) and dim_account (account_id, churned_at, account_status, employee_band, acquisition_channel), rebuild churn on the renewal-eligible base, separate calendar effects from customer behaviour, and state whether retention actually changed. Note that churned_at is sometimes set when the record was updated rather than at term end.

Approach

Rebuild the denominator as accounts whose term_end_date falls in the month. An account eleven months from renewal sits in the current denominator while being structurally incapable of entering the numerator, so on annual contracts the published rate understates the truth by roughly the reciprocal of the annual renewal fraction and moves with the signing calendar.
Plot the renewal-eligible base by month across two years. A signing surge twelve months earlier reproduces itself as an eligibility surge now, and a rate whose denominator ignores that tracks the sales calendar rather than customer sentiment.
Date each churn by term_end_date, never by churned_at or an update timestamp. Inspect the distribution of churned_at minus term_end_date: a backlog cleared in one batch appears as a mass at a single date and shifts losses into whichever month the operations team did its paperwork.
Apply the 45-day grace for late renewal paperwork so the most recent month and a half is marked not reportable, rather than printing a number that will rise once the paperwork lands.
Compare corrected gross logo retention against its own trailing distribution, and if a real change survives, cut it by employee_band, acquisition_channel and plan_tier before proposing any cause.

Follow-up

Some contracts in the window have not reached their renewal date. When does this require a survival estimator rather than a simple rate, and which one would you use?
How do you report churn to an audience that wants a monthly number when the underlying event is annual and lumpy?

Suggested practice checklist

Instead of guessing where the week should go, day one measures it under a fixed rubric and allocates the remaining hours in proportion to the gaps. The method is deliberately rigid: the allocation is written down before any studying starts and is not renegotiated when a topic turns out to be unpleasant.

Day 1 — Diagnostic, scored before you study anything

Sit a 100-minute timed diagnostic in four blocks: 30 minutes of SQL across three prompts, 25 minutes of short-answer statistics, 25 minutes on one modelling or case prompt, and 20 minutes delivering one behavioural story aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct, and 0 is stuck, grading the output rather than how the attempt felt.
Allocate the hours for days two to five roughly in proportion to 3 minus the score in each block, write the allocation down, and commit to not revising it midweek.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Break the weakest area into five named sub-skills (for query work: grain control, window frames, date arithmetic, set logic with NULLs, and reading a query plan) and rate each one, so the rest of the week targets a sub-skill instead of a subject.
Solve three problems chosen to sit just above where the rating drops off, and for each write the first move you failed to make.
Re-solve one of them from memory four hours later, on paper, with nothing open.

Deliverable: A five-item sub-skill map with the two blocking sub-skills circled.

Day 3 — Largest gap: drill the blocking sub-skill

Do eight short repetitions of the same shape rather than eight different problems, so what you practise is the pattern and not the puzzle.
Write the rule you now hold in one sentence, then test it against a case built to break it: a ranking function over a column with ties, or a two-sample test on observations that are obviously dependent.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on your strongest area

Run the same sub-skill map and boundary protocol on the second-largest gap, compressed into half the day.
Spend 25 timed minutes on your strongest area to stop it decaying, choosing the hardest problem you can still finish rather than an easy warm-up.
Compare how the two areas fail: whether you lose time on recall, on setup, or on arithmetic, because the fix differs for each.

Deliverable: A second sub-skill map plus a one-line diagnosis of how each area fails you.

Day 5 — The gap that is not a skill

Record yourself answering one technical and one behavioural prompt, then count two things in the playback: how many seconds before your first clarifying question, and how many sentences you started without knowing where they ended.
Rewrite your three most-used stock phrases into shorter versions, and practise saying "I do not know, here is how I would find out" without softening it into a guess.
Deliver one answer again with a hard 90-second limit to force structure before detail.

Deliverable: Two recordings with a counted improvement in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 100-minute diagnostic structure with new prompts of comparable difficulty and score it on the identical rubric.
Compare block by block, and for any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write which single block you would still lose the offer on.

Deliverable: A second scored rubric placed next to the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a 60-minute mock covering the two blocks that moved least, with an interviewer instructed to interrupt and change direction.
Write your recovery script for the moment you go blank: restate the question, state your assumption, name the first thing you would check.
Reduce the week to the rule statements you wrote, each with its preconditions attached, then say every one of them out loud without reading it and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive being interrupted.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Where your data came from

Data people depend on systems owned by other teams, and much of the job is negotiating for instrumentation, access, or a fix to a broken pipeline. Prepare an example of getting something changed upstream that you did not control. Describe what you asked for, what you traded, and how you worked while you waited.

Can you provide an example of how you have contributed to a team’s success?

Medium · behavioural and stakeholder questions

Approach

State the situation in two sentences and spend the rest on your reasoning.
Name the disagreement or constraint, and how you resolved it with evidence.
Pick a story where you drove the decision, not one where you observed it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that project again?

Announce a metric fix that cuts the headline number

Medium · metric definitions · stakeholders · communication

Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.

Approach

The interviewer is probing whether you can land a correction as an operational change with a plan attached, rather than as an announcement other people then have to clean up after.
Quantify each exclusion separately before telling anyone: internal accounts, synthetic monitors, load tests. Three known quantities are a discussion; one alarming total is an argument.
Be precise about which side of the metric each exclusion touches, because one team's goal is on a count and the other's is on a ratio. The traffic-class filters remove requests, so they shrink the numerator only. Dropping internal accounts removes them from the ratio's denominator as well, since internal accounts carry ordinary account_status values and therefore sit in that denominator. Internal accounts are active in almost every week while the real base is not, so the numerator loses a larger share than the denominator and the ratio falls by less than the count does. Compute both and say which one the 11 percent is before anybody assumes.
Check whether the trend changes, not only the level. A constant 11 percent shift is a rebasing and nothing more. A shift that widens over time means the reported growth was partly internal or synthetic, which makes the existing goals unachievable as written and changes what you are asking teams to do.
Sequence the disclosure: the metric owner and the two teams whose goals move first and privately, then the board channel with a written bridge, then the dashboard. The dashboard is last because a number that changes without explanation is read as instability rather than as a fix.
Run both series for one reporting period with the bridge visible, restate history rather than letting the series break at a date, and set the date the old series is removed.
Propose the goal treatment yourself: rebase each target by the shift measured on the metric that target is written against, rather than leaving each team to negotiate individually, which is where corrections of this kind usually die.

Follow-up

One team's quarterly goal is now unreachable. Rebase the target or let it miss, and what does each choice teach the organisation?
How would this have been caught when the metric was first defined?
What else on that dashboard shares this failure mode, and how would you find out this week?

Disagree with a product manager about an adoption claim

Medium · conflict · telemetry hygiene · adoption metrics

A product manager is about to present that a new SDK release drove a 40 percent rise in requests among adopting accounts, computed from fct_api_request counts grouped by sdk_version. You find the rise is concentrated in traffic_class equal to ci, that rows with is_retry true grew alongside it, and that restricting to interactive non-retry traffic leaves a 3 percent lift. The launch review is in two days. Decide how you raise this, with whom and in what order, and what you propose the claim becomes.

Approach

The interviewer is probing whether you can correct a colleague without ambushing them, and whether your own counter-analysis carries the caveats theirs lacked. Go to the product manager privately before the review. A correction delivered in the room is a status move and loses the argument you are actually trying to win.
Bring a decomposition rather than a verdict: the same accounts and window, requests split by traffic_class with retries held out as their own column, so their 40 percent and your 3 percent reconcile line by line and neither has to be taken on trust.
Reproduce their figure exactly first. If you cannot land on 40 percent with their method, you do not yet know what you are disagreeing with.
Ask whether the continuous-integration lift is itself valuable. An account wiring the SDK into its pipeline has increased integration depth, which is the dominant switching cost in this domain, so the honest claim may be that integration depth rose while interactive usage moved 3 percent. Improving the claim beats deleting it.
Name the mechanism that makes the raw count dangerous: clients retry when the platform degrades, so retry volume climbs exactly when the customer is most at risk. Pull the 5xx rate for the same accounts and window before anyone concludes anything, and note that billable_units is zero on 5xx rows, so request counts and billable quantities diverging is itself the signal.
Close with a standing definition for launch metrics so the next release does not repeat the exercise.

Follow-up

The product manager argues that continuous-integration traffic is real usage and declines to split it out. Is that position defensible, and under what metric definition?
Suppose the 5xx rate for those same accounts also rose 40 percent. What is the claim now?
The review happens and the raw number is presented regardless. What do you do next, and what do you not do?