Interview guides · Castellum
Castellum Data Scientist Interview Questions & Guide 2026
A two-week prep plan for Castellum data scientist candidates: write your answer before reading the solution, revisit misses three days later.
The 60-second brief
This guide covers what a Data Scientist at Castellum is expected to do and how to prepare for the interview.
If the team owns experimentation, expect depth past a two-sample test: minimum detectable effect and its roughly inverse-square-root dependence on sample size (holding power, significance level and variance fixed), variance reduction from pre-period covariates, interference between units, and when a sequential design is the right call.
Focus areas
- Pick a randomisation unit that respects interference
- Turn a vague request into a measurable question
- Define numerator, denominator and window precisely
Where to focus your preparation
Treat the sections below as preparation areas and confirm the format with your recruiter.
Preparation focus
No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.
What to demonstrate
- Breadth across SQL, experimentation and product reasoning
- Ability to state assumptions before choosing a method
How to prepare
- Drill the practice exercises below and time yourself
- Prepare three quantified stories about decisions you drove
Where candidates lose points
Comparing cohort retention curves of different maturities, or building the curve from users who are still present
A cohort four weeks old has no week-8 value, so an average taken across cohorts silently drops young cohorts from the later columns and keeps them in the earlier ones. The curve then bends upward at the tail, and the reading that 'retention is improving over time' is an artefact of which cohorts survived to be measured. The same error appears in the denominator when retention is computed over users active in the current period rather than over the full original cohort, which conditions on survival and guarantees a flattering number. The fix is a triangle: fix the cohort at signup, bound every window on both sides, and only compare cells where every cohort has had the full elapsed time, publishing the rest as blank rather than as a partial average.
Slicing a flat experiment until a segment reaches significance
Testing one metric across twenty segments at a nominal 5% level produces a significant result about two thirds of the time when nothing is happening anywhere, and the segment that surfaces is by construction the one with the most favourable noise. The reported effect in that slice is then badly overstated, because selection on significance conditions the estimate on being large. What makes it dangerous rather than merely wrong is that a post-hoc segment always has a plausible story attached, so it survives the meeting. The controls are declaring the small number of segments of interest before launch, correcting across the ones tested, and treating anything discovered afterwards as a hypothesis that needs its own adequately-powered test rather than a finding.
Reporting a p-value with no effect size or interval
Give the estimated difference with a confidence interval in the units the business cares about, then say whether that whole interval is worth acting on. A p-value only addresses whether you can rule out exactly zero; it says nothing about magnitude.
Never asking what decision the analysis will inform
Open with who makes the decision, what the options are, and by when. The answer determines the precision you need, the segments worth cutting, and whether an observational read suffices or an experiment is required.
Questions & practice
How do you approach feature selection when dealing with high-dimensional data?
Medium · statistics and probability
Approach
- Sanity-check the answer against a simple bound or a simulated case.
- Say what the estimate is of, and over what population it generalises.
- Translate the result into the decision it informs, in one plain sentence.
Follow-up
- How would you explain this result to someone who does not know statistics?
- What sample size would you need to detect an effect half this size?
Explain the difference between supervised and unsupervised learning in the context of a recent project.
Medium · statistics and probability
Approach
- Quantify uncertainty explicitly rather than reporting a point estimate alone.
- Translate the result into the decision it informs, in one plain sentence.
- Sanity-check the answer against a simple bound or a simulated case.
Follow-up
- What sample size would you need to detect an effect half this size?
- Which assumption here is most likely to be violated in practice?
If a model's performance degrades in production, what is your step-by-step process for root-cause analysis?
Medium · machine learning and modelling
Approach
- Say how the offline result would be validated online before it is trusted.
- Check what information would not exist at prediction time, and exclude it.
- Pick an evaluation metric that matches the cost of each error type, not a default.
Follow-up
- What would you monitor after launch to know the model is still valid?
- Where could label leakage enter this setup?
How would you explain the trade-offs between a Random Forest model and a Gradient Boosting machine to a non-technical stakeholder?
Medium · machine learning and modelling
Approach
- Pick an evaluation metric that matches the cost of each error type, not a default.
- Set a baseline first, so any model has something honest to beat.
- Say how the offline result would be validated online before it is trusted.
Follow-up
- How would you choose the decision threshold, and who owns that choice?
- What would you monitor after launch to know the model is still valid?
How do you validate the performance of a predictive model in an environment where ground truth is delayed?
Medium · machine learning and modelling
Approach
- Pick an evaluation metric that matches the cost of each error type, not a default.
- Frame the prediction: the label, the moment of prediction, and the action it triggers.
- Say how the offline result would be validated online before it is trusted.
Follow-up
- Where could label leakage enter this setup?
- What would you monitor after launch to know the model is still valid?
Rebuild per-visitor ordering without groupby convenience methods
Easy · pandas · vectorisation · window logic
You have a DataFrame of 2 million fct_event rows with visitor_id, occurred_at_utc and event_id, unsorted and containing duplicate timestamps within a visitor. Produce three new columns: event_rank, the 1-based position of the event within its visitor ordered by occurred_at_utc; seconds_since_prev, the gap to that visitor's previous event, NULL for the first; and is_first_for_visitor. You may use sort_values, shift, cumsum, numpy and boolean masking. You may not use groupby.transform, groupby.apply, groupby.cumcount, groupby.rank or merge_asof. Break timestamp ties on event_id.
Approach
- Sort once by ['visitor_id', 'occurred_at_utc', 'event_id'] and reset the index. The whole exercise reduces to row arithmetic on a sorted frame, and the tiebreak on event_id is what makes the result reproducible across runs.
- Mark visitor boundaries with is_first = df['visitor_id'].ne(df['visitor_id'].shift()). This is the single fact every other column derives from.
- Compute seconds_since_prev as the diff of the timestamp column, then overwrite it with NaT/NaN wherever is_first is True. The shift crosses the boundary between visitors and will otherwise hand the first row of each visitor the last event of the previous one.
- Build event_rank from a running counter that resets at boundaries: take a global cumulative position (np.arange(len(df))) and subtract, per row, the global position at which that visitor started. Get the start position by forward-filling the positions where is_first is True, which is a cumsum-free reset and is O(n).
- Verify against the forbidden method once, as a test rather than as the implementation, and confirm the two agree on every row.
Worked solution (20 min)
- Sort on the three-key tuple and reset_index(drop=True).
- Compute is_first via .ne(.shift()), which is True for row 0 because the shifted value is NaN.
- pos = np.arange(len(df)); start = pd.Series(np.where(is_first, pos, np.nan)).ffill(); event_rank = (pos - start + 1).astype(int).
- gap = df['occurred_at_utc'].diff().dt.total_seconds(); gap[is_first] = np.nan.
- Assert event_rank equals df.groupby('visitor_id').cumcount() + 1 on the sorted frame.
Expected result: Three columns on the sorted frame: event_rank starting at 1 for every visitor and increasing by 1 with no gaps, seconds_since_prev null exactly where is_first_for_visitor is True, and is_first_for_visitor summing to df['visitor_id'].nunique().
Verify it yourself
- is_first_for_visitor.sum() equals the number of distinct visitor_id values.
- seconds_since_prev.isna().sum() equals that same number.
- event_rank.max() per visitor equals that visitor's row count, and event_rank is non-negative everywhere.
- A visitor with two events at the identical timestamp gets ranks 1 and 2 with a gap of 0.0, ordered by event_id.
Follow-up
- The frame does not fit in memory. How does your approach change if you can only process one visitor-partitioned chunk at a time?
- occurred_at_utc is client-supplied and sometimes runs backwards within a visitor. Does your seconds_since_prev go negative, and should it?
- How would you extend this to reset the counter at every change of surface as well as visitor?
Read an experiment from first exposure, not assignment
Hard · experiments · window-functions · srm
fct_experiment_exposure holds experiment_id, unit_type, unit_id, variant, user_id, assigned_at_utc, first_exposed_at_utc, is_in_analysis_population and planned_end_utc. fct_event holds user_id, occurred_at_utc, is_core_action, and carries events up to a known data cut, :data_cut_utc. For one experiment randomised on unit_type = 'user', return per variant: exposed units, units with at least one core action in the seven days after that unit's own first exposure, the rate, and the variant share of exposed units. Only units whose seven-day window has fully elapsed as of the data cut belong in the readout. Units appearing under more than one variant are excluded from both arms and counted separately.
Approach
- Run the contamination pass as an aggregate, not a window: SELECT unit_id FROM fct_experiment_exposure WHERE experiment_id = :exp GROUP BY unit_id HAVING COUNT(DISTINCT variant) > 1, then anti-join it away. PostgreSQL rejects COUNT(DISTINCT variant) OVER (PARTITION BY unit_id) outright, since DISTINCT is not implemented for window functions; if you want the test inline, MIN(variant) OVER (PARTITION BY unit_id) <> MAX(variant) OVER (PARTITION BY unit_id) is the equivalent that does run.
- Do not resolve contamination by keeping the earliest variant. A unit that saw both arms carries treatment from both, so assigning it to either one biases that arm.
- Define the population as is_in_analysis_population = TRUE AND unit_type = 'user' AND first_exposed_at_utc < planned_end_utc AND first_exposed_at_utc + interval '7 days' <= :data_cut_utc. The horizon filter is what makes the readout reproducible next week instead of drifting with every re-run; the data-cut filter is the one that actually buys seven days of follow-up, since a unit exposed an hour before the horizon otherwise contributes an hour of observation to a seven-day rate.
- Measure the outcome on a per-unit relative window: LEFT JOIN fct_event on user_id with is_core_action = TRUE and occurred_at_utc in [first_exposed_at_utc, first_exposed_at_utc + interval '7 days'). LEFT JOIN so units with no outcome stay in the denominator at zero rather than being deleted by an inner join.
- Check the sample ratio before reading the effect: variant share of exposed units against the intended split, tested as a binomial. Run it on the truncated population as well as on the full exposed set, because if one arm exposes later on average the data-cut filter removes more of that arm and can manufacture a ratio mismatch the randomisation did not have. A mismatch on the full set means the exposure data is not a valid randomisation and invalidates the readout rather than being a footnote under it.
- Report the per-variant rate, the absolute difference, and the fact that the variance unit is unit_id. That is straightforward here only because the grain is already one row per user; a per-session outcome under user randomisation would need a delta-method or bootstrap standard error instead.
Follow-up
- Some units were assigned days before they were exposed. What does analysing the assigned set instead do to the estimated effect, and in which direction?
- The split is 51/49 on 400,000 exposed units. Do you read the result?
- The treatment arm exposes on average two days later than control. What does that do to a fixed calendar outcome window, and which arm does it favour?
Weekly visit-to-signup conversion split by acquisition channel
Easy · funnel · conversion · aggregation
From fct_session (session_id, visitor_id, started_at_utc, referrer_channel, is_bot_flagged, consent_state) and fct_event (visitor_id, occurred_at_utc, event_name), compute visit-to-signup conversion for one ISO week, split by channel. session_id is the unique key of fct_session. Denominator: distinct visitor_id with a session starting in the week, is_bot_flagged = FALSE and consent_state <> 'denied'. Numerator: those visitors with a 'signup_completed' event in the same week. Label each visitor with the referrer_channel of their first session in the window. Return channel, visitors, signups and rate, plus one all-channel total row.
Approach
- Build a visitor spine that is one row per visitor: filter sessions to the week, drop is_bot_flagged and consent_state = 'denied', then take the first session per visitor with ROW_NUMBER() OVER (PARTITION BY visitor_id ORDER BY started_at_utc, session_id) = 1 to carry the channel label. Collapsing to one row here is what makes the channel buckets mutually exclusive and the totals additive.
- session_id is the unique key, so that ordering is total and the label is reproducible. If the table carried no unique key you would have to write an explicit tie rule instead, because two sessions on different channels at the identical timestamp would otherwise label the visitor differently between runs.
- Attach the outcome as a semi-join (EXISTS on a signup_completed event for that visitor inside the same week) rather than a join to the event table, so a visitor who fires the event twice does not count twice and inflate the numerator past the denominator.
- Aggregate with COUNT() as visitors and COUNT() FILTER (WHERE signed_up) as signups, and compute the rate as signups::numeric / NULLIF(visitors, 0) so an empty channel returns NULL rather than a division error.
- Produce the total with GROUP BY GROUPING SETS ((channel), ()), which re-sums numerator and denominator for the total row. Averaging the channel rates gives a different and wrong number whenever channel volumes differ, which they always do.
- Verify the spine before trusting the output: COUNT(*) must equal COUNT(DISTINCT visitor_id), and the per-channel visitor counts must sum to the total row.
Worked solution (20 min)
- Write the filtered session CTE and check its row count against an unfiltered count, so you know how much volume the bot and consent filters removed.
- Add the ROW_NUMBER first-session pick and assert one row per visitor.
- Add the EXISTS outcome flag and aggregate with FILTER.
- Add GROUPING SETS for the total and format the rate to four decimal places.
- Spot-check one channel by hand: pull its visitor list, count signups directly, compare.
Expected result: One row per referrer_channel plus a total row, with visitors, signups and a decimal rate. Channels with zero signups appear with rate 0, not missing. The total rate lies strictly between the minimum and maximum channel rate unless one channel holds all the volume.
Verify it yourself
- Per-channel visitor counts sum exactly to the total row's visitor count.
- COUNT(*) equals COUNT(DISTINCT visitor_id) on the spine CTE.
- Replacing the EXISTS with a plain inner join changes the signup count only if duplicate signup_completed events exist; if it does change, say by how much.
- Every rate is between 0 and 1.
- Re-running the first-session pick with session_id removed from the ORDER BY returns the same labels; if it does not, visitors hold same-timestamp sessions on different channels and the tie rule is doing real work.
Follow-up
- The denominator is distinct visitors. If a browser release shortens cookie lifetime, what happens to this rate, and how would you tell that apart from a genuine drop?
- A visitor's first session is direct and their signup session is paid search. Your label says direct. When is that the wrong answer for the decision being made?
- How do you roll four weeks into a month, and why is averaging the four weekly rates wrong?
Walk me through how you would design an A/B testing framework for a new product feature.
Medium · a/b testing and experimentation
Approach
- Say whether units interfere with each other, and switch design if they do.
- State the primary metric and the minimum effect worth shipping, then size the test.
- Decide the analysis before seeing data, including how long it runs and when you look.
Follow-up
- What would you do if you could not randomise at all?
- How would you handle interference between treated and control units?
Size an activation test before anyone writes code
Easy · power · minimum detectable effect · proportions
A team wants to test a new onboarding checklist against the seven-day activation rate: users with is_core_action = TRUE events on at least two distinct UTC dates inside the first seven days, counted from dim_user.account_created_at_utc with is_internal = FALSE. Baseline is 22%. Weekly non-internal signups are 9,000, split evenly between two arms. State the minimum detectable effect after two weeks of enrolment, the enrolment needed to detect a 1.5 percentage point absolute lift, and the earliest date a readout would be honest.
Approach
- Use the two-proportion sizing shortcut n = 16 p(1-p) / d^2 per arm, where d is the absolute lift. The 16 is 2(1.96 + 0.84)^2 = 15.7 rounded, i.e. two-sided alpha 0.05, 80% power, equal arms.
- Invert it for the MDE at fixed n: d = sqrt(16 p(1-p) / n). Two weeks gives 9,000 per arm, so d = sqrt(2.746 / 9000) = 0.0175, which is 1.75 percentage points absolute or 7.9% relative on a 22% base.
- Run it forward for d = 0.015: n = 2.746 / 0.000225 = 12,202 per arm, 24,404 total, about 2.7 weeks of enrolment. Round up to three whole weeks so the enrolment window contains no partial week, since signups are not uniform across weekdays.
- Add the metric's own lag rather than reporting the enrolment end date. Activation is unobservable until seven days after the last enrolled signup, and the definition publishes on an eight-day lag, so the readout lands 29 days after launch.
- Close by testing whether the ask is realistic. If the team's honest prior is a 0.5pp lift, the requirement is about 110,000 per arm and roughly 24 weeks of enrolment, and saying so before launch is the deliverable.
Worked solution (20 min)
- Compute p(1-p) = 0.22 x 0.78 = 0.1716 and 16 x 0.1716 = 2.746.
- MDE at two weeks: n = 4,500 per arm per week x 2 = 9,000; d = sqrt(2.746 / 9000) = 0.0175.
- Sample for d = 0.015: n = 2.746 / 0.000225 = 12,202 per arm, which is 2.71 weeks of enrolment, so enrol three whole weeks.
- Readout date = 21 days of enrolment + 8 days of follow-up and publication lag = 29 days from launch.
- State the boundary of what this quarter can answer: below roughly 0.7pp the requirement passes 56,000 per arm and more than a full quarter of enrolment.
Expected result: MDE after two weeks is 1.75 percentage points absolute, 7.9% relative. Detecting 1.5pp needs 12,202 users per arm, so three weeks of enrolment and a readout 29 days after launch. Detecting 0.5pp needs about 110,000 per arm, roughly 24 weeks, which is the answer the team needs to hear before the code is written.
Verify it yourself
- MDE must fall as 1 / sqrt(n). Doubling enrolment from two weeks to four should move 1.75pp to 1.24pp, not to 0.88pp.
- Substituting d = 0.0175 back into 16 p(1-p) / d^2 must return about 9,000 per arm.
- An exact two-proportion calculator at alpha 0.05 two-sided, power 0.80, p1 = 0.22, p2 = 0.235 returns about 12,260 per arm, within half a percent of the shortcut.
Follow-up
- The team also wants trial-to-paid read from the same test. What does a second primary metric do to your alpha, and what would you actually do about it?
- Signups are heavier on weekdays. What goes wrong if enrolment runs 17 days instead of 14?
- The activation definition uses a two-distinct-days threshold. If that were loosened to one day, what happens to the baseline rate and to the sample you need?
Separate a definition edit from a behaviour change
Easy · definition-change · metric-versioning · cohorts
Seven-day activation rate is documented as core actions on at least 2 distinct UTC dates inside the first seven days after account_created_at_utc, and every cohort published through last week was computed on that definition. The dashboard now reads 38% for the signup cohort that started Monday, against 31% for the cohort before it. Two things landed in that week: an onboarding change, and a commit to the metric's SQL that you have not read yet and that came with no documentation update. You have dim_user, fct_event and the metric's version history. Deliver two numbers: how many of the 7 points are behaviour and how many are the definition. Then say which published cohorts are no longer comparable.
Approach
- Diff the metric SQL across the commit boundary and enumerate every changed predicate individually: the distinct-day threshold, the is_internal exclusion, the is_core_action source, the window bounds, and the identity key. A single commit routinely changes more than one, and each has its own sign.
- Dual-run both definitions over the same 26 cohort weeks. The definition effect is the level shift between the two series computed on identical data, which isolates it from anything that happened in the world.
- Read the behaviour effect only from the old-definition series, comparing the new cohort to its own trailing cohorts. The new-definition series cannot answer the behaviour question because it has no pre-period.
- Explain the mechanism of the changed predicate rather than only its size. Lowering a distinct-day threshold from 2 to 1 admits every user who acted once in a single session, which is a large and low-intent population, so a several-point jump is the expected magnitude and not a surprise.
- State the restatement plan: pick one definition, backfill the full history on it, and annotate the break date. A series carrying two definitions is not a time series.
Follow-up
- The onboarding team wants the 1-day threshold kept because it is easier to move. What is your argument, and what would change your mind?
- How would you make a definition change visible to every consumer of this metric without relying on people reading a changelog?
Suggested practice checklist
Instead of guessing where the week should go, day one measures it under a fixed rubric and allocates the remaining hours in proportion to the gaps. The method is deliberately rigid: the allocation is written down before any studying starts and is not renegotiated when a topic turns out to be unpleasant.
Day 1 — Diagnostic, scored before you study anything
- Sit a 100-minute timed diagnostic in four blocks: 30 minutes of SQL across three prompts, 25 minutes of short-answer statistics, 25 minutes on one modelling or case prompt, and 20 minutes delivering one behavioural story aloud.
- Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct, and 0 is stuck, grading the output rather than how the attempt felt.
- Allocate the hours for days two to five roughly in proportion to 3 minus the score in each block, write the allocation down, and commit to not revising it midweek.
Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.
Day 2 — Largest gap: find the boundary rather than the subject
- Break the weakest area into five named sub-skills (for query work: grain control, window frames, date arithmetic, set logic with NULLs, and reading a query plan) and rate each one, so the rest of the week targets a sub-skill instead of a subject.
- Solve three problems chosen to sit just above where the rating drops off, and for each write the first move you failed to make.
- Re-solve one of them from memory four hours later, on paper, with nothing open.
Deliverable: A five-item sub-skill map with the two blocking sub-skills circled.
Day 3 — Largest gap: drill the blocking sub-skill
- Do eight short repetitions of the same shape rather than eight different problems, so what you practise is the pattern and not the puzzle.
- Write the rule you now hold in one sentence, then test it against a case built to break it: a ranking function over a column with ties, or a two-sample test on observations that are obviously dependent.
- Have someone else read your one-sentence rule and find the precondition you left out.
Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.
Day 4 — Second gap, plus maintenance on your strongest area
- Run the same sub-skill map and boundary protocol on the second-largest gap, compressed into half the day.
- Spend 25 timed minutes on your strongest area to stop it decaying, choosing the hardest problem you can still finish rather than an easy warm-up.
- Compare how the two areas fail: whether you lose time on recall, on setup, or on arithmetic, because the fix differs for each.
Deliverable: A second sub-skill map plus a one-line diagnosis of how each area fails you.
Day 5 — The gap that is not a skill
- Record yourself answering one technical and one behavioural prompt, then count two things in the playback: how many seconds before your first clarifying question, and how many sentences you started without knowing where they ended.
- Rewrite your three most-used stock phrases into shorter versions, and practise saying "I do not know, here is how I would find out" without softening it into a guess.
- Deliver one answer again with a hard 90-second limit to force structure before detail.
Deliverable: Two recordings with a counted improvement in time-to-first-question.
Day 6 — Retest under day-one conditions
- Sit the same 100-minute diagnostic structure with new prompts of comparable difficulty and score it on the identical rubric.
- Compare block by block, and for any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
- Write which single block you would still lose the offer on.
Deliverable: A second scored rubric placed next to the first, with one named remaining risk.
Day 7 — Full loop under interview conditions
- Run a 60-minute mock covering the two blocks that moved least, with an interviewer instructed to interrupt and change direction.
- Write your recovery script for the moment you go blank: restate the question, state your assumption, name the first thing you would check.
- Reduce the week to the rule statements you wrote, each with its preconditions attached, then say every one of them out loud without reading it and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive being interrupted.
Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.
Ownership when the result is inconvenient
Sometimes the honest read is that the initiative did not work, and the person who commissioned the analysis was hoping otherwise. Interviewers want to know whether you softened it. Prepare the case where you delivered an unwelcome result, how you presented the uncertainty without hiding behind it, and what the team did next.
Describe a time you had to clean a highly unstructured dataset. What specific techniques did you use to ensure data integrity?
Medium · behavioural and stakeholder questions
Approach
- Close with what you would do differently, concretely.
- State the situation in two sentences and spend the rest on your reasoning.
- Quantify the outcome, including what you would not claim credit for.
Follow-up
- How did you know the outcome was caused by your change?
- What would you do differently if you ran that project again?
Choose between three teams' requests with one analyst-week
Medium · prioritisation · stakeholder · judgement
You have one analyst-week. Three requests land the same morning. A growth team wants a paid-channel readout before a Friday spend decision. A billing team wants gross monthly revenue churn rebuilt, because the current figure recognises cancellation at canceled_at_utc rather than period_end_utc and is therefore wrong. A product team wants a dashboard for a feature launching in six weeks. All three sponsors are peers of your manager. Deliver your ranking, the explicit rule that produced it, and the message you send to the two teams you defer.
Approach
- Recognise what is being probed: whether you prioritise on decision value and reversibility or on who asked most recently and most loudly. The generic answer sorts by importance; the strong one states a rule, applies it, and accepts the ranking it produces even where that is uncomfortable.
- Score each request on three statable things: the decision it unblocks and the date that decision is made, the cost of being wrong in the meantime, and whether the work is one-off or compounding. A wrong published churn figure compounds, because it is quoted downstream and enters forecasts; the channel readout has a fixed date that cannot move; the dashboard has six weeks of slack.
- Notice the tension between value and urgency rather than resolving it by feel. The churn defect is the most valuable item and the least urgent one, which is exactly the shape of work that never gets done.
- Break the churn item in two. A one-hour severity check, sizing the gap between the two recognition points in MRR, is cheap enough to do before ranking anything and may promote the item outright. Do that first, then rank.
- Make the deferrals concrete. Each deferred team gets a date, a reason expressed as another team's decision deadline rather than as relative importance, and the smallest thing you can hand them immediately.
Follow-up
- The dashboard team escalates to your manager. What do you say in that conversation?
- Your severity check shows churn is overstated by 15%. Does the ranking change, and does anybody need to be told today regardless of the ranking?
- A fourth request arrives Wednesday with a Thursday deadline. What comes off the list, and who do you tell first?
Walk through an analysis you got wrong and what changed
Easy · postmortem · data quality · self-assessment
Describe an analysis of yours that turned out to be wrong after somebody had already acted on it. You have four minutes. The account must name the defect mechanically, the join, the filter, the window or the identity key, rather than describing it as a communication problem. It must also say who did what because of the wrong number, how the error surfaced, how long it stood, and what control you put in place so that class of error cannot reach a decision again. Do not pick an error nobody acted on.
Approach
- Recognise what is being probed: whether you can be specific about your own failure without minimising it or performing contrition. The discriminator is whether the defect has a mechanism the listener could reproduce in their own warehouse.
- Choose the case by blast radius rather than by comfort. An error nobody acted on tests nothing, and picking one signals that you are managing the interview instead of answering it.
- Structure the account in six beats: the number, the decision it drove, the defect, the detection, the correction, the control. Keep the defect to one reproducible sentence, for example an inner join to fct_subscription_period that dropped accounts with no subscription row and so computed retention over payers only.
- State the direction of the bias, not only its existence. A filter or join that removes rows usually moves a metric predictably, and knowing which way shows you diagnosed the mechanism rather than patched the symptom.
- Be exact about detection and elapsed time. 'A colleague noticed' and 'the row-count assertion failed before publication' are different answers about the same organisation, and the second one is the one your control is supposed to produce next time.
- End on the control, its cost, whether it has fired since, and one thing it does not cover.
Follow-up
- What did the control cost, and has it fired since? If it never has, how do you know it works?
- How long did the wrong number stand before anyone questioned it, and what does that say about the review path it went through?
- What is the equivalent mistake you are most likely to make in this role, given the tables you would be working in?