Answer Digital Software Engineer Interview Guide

The 60-second brief

Candidates describe Answer Digital as a company that delivers software for client businesses. The Software Engineer role covers designing, building and maintaining those solutions inside agile teams, with stand-ups and product owners in the picture. Integration work comes up repeatedly in the reported material, and senior integration engineers are named among the colleagues you would work with. Projects can be new builds or upkeep of infrastructure that is already running, and engineers talk to client stakeholders directly to clarify requirements and give technical updates. The role description also mentions code reviews and mentoring junior colleagues. For you, that means the day-to-day picture is client-facing delivery where connecting systems correctly matters as much as writing a clean method.

This guide is for candidates interviewing for a Software Engineer position at Answer Digital, at early-career or more senior level. It covers the five reported questions (project challenges, database design for high traffic, debugging an integration issue, maintainable and scalable code, and integration patterns and tools), the reported topics of integration testing, hands-on coding, Java method implementation and communication with non-technical stakeholders, and a plan for rehearsing all of it. No round sequence has been reported, so confirm the format with your recruiter.

No fixed round order is reported. Candidates mention a mix of remote technical tests, group tasks and conversations with developers at different levels of seniority, in person or remote.

Focus areas

• Integration Testing
• Practical Coding (Hands-on Implementation)
• Java (Method Implementation)

Overview

The work candidates describe is client delivery in agile teams, and a good share of it is integration: getting one system to exchange data correctly with another, then keeping it reliable once it is live. That shapes what to practise. Expect to be asked about integration testing, about writing working code by hand (Java method implementation is a reported topic), and about the database design behind an application. The role description lists one major programming language, data structures and algorithms, and database design as the core requirements, with cloud infrastructure, modern integration patterns and agile or scrum experience as extras.

The technical material candidates report is practical rather than puzzle-driven: a live coding session, a database schema to design from a business requirement given during the conversation, pseudocode to read and reason about without a particular syntax, and a technical assessment built around a simple integration task. Prepare by doing these on paper or in a plain editor. Write Java methods without autocomplete, sketch schemas with keys and indexes, trace pseudocode with a table of variable values, and build one small integration with a stubbed counterparty so that you have tested the failure paths.

The other half is communication. Communication with non-technical stakeholders and teamwork are both reported topics, and candidates mention group tasks and peer-led sessions where collaboration is part of what is seen. Prepare two or three real projects from your CV that you can explain at two levels, one for engineers and one for a client who cares about cost, risk and delivery dates. Candidates are also told to explain why they chose a tool, not just which one, and to be authentic rather than recite stock answers, so build every story around a decision you made and the trade-off it carried.

Where to focus your preparation

No round order has been reported, so the single stage below groups the formats candidates mention. Ask your recruiter for the actual sequence and which stages are remote.

Preparation focus

Candidates mention several formats without a fixed order: remote technical tests, group tasks, and in-person conversations with developers at various levels of seniority. Reported technical content includes live coding, a database schema to design from a business requirement, pseudocode to interpret, and a technical assessment involving a simple integration task. The conversational side covers past projects, career aims and working in a team. Treat the list as a menu, not a schedule: ask the recruiter which formats apply to your team, then prepare each one. For coding, practise in a plain editor and say your assumptions aloud. For the schema, ask about entities, volumes and access patterns before drawing tables. For pseudocode, trace values step by step and state what the logic computes. For conversations, have two or three projects ready with the decisions and trade-offs behind them.

What to demonstrate

• Working, readable code for a practical task, which is what a hands-on implementation format tests directly
• Database design from a stated business requirement: entities, keys, normalisation and performance trade-offs
• Reading pseudocode and reasoning about its logic without relying on one language's syntax
• Explaining technical choices to people outside engineering, and collaborating in group or peer-led settings

How to prepare

• Ask your recruiter for the sequence, which stages are remote, whether coding is live, and which language you can use
• Write five Java methods by hand (group-by with a map, order-preserving dedupe, merge of two sorted lists, input validation) and test each with edge cases
• Sketch two schemas from a one-paragraph business requirement, with primary and foreign keys, constraints and the indexes your main queries need
• Build one small integration against a stubbed HTTP service and test success, timeout, malformed payload and retry behaviour

Where candidates lose points

Answering the project question with a feature list instead of the technical challenges you faced

Pick one project and one hard problem in it. State the constraint, the two options you weighed, the one you chose and what it cost you. Rehearse it aloud until it fits a short, clear answer, and keep the feature tour to a sentence.

Drawing tables straight away when asked to design a database for a high-traffic application

Ask first: which entities, how many reads against writes, which queries matter most. Normalise, add keys and constraints, then say where you would add indexes, a read replica, caching or denormalisation and what each costs. Name the query your design makes slow.

Guessing at code when asked to debug a complex integration issue

Narrate a sequence: reproduce or find a failing request, work out which side of the boundary misbehaves from logs and captured payloads, compare the actual message to the agreed contract, fix it, then add a test that would have caught it. Say what each step rules out.

Writing Java methods in silence and skipping boundary cases

State assumptions about input before typing. Handle null, empty and single-element cases, then trace one example through the finished method and say the complexity. If the task is given as pseudocode, convert it with a value table before you translate it.

Explaining a technical trade-off only in jargon to a stakeholder, or going quiet or dominating in a group task

Restate each trade-off as cost, risk and delivery impact, and check that the listener followed. In group and peer-led settings, ask others for their view, summarise what has been agreed and contribute your own pieces so the session reads as shared work.

Questions & practice

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

• The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
• How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.

Follow-up

• The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
• Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

1. Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
2. Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
3. State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
4. Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.

Follow-up

• The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
• A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

1. Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
2. Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
3. For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
4. State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.

Follow-up

• A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
• How do you honour an erasure request while resource_revision.actor_user_id still references this table?

Explain a project you have worked on, focusing on the technical challenges you faced.

Medium · Technical Proficiency and Domain Knowled

Approach

1. Pick one project where you owned a hard technical problem yourself, such as an integration between two systems, a data migration or a performance fix, not one where you only watched the decision.
2. Set the context in a few sentences: what the system did, who used it, its rough load or data volume and the constraint that made the problem hard (a legacy API, a fixed deadline, data you could not lose).
3. Spend most of the answer on the challenge itself: the options you considered, the trade-off between them (for example polling versus webhooks, or a batch job versus streaming) and why you chose one. Say 'I chose' and 'I built', not 'the team decided'.
4. Close with evidence: what you measured before and after (latency, error rate, a manual step removed), how you tested the change, and one thing you would design differently now.
5. Be ready to sketch the architecture on a whiteboard: components, data flow and where the failure points were. Explain them in plain terms too, because the reported topics include communication with non-technical stakeholders.

Follow-up

• Why did you not choose the alternative you rejected, and when would it have been the better option?
• How did you test the change before it reached production, and what did your tests miss?
• If the load on that system grew tenfold, which part would fail first?

How would you approach the design of a database for a high-traffic application?

Medium · Technical Proficiency and Domain Knowled

Approach

1. Start by clarifying the access patterns: the read-to-write ratio, the hottest queries, how much data, the latency target and which data needs strong consistency (orders, balances) and which can be slightly stale (counts, feeds).
2. Model the schema in third normal form first so every fact lives in one place, then denormalise only the hot read paths you can name, and say what keeping the copy in sync costs on writes.
3. Design indexes from the queries. A composite B-tree index on (a, b) serves filters on a, or on a and b together; a filter on b alone usually needs its own index. Every extra index slows inserts and updates, so on a write-heavy table, index only for queries that actually run.
4. Scale reads with connection pooling, a cache-aside layer such as Redis with TTLs and explicit invalidation on write, and read replicas. Replicas lag, so route reads that must see the user's own just-written data to the primary.
5. Scale writes last, with partitioning or sharding on a key such as tenant or customer id. Name the cost: queries that span shards become scatter-gather, hot keys can overload one shard, and resharding is expensive, so pick the key from the dominant query.

Follow-up

• A user saves a change and immediately reloads the page from a replica that has not caught up. How do you guarantee they see their own write?
• How would you add a column or change a column type on a large, busy table without downtime?
• When would you choose a document or key-value store over a relational database for this application?

How do you ensure your code is maintainable and scalable?

Medium · Technical Proficiency and Domain Knowled

Approach

1. Answer with practices you actually use, each tied to a reason. Maintainability: small units with one responsibility, clear names, interfaces at the boundaries, and dependencies injected so a class can be tested with a fake instead of a live database or API.
2. Make tests the safety net that keeps refactoring cheap: fast unit tests for the logic, integration tests at the database and external-API boundaries, and CI running them on every pull request alongside linting and static analysis.
3. Treat code review as part of maintainability: small pull requests, review comments about design rather than style (leave style to the formatter), and documenting the reason behind a non-obvious decision next to the code.
4. Scalability at code level: know the complexity of your hot paths. For example, replace a list search inside a loop (O(n*m)) with a HashMap or HashSet lookup, which is O(1) on average, remove N+1 queries by batching or joining, and paginate anything that can return unbounded results.
5. Scalability at service level: keep services stateless so you can add instances behind a load balancer, move session or shared state into a database or cache, and profile or load-test before you optimise, so the change targets the real bottleneck.

Follow-up

• Show an example of a piece of code you refactored: what was wrong with it, and how did you know the refactor did not change its behaviour?
• How do you decide between shipping a quick fix and taking the time to restructure the code?
• Your service runs as one instance and has to handle five times the traffic. What would you check before adding instances?

Can you walk us through your process for debugging a complex integration issue?

Medium · Technical Proficiency and Domain Knowled

Approach

1. Pin down the symptom exactly before forming a theory: which call fails, the error or status code, what fraction of requests fail, whether it is consistent or intermittent, and when it started. Then line that start time up against deploys, config changes, certificate or credential rotations and changes on the partner's side.
2. Find the hop where it breaks. Follow one failing request across systems using a correlation or trace id, and compare the payload, headers and response at each boundary, so you know which side is wrong before you change anything.
3. Check the usual integration faults one by one: contract or schema drift (a renamed or newly nullable field), expired tokens or clock skew, timeouts set shorter than the downstream's real latency, character encoding and time-zone handling, and retries without idempotency that create duplicates.
4. Reproduce it outside production. Replay the captured failing payload against a stub or a test instance of the other system, and narrow the cause by changing one variable at a time.
5. Fix the cause rather than the trigger, then lock it in: add an integration or contract test that replays the failing case, add an alert on the failing signal, and write up the root cause for the other system's owners.

Follow-up

• The failure happens only in production and only for a small share of requests. How do you gather evidence without being able to attach a debugger?
• The fault is in a third-party system you do not control. What do you do in the meantime, and how do you show them the evidence?
• After you added retries, the downstream system started receiving duplicate records. How do you make the integration idempotent?

Suggested practice checklist

The plan follows the reported topics and questions: Java methods, integration testing and debugging, database design, pseudocode and group work, then spoken answers about your own projects.

Day 1 — Confirm the format and inventory your stack

• Email your recruiter asking for the stage sequence, which stages are remote, whether coding is live or a test you complete alone, and which language you may use.
• List every technology on your CV with one sentence on why you chose it for that project and what you would use instead today.

Deliverable: A sent recruiter email and a one-page stack list with a reason for each tool.

Day 2 — Java method implementation by hand

• In a plain editor with no autocomplete, write a group-by using a HashMap, an order-preserving dedupe with LinkedHashSet, a merge of two sorted lists, and an input validator.
• Add three edge-case tests to each method (null, empty, duplicates) and write its time and space complexity.

Deliverable: Four tested Java methods plus the practice solution, each with complexity noted in a comment.

Day 3 — Integration testing and debugging an integration issue

• Build a small Java client that calls a stubbed HTTP endpoint and write tests for success, timeout, 5xx with retry, and a malformed payload.
• Say aloud, then write out, your debugging sequence for a complex integration issue: reproduce, locate the failing side, compare the payload to the contract, fix, add a test.

Deliverable: A tested client with four failure-path tests, the practice solution, and a written debugging sequence you can recite.

Day 4 — Database design from a business requirement

• Take a short business requirement (orders, customers and invoices works well) and produce a schema with keys, constraints and indexes, stating what normal form you reached.
• Write an answer to designing a database for a high-traffic application: access patterns, indexes, read replicas, caching and the query your design makes expensive.

Deliverable: One schema with DDL, a written answer on the high-traffic design, and two solved SQL practice problems.

Day 5 — Pseudocode reading and group-task practice

• Trace three pseudocode snippets (a loop with a counter, a nested condition, a recursive step) on paper with a table of variable values, then state in one sentence what each computes.
• Do a short problem with one or two other people, practising asking for input, summarising what is agreed and splitting the work.

Deliverable: Three completed trace tables with one-line summaries, and a note on what you would do differently as a group participant.

Day 6 — Say your project and integration stories aloud

• Write and rehearse the project-challenges story: one problem, the options, your choice and its cost. Then rehearse your answer on keeping code maintainable and scalable, with two concrete habits such as tests, code review and clear module boundaries.
• Write the integration patterns and tools answer using two real integrations, covering pattern choice, failure handling and testing.

Deliverable: Three spoken answers outlined on one page each, plus two outlines for the practice stories.

Day 7 — Mixed dry run and questions to ask

• Run one mixed session: a Java method task, then a schema from a requirement someone reads out, then a plain-language explanation of one trade-off as if to a client.
• Write two questions for your interviewers about how their teams run projects and integrations.

Deliverable: A completed mixed session with a list of weak spots, and two written questions for interviewers.

Projects, integrations and working with clients

Candidates report communication with non-technical stakeholders and teamwork and collaboration among the tested topics, and the role description mentions stand-ups, product owners, code reviews and mentoring juniors. Prepare stories where you made a technical decision, explained it to someone outside engineering, and worked through a disagreement or a shared problem with colleagues.

1. Explain a project you have worked on, focusing on the technical challenges you faced.
2. Describe a time you explained a technical trade-off to a non-technical person and what they decided as a result.
3. Tell me about a disagreement with a teammate over a technical approach and how it was resolved.
4. Describe a bug that sat across your system and another team's. How did you find which side owned it, and how did you work together on the fix?
5. Tell me about a code review where feedback you gave or received changed the design.

Describe your experience with various integration patterns and tools.

Medium · Technical Proficiency and Domain Knowled

Approach

1. Choose two or three integrations you personally built or debugged and name the pattern for each: a synchronous REST call, a message queue with publish and subscribe, a webhook, a scheduled file or batch transfer, or an API gateway or middleware layer. Name the actual tool you used. Do not list tools you have only read about.
2. For each one, say which systems were connected, the direction and rough volume of data, and why that pattern beat the alternative (sync versus async, polling versus push, batch versus event). The reasoning is the answer, since the question asks for experience rather than a glossary.
3. Cover failure handling, which is where real experience shows: timeouts, retries with backoff, idempotency keys so a replayed message does not create a duplicate, a dead-letter queue for messages that keep failing, message ordering, and what happened when the other side changed its payload.
4. Cover testing: contract tests against the agreed schema, stubs or sandboxes for the counterparty, and an end-to-end check with realistic data. Integration testing is a reported topic, so show how you knew the connection worked before go-live.
5. Finish with the outcome and one thing you would change. If a client or product owner was involved, say how you explained the trade-off to them. The mistake that sinks this answer is a tool inventory with no decision you owned.

Follow-up

• Why did you pick a queue over a direct API call there, and what did you give up?
• How did you stop a retried message from creating duplicate records downstream?
• How did you test the integration when the other system was not available to you?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.

Follow-up

• What would you do differently in the first five minutes, given the same dashboard and no more information?
• Which follow-up action did you deliberately not take, and why was dropping it the right call?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

1. Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
2. Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
3. Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
4. Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.

Follow-up

• How would you detect a consumer that reads the field only during a monthly export?
• One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?