Incident Software Engineer Interview Guide

The 60-second brief

A Software Engineer at Incident plays a pivotal role in building and maintaining the robust systems that define our operational excellence. You will be responsible for translating complex requirements into efficient, scalable, and maintainable code that directly impacts our product performance and user experience. This role is critical to the continued success of Incident, as our engineering team sits at the heart of our ability to innovate and solve real-time challenges. You will contribute to meaningful projects, collaborating with cross-functional teams to design solutions that handle significant data loads and provide high availability. You can expect a fast-paced environment where your technical contributions are both visible and highly valued.

This guide is scoped to a Software Engineer candidate at Incident.

Focus areas

• Software Engineering (General)
• Communication Skills
• Behavioral Interview Skills

The Incident interview process

Incident candidates report 3 rounds over 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

Initial review of the candidate's application to assess qualifications.

What to demonstrate

• Initial review of the candidate's application to assess qualifications
• Depth in Software Engineering (General)

How to prepare

• Be able to walk your CV end to end in two minutes, and say why this company specifically.
• Have your salary expectations, notice period and location constraints ready, and ask for the rest of the loop in writing.

Virtual Engagement

A virtual interaction to evaluate the candidate's background and communication skills.

What to demonstrate

• A virtual interaction to evaluate the candidate's background and communication skills
• Depth in Software Engineering (General)

How to prepare

• Answer aloud and timed: Can you describe a time you faced a technical challenge and how you resolved it?
• Answer aloud and timed: How do you handle feedback on your code or design decisions?

Behavioral Assessment

Assessment phase focusing on the candidate's alignment with the company's mission and values.

What to demonstrate

• Assessment phase focusing on the candidate's alignment with the company's mission and values
• Depth in Software Engineering (General)

How to prepare

• Prepare three examples from your own work, each with a decision you made and an outcome you can quantify.
• Re-read the description of the behavioral assessment above and write down what you would ask to confirm before it.

Where candidates lose points

Prioritize clarity

When explaining your technical work, start with the high-level goal before diving into the implementation details.

Ask questions

We love candidates who are curious about our product and our engineering culture; come prepared with thoughtful questions for your interviewers.

Review your resume

Be ready to talk through every project or experience you have listed, as this is often where the most insightful conversations begin.

Do not attempt to memorize scripted answers

Interviewers value genuine, thoughtful responses over rehearsed ones.

Questions & practice

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

1. Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
2. Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
3. Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
4. Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.

Follow-up

• The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
• The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

• The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
• How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

1. Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
2. Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
3. Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
4. Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.

Follow-up

• The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
• How do you verify the backfill actually finished, given rows are still being inserted while it runs?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

1. Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
2. For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
3. Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
4. Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.

Follow-up

• 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
• A dashboard runs this for 40 owners in one page load. What changes about the design?

Rebuild the search projection while it serves nine thousand queries

Medium · projections · eventual consistency · rebuild

The read-model service answers about 9k queries/second at a 120 ms p99 from a projection built off the event log, normally under 2 seconds behind. A mapping change forces a full rebuild from resource_revision, during which apply lag rises to minutes. Writes continue at 1.2k/second and events at 4k/second. Design the rebuild: how the new index is populated and cut over, how position is tracked per log partition, what the API returns alongside results so a client can tell a stale answer from a current one, and the criterion for cutting over.

Approach

1. Build into a second index and swap an alias rather than mutating the live one. The rebuild is then reversible by pointing the alias back, so a bad mapping costs a wasted rebuild instead of an outage. The price is peak storage for two full copies and double apply load during catch-up, and both numbers should be stated up front rather than discovered when disk fills.
2. Track position per log partition, not globally. Order is guaranteed only within an aggregate's partition, so progress is a vector, and the only number safe to publish is taken from the least advanced partition - the slowest one is what bounds completeness. Publishing the most advanced partition's position declares the projection current while another partition sits twenty minutes behind.
3. Make apply idempotent so the backfill and the live tail can overlap without a freeze. Each document records the aggregate_version it reflects, and any event at or below that version is discarded. This is why the event carries the full fact and not a delta: a delta cannot be discarded safely, and a consumer that calls back to read current state applies a state newer than the event it is processing, which is how a projection ends up with changes applied out of order.
4. Turn lag into a contract instead of a surprise. Return the watermark with every result set, and return the version produced by a write so the client can compare the two. A client that wrote version 7 and receives results at a watermark older than its own commit can show that its change is still landing, rather than rendering the previous value as current. Blocking the read until the projection catches up would convert a staleness problem into an availability problem at 9k queries/second, and choosing not to do that is the trade.

Follow-up

• The rebuilt index disagrees with the primary tables for 300 documents. Which is authoritative, and how do you decide without freezing writes?
• The rebuild doubles load on the log and pushes the projection p99 from 120 ms to 400 ms. What do you throttle, and which signal sets how much?

Keep one unresponsive destination from stalling all webhook delivery

Medium · fan-out · circuit breaker · timeouts

Egress delivery sends about 1.5k webhooks/second to 40k destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. One destination begins accepting connections and never responding; within the hour 150 destinations behave the same way. Design the delivery path so unrelated destinations are unaffected: the pool structure, the timeouts, the retry policy, the per-destination circuit, and what is recorded so a retry is not a second effect at the receiver. State how many in-flight slots the degraded destinations hold and why that number decides the design.

Approach

1. Start with the number, and with the law that produces it. In-flight work is arrival rate times time in service, so 1.5k/second against a healthy 200 ms response needs about 300 concurrent slots. Per destination the same product applies, ceilinged by the concurrency cap: at the fleet average of 0.0375 deliveries/second per destination (1.5k spread over 40k) a 10-second timeout is 0.375 slots. A destination that has queued retries behind it is a different regime - every slot refills the instant an attempt expires, so it sits pinned at its cap of 4 - and 150 of those hold 600 slots, more than a pool sized for healthy traffic, entirely consumed by endpoints that will never answer. The per-destination cap bounds one endpoint and says nothing about the aggregate, which is exactly why it alone is not containment.
2. Contain with bulkheads and an admission bound rather than a larger pool. Cap total in-flight per pool and shard destinations across pools by a hash of destination id, so a correlated group - one provider, one region - cannot exceed its pool's share. A delivery refused admission and re-queued with backoff is strictly better than one holding a slot on behalf of a receiver that is not listening.
3. Treat the timeout as two timeouts, and be exact about what shortening one buys. Connect and read are separate failures and both must be shorter than the budget of whatever is waiting. Occupancy is min(cap, arrival rate x timeout), so dropping the read ceiling from 10 seconds to 3 cuts a merely slow destination's occupancy proportionally, 0.375 slots to 0.11 at the fleet average. It does not cut the 4 slots held by one of the 150: a destination with a retry backlog arrives far above cap/timeout - 0.4/second at a 10-second timeout, 1.33/second at 3 - so it stays pinned at the cap either way and only the slot-seconds per attempt fall. What that does buy is detection rate: 3.3x more failures observed per second on the same four slots, which is how fast the circuit reaches its threshold. Pick the value from the measured latency distribution of successful deliveries, with their high percentile as the floor, not from a round number.
4. Add a circuit per destination, counting a timeout as a failure. Once open, fail fast without taking a slot - that is the whole point, converting 4 held slots into zero. Half-open on a schedule with exactly one probe and close only if the probe succeeds, so a permanently dead endpoint costs one request per interval instead of a growing retry queue.

Follow-up

• The destination is not dead - it answers in 9.5 seconds with a 200. Does a failure-rate circuit open? Should anything shed that traffic, and on what signal?
• One destination requires deliveries in order. What does a per-destination concurrency of 4 do to that guarantee, and what would you change to offer it?

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

1. Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
2. Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
3. Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
4. Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.

Follow-up

• The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
• Where should the 10-second timeout number actually come from?

Suggested practice checklist

Built from the rounds and topics Incident candidates report.

Day 1 — Map the Incident loop

• Write out the reported sequence: Application Review, Virtual Engagement, Behavioral Assessment.
• For each round, write one sentence on what it is judging, from the description above, and mark the one you are least ready for.

Deliverable: A one-page map of the 3 reported rounds, with the weakest marked.

Day 2 — Work Software Engineering (General)

• Spend the session on Software Engineering (General), which Incident candidates report being tested on.
• Write one worked example in Software Engineering (General) and time yourself on it.

Deliverable: One timed worked example in Software Engineering (General).

Day 3 — Work Communication Skills

• Spend the session on Communication Skills, which Incident candidates report being tested on.
• Write one worked example in Communication Skills and time yourself on it.

Deliverable: One timed worked example in Communication Skills.

Day 4 — Work Behavioral Interview Skills

• Spend the session on Behavioral Interview Skills, which Incident candidates report being tested on.
• Write one worked example in Behavioral Interview Skills and time yourself on it.

Deliverable: One timed worked example in Behavioral Interview Skills.

Day 5 — Answer out loud: Behavioral and Personal Background

• Answer aloud, timed: Can you tell me about yourself?
• Answer aloud, timed: Why are you interested in a position at Incident?

Deliverable: Spoken answers to 2 reported Behavioral and Personal Background question(s), under time.

Day 6 — Rehearse your own examples

• Prepare three examples from your own work where you made the decision, each with the outcome you can quantify.

Deliverable: Three examples written out, each with a number attached.

Day 7 — Dry run for Incident

• Run one full mock under time, then write down the two questions you most want to ask your interviewers.

Deliverable: A completed timed mock and two questions to ask.

Behavioural preparation

Behavioural rounds judge the decision you made and what it cost.

Can you tell me about yourself?

Medium · Behavioral and Personal Background

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.
4. Name the disagreement and how you resolved it with evidence.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

Why are you interested in a position at Incident?

Medium · Behavioral and Personal Background

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.
4. Name the disagreement and how you resolved it with evidence.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

Can you describe a time you faced a technical challenge and how you resolved it?

Medium · Behavioral and Personal Background

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.
4. Name the disagreement and how you resolved it with evidence.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

How do you handle feedback on your code or design decisions?

Medium · Behavioral and Personal Background

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.
4. Name the disagreement and how you resolved it with evidence.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?