Interview guides · Early Warning Services
Early Warning Services Software Engineer Interview Questions & Guide 2026
Early Warning Services software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.
The 60-second brief
As a Software Engineer at Early Warning Services, you are at the heart of the infrastructure that powers real-time payment solutions, most notably Zelle. This role is critical to maintaining the trust and reliability of financial transactions for millions of users. You will contribute to high-scale, secure, and performant systems that require a deep understanding of distributed architectures and robust backend engineering.
If the loop includes an asynchronous take-home, treat it as a code review of you rather than as a puzzle: structure, what you chose to test, and what you wrote down about the constraint you were working under. Hold the stated time box and say what you would have done with more of it.
Focus areas
- Detect concurrent edits instead of losing writes
- Trace a symptom to a mechanism under load
- Bound every outbound call with a timeout
Overview
The work environment at Early Warning Services is centered on navigating the intersection of traditional banking stability and modern, high-concurrency software demands. You will likely work on complex event-driven systems, API integrations, and database modeling. Success in this role requires a balance of technical rigor and the ability to solve practical, real-world problems in a regulated financial environment.
This position is ideal for engineers who thrive in environments where system reliability, security, and data integrity are paramount. While the scope is large, the day-to-day work is often focused on specific, high-impact modules that ensure the seamless operation of digital payment ecosystems.
The Early Warning Services interview process
Early Warning Services candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.
Recruiter Contact
The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.
What to demonstrate
- Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
- Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
- Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year
How to prepare
- Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
- Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
- Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter
Remote Screening
Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.
What to demonstrate
- Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
- Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
- Whether you can be redirected in the middle of an answer without restarting it from the beginning
- Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it
How to prepare
- Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
- Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
- Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing
Technical Deep Dive
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Behavioral Assessment
Many of these questions are about something that went wrong, and the grading sits mostly in the hours after you knew. Who found out first, whether that was you or an alert or a user, how long it took you to say it out loud, and whether the people who needed the news got it while they could still act on it. Engineers under-tell this part because it feels like confessing. The pattern it is looking for is the opposite: the quiet fix, an incident absorbed without telling anyone, after which nothing changed and the same failure is still available.
What to demonstrate
- How the problem was found, and whether that route was one you had built or one that happened to you, since a user reporting it first means your instrumentation did not cover that failure
- Whether time-to-detect and time-to-tell are separate numbers in your account and whether you know both, because a fast fix that nobody heard about until the retro is a different answer from a slow one that was announced immediately
- Whether the resolution left something durable behind, a check that fires or a default that changed, rather than depending on people remembering to be careful
- Whether you can say what the failure cost without either inflating it or waving it away
How to prepare
- Reconstruct one incident you were part of as a timeline with clock times: first bad request, first signal, first person who knew, first message outside the team, mitigation, permanent fix. The gaps between those entries are what gets asked about
- Look up the configuration of the signal that caught it, including its evaluation window and threshold. An alert defined on a five-minute aggregate cannot fire until the condition holds across that window, which puts a floor under time-to-detect that has nothing to do with how severe the failure was. Be able to say what that floor was and whether anyone had chosen it deliberately
- Prepare one story where you escalated early and the severity turned out to be smaller than you thought, including what it cost the people you pulled in. Without it, every answer you give about raising alarms is unfalsifiable
Final Team Interviews
Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.
What to demonstrate
- Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
- Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
- Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
- Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience
How to prepare
- Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
- For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
- Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed
Where candidates lose points
Choosing an index from the columns a query mentions rather than from how it filters and orders
A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.
Paginating with LIMIT/OFFSET over a set that changes while the client is reading it
OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.
Issuing one query per row of a result set
Fetch related rows in a single batched query keyed by the ids you already hold, or join them into the original query. A per-row round trip multiplies network latency by the row count, and it looks perfectly fine against the ten rows in your development database.
Assuming fixed-width integer arithmetic cannot overflow
In languages with fixed-width integers, including C, C++, Java, Go and Rust, computing a midpoint as (lo + hi) / 2 overflows once the sum passes the type's maximum, so write lo + (hi - lo) / 2 instead. Say which language you are in: arbitrary-precision integers, as in Python or Ruby, remove this specific hazard and none of the others.
Questions & practice
Archive a resource graph without breaking live references or recursing
Medium · graph traversal · topological order · tenant isolation
Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.
Approach
- Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
- Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
- Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
- Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
- Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
- Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.
Follow-up
- The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
- The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
- An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?
Find overlapping job attempts and peak concurrency from lease records
Medium · sweep line · intervals · leases
A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.
Approach
- Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
- For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
- For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
- Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
- State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
- Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.
Follow-up
- A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
- Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
- How would you compute both answers incrementally as records arrive rather than in a daily batch?
Track a rolling failure rate per destination for circuit decisions
Easy · sliding window · ring buffer · circuit breaker
The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.
Approach
- Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
- Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
- State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
- Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
- Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
- Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.
Worked solution (20 min)
- Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
- Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
- Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
- Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.
Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.
Verify it yourself
- After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
- Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
- Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.
Follow-up
- The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
- A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
- How would you make the window survive a process restart, and is it worth the cost?
Find version gaps and relay lag with window functions
Medium · window functions · event tables · frame semantics
outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.
Approach
- Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
- Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
- Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
- Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
- Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
- Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.
Follow-up
- Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
- Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
- The consumer claims it never received event 4,812,006. What do you look at, in what order?
Denormalise tenant onto revisions and backfill it live
Hard · denormalisation · online migration · backfill
resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.
Approach
- Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
- Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
- Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
- Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
- Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
- Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.
Worked solution (40 min)
- Write the five steps as separate scripts and state, for each, the lock mode it acquires and the deploy it pairs with.
- On a 20M-row copy, run the ADD COLUMN while a 30-second transaction holds a lock on the table, and record how long unrelated queries queue behind it.
- Run the batched backfill at 5,000 rows, kill it mid-run, restart from the persisted cursor, and confirm no row is processed twice and none is skipped.
- Build the index concurrently under concurrent write load, then add the CHECK ... NOT VALID, VALIDATE it and SET NOT NULL, timing each.
- Compare the audit-feed plan before and after: join-and-filter versus an index seek with no Sort.
Expected result: ADD COLUMN completes in milliseconds once it holds the lock, but with a 30-second transaction open it stalls all traffic on that table for the full 30 seconds unless lock_timeout cuts it off. The backfill resumes exactly at the persisted revision_id. After VALIDATE, SET NOT NULL returns without a table scan. The audit feed drops from scanning the tenant's history to reading about 50 rows.
Verify it yourself
- SELECT count(*) FROM resource_revision WHERE tenant_id IS NULL returns 0, re-checked after the dual-writer has been live for a full deploy cycle.
- A sampled join confirms rr.tenant_id = r.tenant_id across 100k random revisions; the composite foreign key makes a mismatch unwritable from then on.
- pg_stat_activity shows no transaction older than one batch's duration while the backfill runs.
- EXPLAIN of the audit feed shows an Index Scan with no Sort node.
Follow-up
- The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
- How do you verify the backfill actually finished, given rows are still being inserted while it runs?
- A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?
How would you design a system for distributed processing with high throughput?
Medium · system design and architecture
Approach
- Name the read and write paths separately; they rarely have the same bottleneck.
- Name the failure you are designing for, then the recovery path.
- Fix the scope first: who calls this, how often, and what they do when it fails.
Follow-up
- What would you drop to keep the system up under load?
- How does this behave when that dependency is down for an hour?
Design a simplified version of a high-traffic service (e.g., a ticket booking system).
Medium · system design and architecture
Approach
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
- Choose a partition key and say what query it makes expensive.
Follow-up
- How does this behave when that dependency is down for an hour?
- What would you drop to keep the system up under load?
How do you manage data modeling for complex, scalable services?
Medium · system design and architecture
Approach
- Name the failure you are designing for, then the recovery path.
- State the consistency you need, and where you are willing to be stale.
- Choose a partition key and say what query it makes expensive.
Follow-up
- How does this behave when that dependency is down for an hour?
- What would you drop to keep the system up under load?
What are the common design patterns used in Spring MVC architecture?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
What are the best practices for implementing and securing REST APIs?
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- State your assumptions explicitly before working the problem.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Make resource creation safe for a client that retried a timeout
Medium · idempotency · retries · race conditions
POST /v1/resources creates a resource row and, in the same transaction, a resource_revision and an outbox_event. A partner's HTTP client times out at 2 seconds and retries twice; under load the first request is still executing when the retry arrives, and duplicate resources are appearing. Using idempotency_key (tenant_id, idempotency_key, request_fingerprint, state, response_status, response_body, resource_id, locked_until, expires_at), specify the whole contract: first call, concurrent retry, retry after completion, a key reused with a different body, and what the client does with each response.
Approach
- Name why the plain REST shape fails here. POST is not idempotent, a timeout leaves the client unable to distinguish a lost request from a lost response, so it must retry - and the retry is a second create. Checking whether the key exists and then inserting it is check-then-act: under exactly the concurrency that caused the timeout, both attempts look, both see nothing, both proceed.
- Let the unique constraint arbitrate instead. INSERT the key row with state 'in_flight' and locked_until set to now plus the request deadline, and commit that reservation on its own before any work is attempted - an uncommitted row is invisible to the concurrent retry, which is the entire point of writing it first. One caller wins the insert and proceeds; the other takes the unique violation and reads the row rather than falling through to the work.
- Branch the loser on what it found. 'succeeded' replays response_status and response_body verbatim, with a header marking it a replay. 'in_flight' with locked_until in the future returns 409 and a Retry-After - do not park a request thread waiting on another request, that is how a concurrency bound is consumed by retries. 'in_flight' past locked_until is reclaimable by a conditional UPDATE predicated on the old locked_until, so exactly one reclaimer wins. A fingerprint mismatch is 422 and never the stored response: a different body under the same key is a client defect, and serving the old response hides it forever.
- Commit the effect and the key's completion together - resource insert, revision row, outbox row, and the UPDATE of the reserved key to 'succeeded' with the stored response, in one transaction. That makes a crash binary: either all of it is durable and the key reads 'succeeded', or the transaction rolls back and the only durable trace is the reservation row, still 'in_flight', with no resource behind it. The reclaim branch finishes that case - once locked_until passes, one retry takes the row by conditional UPDATE and runs the work for the first time. Splitting the two commits is what creates the window with no recovery: the effect durable, the key still 'in_flight', and a reclaimer that cannot distinguish that from work never started, so it creates a second resource.
- State the lifetime explicitly. Keys expire - 24 hours is a common choice - and are swept, so the client's total retry window must be shorter than the TTL, or a late retry re-executes with no record that it is a duplicate. The key is scoped by tenant so two tenants choosing the same string do not collide, and the client generates one key per logical attempt, not per network send.
- Write the client rules: on timeout, retry the same key with capped backoff; on 409, wait Retry-After and retry the same key; on 422, stop and fix the caller; on 2xx, treat the body as authoritative whether it was executed or replayed.
Worked solution (30 min)
- Write the two-request interleaving that defeats read-then-insert, with the line each request is executing at each step.
- Write the insert-first sequence and the four branches taken on unique violation, as a table of state, locked_until and response.
- Put the completion UPDATE inside the same transaction as the resource, revision and outbox writes, then state what a crash before that commit leaves durable and which branch cleans it up.
- Define the key's scope, lifetime and generation rule, then bound the client's retry budget by the TTL.
- Fire two concurrent requests carrying one key and count rows in resource and resource_revision.
Expected result: One resource row, one revision row and one outbox row; the second caller either replays the first response or receives 409 with Retry-After; a different body under the same key is rejected 422; no request thread blocks waiting on another request.
Verify it yourself
- A hundred runs of two simultaneous same-key requests produce exactly one resource and one revision every time.
- Killing the process mid-work leaves zero resource rows and the reservation still 'in_flight'; after locked_until passes a retry reclaims the key and executes, so the crash and the retry together produce exactly one resource row.
- The documented client retry budget is strictly shorter than expires_at minus created_at.
Follow-up
- Response bodies are pruned at 24 hours but resource_id is kept. What do you return to a replay that arrives after pruning?
- The endpoint now also sends a welcome email. Which part of this contract prevents a second send, and which part cannot?
- What exactly breaks if the key is scoped globally rather than per tenant?
Exports duplicate a row range about once a week
Hard · leases · clock skew · idempotency · fencing
Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.
Approach
- Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
- Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
- Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
- Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
- Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
- Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.
Follow-up
- The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
- You now count takeovers. What alert fires on that counter, and at what threshold?
- What breaks if you simply raise the lease to five minutes?
Suggested practice checklist
Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.
Day 1 — Diagnostic, scored before you study anything
- Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
- Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
- Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.
Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.
Day 2 — Largest gap: find the boundary rather than the subject
- Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
- Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
- Re-attempt one of them from blank four hours later with nothing open.
Deliverable: A sub-skill map with the two blocking sub-skills circled.
Day 3 — Drill the blocking sub-skill by repeating the shape
- Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
- State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
- Have someone else read your one-sentence rule and find the precondition you left out.
Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.
Day 4 — Second gap, plus maintenance on the strongest area
- Run the same sub-skill decomposition on the second-largest gap in half the time.
- Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
- Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.
Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.
Day 5 — The gap that is not a skill
- Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
- Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
- Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.
Deliverable: Two recordings with a counted reduction in time-to-first-question.
Day 6 — Retest under day-one conditions
- Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
- For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
- Write down which single block you would still lose the offer on.
Deliverable: A second scored rubric placed beside the first, with one named remaining risk.
Day 7 — Full loop under interview conditions
- Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
- Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
- Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.
Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.
Bounding a vague ticket before writing code
When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.
Describe a challenging project you worked on and the role you played in its success.
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
- Close with what you would do differently, concretely.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Tell me about a time you had a conflict with a co-worker and how you resolved it.
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Give the blast radius: what could have broken, and what you measured.
- State the situation in two sentences and spend the rest on the reasoning.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?
How do you feel about documentation and its role in a team environment?
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
Follow-up
- How did you know your change caused the improvement?
- What did you decide not to do, and why?