cswork / libraryOpen practice workspace

Interview guides · American Superconductor NE

American Superconductor NE Software Engineer Interview Questions & Guide 2026

A prep plan for American Superconductor NE software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

A Software Engineer at American Superconductor NE plays a vital role in bridging the gap between advanced engineering research and practical, high-impact industrial solutions. This position is not merely about writing code; it involves contributing to complex systems that power energy infrastructure and power grid resilience. You will be expected to translate technical requirements into robust software solutions, often working in environments where precision and reliability are paramount.

The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.

Focus areas

  • Bound every outbound call with a timeout
  • Choose indexes from the query's access path
  • Trace a symptom to a mechanism under load

Overview

The work is intellectually demanding and offers the opportunity to collaborate with teams deeply invested in cutting-edge technology. Whether you are working on data-driven research or system-level applications, your contributions directly impact how American Superconductor NE delivers value to its clients. You will be part of a culture that values academic rigor, practical problem-solving, and the ability to articulate complex technical findings to both peers and leadership.

The American Superconductor NE interview process

American Superconductor NE candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Manager-Led Technical Interviews

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

  • Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
  • Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
  • Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
  • Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

  • From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
  • Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
  • Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Choosing an index from the columns a query mentions rather than from how it filters and orders

A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Saying 'eventually consistent' without naming the anomaly a user would see

Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.

Questions & practice

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

  1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
  2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
  3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
  4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
  5. Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
  6. Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Follow-up

  • The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
  • The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
  • An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

  1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
  2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
  3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
  4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
  5. Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
  6. Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).

Follow-up

  • The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
  • How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
  • Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

  1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
  2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
  3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
  4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
  5. Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
  6. Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Worked solution (35 min)

  1. Compute the naive cost explicitly at 40,000,000 reads and 0.5 ms each, then at 100 concurrent, and state what those connections do to a pool already carrying 1,200 writes per second.
  2. Write the merge-join version over (tenant_id, resource_id) and state its memory.
  3. Define the range aggregate: the range key, the per-row hash input, and the combining function, with one sentence excluding XOR.
  4. Work an example with 40,000,000 rows, branching factor 256 and 5 differing rows, and count the ranges examined.
  5. Add the watermark filter and the resume point, and name the throttle signal the loop watches.

Expected result: A rejection of per-row point reads backed by the time cost and the cache-eviction argument, a single ordered merge join as the dense-case answer at O(n) time and O(1) memory, a range-hash drill-down examining O(d log_B(n/d)) ranges using a sum or multiset hash rather than XOR, a watermark excluding recently written rows, and a resumable throttled run loop.

Verify it yourself

  • Injecting one known stale row makes exactly one leaf range mismatch and the drill-down reaches it.
  • Duplicating a row on one side is detected, which is precisely the case XOR would miss.
  • Running the diff twice against an idle system reports zero mismatches both times.

Follow-up

  • The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
  • Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
  • How would you run this continuously at low cost instead of only as incident response?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

  1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
  2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
  3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
  4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
  5. Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
  6. Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Follow-up

  • The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
  • A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
  • Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

  1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
  2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
  3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
  4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
  5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
  6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Worked solution (25 min)

  1. Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
  2. Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
  3. Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
  4. Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
  5. Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.

Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.

Verify it yourself

  • COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
  • Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
  • Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.

Follow-up

  • A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
  • Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
  • Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How do you balance individual research goals with the needs of the wider organization?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the requirement backwards to the design.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How do your technical skills align with the requirements of this role?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. State your assumptions explicitly before working the problem.
  3. Work from the requirement backwards to the design.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How do you stay updated with the latest technological advancements in your domain?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Say what you would check first and why it is the highest-information step.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Make resource creation safe for a client that retried a timeout

Medium · idempotency · retries · race conditions

POST /v1/resources creates a resource row and, in the same transaction, a resource_revision and an outbox_event. A partner's HTTP client times out at 2 seconds and retries twice; under load the first request is still executing when the retry arrives, and duplicate resources are appearing. Using idempotency_key (tenant_id, idempotency_key, request_fingerprint, state, response_status, response_body, resource_id, locked_until, expires_at), specify the whole contract: first call, concurrent retry, retry after completion, a key reused with a different body, and what the client does with each response.

Approach

  1. Name why the plain REST shape fails here. POST is not idempotent, a timeout leaves the client unable to distinguish a lost request from a lost response, so it must retry - and the retry is a second create. Checking whether the key exists and then inserting it is check-then-act: under exactly the concurrency that caused the timeout, both attempts look, both see nothing, both proceed.
  2. Let the unique constraint arbitrate instead. INSERT the key row with state 'in_flight' and locked_until set to now plus the request deadline, and commit that reservation on its own before any work is attempted - an uncommitted row is invisible to the concurrent retry, which is the entire point of writing it first. One caller wins the insert and proceeds; the other takes the unique violation and reads the row rather than falling through to the work.
  3. Branch the loser on what it found. 'succeeded' replays response_status and response_body verbatim, with a header marking it a replay. 'in_flight' with locked_until in the future returns 409 and a Retry-After - do not park a request thread waiting on another request, that is how a concurrency bound is consumed by retries. 'in_flight' past locked_until is reclaimable by a conditional UPDATE predicated on the old locked_until, so exactly one reclaimer wins. A fingerprint mismatch is 422 and never the stored response: a different body under the same key is a client defect, and serving the old response hides it forever.
  4. Commit the effect and the key's completion together - resource insert, revision row, outbox row, and the UPDATE of the reserved key to 'succeeded' with the stored response, in one transaction. That makes a crash binary: either all of it is durable and the key reads 'succeeded', or the transaction rolls back and the only durable trace is the reservation row, still 'in_flight', with no resource behind it. The reclaim branch finishes that case - once locked_until passes, one retry takes the row by conditional UPDATE and runs the work for the first time. Splitting the two commits is what creates the window with no recovery: the effect durable, the key still 'in_flight', and a reclaimer that cannot distinguish that from work never started, so it creates a second resource.
  5. State the lifetime explicitly. Keys expire - 24 hours is a common choice - and are swept, so the client's total retry window must be shorter than the TTL, or a late retry re-executes with no record that it is a duplicate. The key is scoped by tenant so two tenants choosing the same string do not collide, and the client generates one key per logical attempt, not per network send.
  6. Write the client rules: on timeout, retry the same key with capped backoff; on 409, wait Retry-After and retry the same key; on 422, stop and fix the caller; on 2xx, treat the body as authoritative whether it was executed or replayed.

Worked solution (30 min)

  1. Write the two-request interleaving that defeats read-then-insert, with the line each request is executing at each step.
  2. Write the insert-first sequence and the four branches taken on unique violation, as a table of state, locked_until and response.
  3. Put the completion UPDATE inside the same transaction as the resource, revision and outbox writes, then state what a crash before that commit leaves durable and which branch cleans it up.
  4. Define the key's scope, lifetime and generation rule, then bound the client's retry budget by the TTL.
  5. Fire two concurrent requests carrying one key and count rows in resource and resource_revision.

Expected result: One resource row, one revision row and one outbox row; the second caller either replays the first response or receives 409 with Retry-After; a different body under the same key is rejected 422; no request thread blocks waiting on another request.

Verify it yourself

  • A hundred runs of two simultaneous same-key requests produce exactly one resource and one revision every time.
  • Killing the process mid-work leaves zero resource rows and the reservation still 'in_flight'; after locked_until passes a retry reclaims the key and executes, so the crash and the retry together produce exactly one resource row.
  • The documented client retry budget is strictly shorter than expires_at minus created_at.

Follow-up

  • Response bodies are pruned at 24 hours but resource_id is kept. What do you return to a replay that arrives after pruning?
  • The endpoint now also sends a welcome email. Which part of this contract prevents a second send, and which part cannot?
  • What exactly breaks if the key is scoped globally rather than per tenant?

One log partition stops advancing while the others drain

Easy · poison message · head-of-line blocking · consumer lag · projections

Search results for a subset of tenants are hours stale; the rest are current. The projection consumer reports lag of zero on 15 of 16 partitions and 400,000 on one. Its error rate is flat and its CPU is idle. outbox_event has no pending rows older than a second, so the relay has published everything it holds. Identify the mechanism, give the ordered checks, and state what you do in the first ten minutes versus what you change permanently.

Approach

  1. Read the lag distribution first. A slow consumer lags everywhere; zero on fifteen partitions and 400,000 on one is not throughput. Idle CPU on the stuck partition means the consumer is not advancing its offset at all, which points at one message it cannot get past rather than at a rate problem.
  2. Exonerate the producer before touching the consumer. No pending outbox rows older than a second means the relay published, so the event exists in the log. This separates never sent from sent and never applied, which are different code paths and usually different owners.
  3. Read the message at the stuck offset and the handler's log lines for its event_id. A flat error rate with no progress has two explanations and you must distinguish them: the handler is throwing and the retry loop is swallowing it, or the handler is blocking on something and never returning. Idle CPU with no error lines favours the second.
  4. Mitigate before diagnosing further. Move the offending event to a dead-letter store and commit the offset past it. Adding consumers does nothing here, because a partition is consumed by exactly one member of the group, and the blast radius is every aggregate hashed to that partition, not only the aggregate that produced the bad event.
  5. Fix permanently by bounding handler attempts and dead-lettering on exhaustion, so no single message can stop a partition. Then replay the dead-lettered event once the handler is fixed: it carries aggregate_id and aggregate_version, so a consumer that discards versions it has already applied can absorb the replay, and resource_revision is the fallback if the event itself is unusable.

Follow-up

  • The dead-lettered event carried aggregate_version 7 and the projection had applied 6. What must the replay do differently if 8 and 9 landed in the meantime?
  • How do you show staleness to the user while the partition is behind, given the API already returns the projection's watermark?
  • What changes if the message is poison because a previous deploy wrote a payload shape the current code cannot parse?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

  • Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
  • Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
  • Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

  • Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
  • Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
  • Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

  • Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
  • Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
  • Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

  • For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
  • Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
  • Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
  • Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

  • Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
  • Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
  • Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
  • Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

  • Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
  • Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
  • Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

  • Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
  • Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
  • Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

How do you handle disagreements on technical direction within a project?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Close with what you would do differently, concretely.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

  1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
  2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
  3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
  4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
  5. Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

  • What would you do differently in the first five minutes, given the same dashboard and no more information?
  • Which follow-up action did you deliberately not take, and why was dropping it the right call?
  • How did you convince yourself the mitigation was safe to apply while the cause was still unknown?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

  1. State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
  2. Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
  3. Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
  4. Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
  5. Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

  • What in that decision was irreversible, and did you know it was irreversible when you made it?
  • How did you tell the people who had already built on top of the original decision?
  • What do you now measure before committing to a change of this size?