cswork / libraryOpen practice workspace

Interview guides · DESY

DESY Software Engineer Interview Questions & Guide 2026

DESY software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

As a Software Engineer at DESY, you are at the intersection of cutting-edge scientific discovery and high-performance computing. You are not just writing code; you are building the control systems, data acquisition frameworks, and integration tools that power world-class research facilities like PETRA IV. Your work directly enables scientists to conduct complex experiments by ensuring that hardware and software communicate seamlessly in real-time.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

  • Make every write idempotent under retry
  • Trace a symptom to a mechanism under load
  • Detect concurrent edits instead of losing writes

Overview

This role is critical to the mission of DESY because your software acts as the nervous system for advanced scientific equipment. Whether you are working on TANGO Controls or developing interdisciplinary research infrastructure, your contributions directly impact the reliability, efficiency, and throughput of high-energy physics experiments. You will be expected to balance technical rigor with the unique demands of a research-driven environment where precision and performance are paramount.

The DESY interview process

DESY candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

In-depth Technical Interviews

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

  • In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
  • In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
  • Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

  • Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
  • Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
  • For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Panel Interviews

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

  • Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
  • Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
  • Whether you establish what decision is on the table before proposing anything
  • Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

  • Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
  • Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
  • Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Project Discussion

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Saying 'eventually consistent' without naming the anomaly a user would see

Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.

Tests that assert on the implementation rather than the behaviour

Assert on what a caller can observe, not on the number of internal calls or the shape of a private field. A test that breaks on every refactor but still passes when the answer is wrong costs more than it protects.

Questions & practice

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

  1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
  2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
  3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
  4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
  5. Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
  6. Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Worked solution (30 min)

  1. Write the edge-loading query with the tenant predicate on both endpoints and state what it does with a cross-tenant edge.
  2. Implement iterative Tarjan with an explicit stack and confirm on a three-node cycle that it emits one component of size three.
  3. Build the transpose restricted to the visited set and mark every node with an in-edge from outside it as refused, carrying the referrer id.
  4. Run Kahn over the condensation and verify the emitted order against the referrer-before-referenced rule.
  5. Size the CSR arrays for 2,000,000 nodes and 8,000,000 edges and compare against a boxed adjacency map.

Expected result: An iterative O(V+E) traversal over a tenant-scoped CSR subgraph, SCC condensation so cycles archive atomically as one component, a transpose-based refusal list naming the external referrer for each blocked resource, and a Kahn topological order over the condensation, with recursion replaced by an explicit stack because of graph depth rather than style.

Verify it yourself

  • A two-node cycle with no external referrers archives as one unit, and adding a single external referrer moves both nodes to the refusal list.
  • No resource appears in the archive order before something that references it, except within its own component.
  • A counter on edge visits matches the loaded edge count exactly, confirming each edge is traversed once.

Follow-up

  • The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
  • The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
  • An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

  1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
  2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
  3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
  4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
  5. Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
  6. Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Follow-up

  • The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
  • Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
  • How would you run this continuously at low cost instead of only as incident response?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

  1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
  2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
  3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
  4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
  5. Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
  6. Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).

Follow-up

  • The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
  • How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
  • Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

  1. Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
  2. For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
  3. Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
  4. Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
  5. Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
  6. Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.

Worked solution (25 min)

  1. Load 2M resource rows across 5k owners in one tenant, run the query under EXPLAIN (ANALYZE, BUFFERS), and record the node reading the most rows plus the Sort Method line.
  2. Create (tenant_id, owner_user_id, created_at DESC) concurrently and re-run, confirming the Sort node is gone and actual rows fall to about 20.
  3. Run the two negative cases and capture the plan for each.
  4. Re-run the original tenant listing query to confirm the new index has not displaced the index that query depends on.

Expected result: With (tenant_id, owner_user_id, created_at DESC) the plan is an Index Scan with no Sort, roughly 20 rows and a handful of buffers, sub-millisecond. owner_user_id alone still scans; ORDER BY updated_at seeks on the pair and then sorts that owner's rows.

Verify it yourself

  • Actual rows at the leaf node is on the order of the LIMIT, not the tenant's row count.
  • No Sort node appears above the index scan.
  • Buffers read stays constant as one owner's row count grows; if it grows with it, the ORDER BY is not being satisfied by the index.
  • pg_stat_user_indexes shows idx_scan climbing on the new index and not on the listing index for this endpoint.

Follow-up

  • 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
  • A dashboard runs this for 40 owners in one page load. What changes about the design?
  • How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

  1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
  2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
  3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
  4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
  5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
  6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Follow-up

  • A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
  • Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
  • Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How do you approach the challenge of integrating legacy equipment with modern software stacks?

Medium · system design and architecture

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Name the failure you are designing for, then the recovery path.
  3. Choose a partition key and say what query it makes expensive.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What would you drop to keep the system up under load?

Explain your process for documenting and testing critical control software to ensure scientific reproducibility.

Medium · system design and architecture

Approach

  1. Choose a partition key and say what query it makes expensive.
  2. Fix the scope first: who calls this, how often, and what they do when it fails.
  3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What breaks first when traffic grows ten times?

How do you manage the integration of heterogeneous hardware components using Python and C++?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. State your assumptions explicitly before working the problem.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

How do you ensure the maintainability and scalability of codebases that support long-term research infrastructure?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Specify the outbound webhook contract a customer endpoint can verify

Medium · webhooks · hmac · at-least-once · ordering

Egress delivery sends resource.created, resource.updated and resource.archived to roughly 40k customer endpoints at about 1.5k deliveries/second, with a per-destination concurrency cap of 4 and a 10 second connect-plus-read timeout, at-least-once from the relay. The receiver is code you do not control, often behind a proxy that may re-encode JSON. Specify the payload envelope, how the receiver verifies authenticity and rejects replays, what ordering it may assume, how it deduplicates, which status codes it should return and what each means to us, and the retry, circuit and replay behaviour.

Approach

  1. Send the fact, not a pointer: event_id, event_type, aggregate_type, aggregate_id, aggregate_version, occurred_at and the payload copied from outbox_event. A receiver told only that resource 42 changed has to call back, and the call back returns a newer state than the event describes - that is precisely how a consumer applies changes out of order while believing it is following them.
  2. Sign the exact bytes on the wire: HMAC-SHA256 over the timestamp, a separator, and the raw body, in a header, compared in constant time before the body is parsed. Instruct receivers to verify the raw bytes, never a re-serialised copy, since a proxy that reorders keys or changes whitespace invalidates a signature computed over re-encoded JSON. Reject a timestamp outside a tolerance - five minutes is typical - and keep a short seen-set of event_ids, because a capture replayed inside the tolerance still verifies.
  3. Support key rotation by sending several signature values in one header during the overlap window and publishing that any one that verifies is sufficient. Without that rule a key rotation is a synchronised outage across every destination at once.
  4. State the guarantee as it actually is: at-least-once and unordered, including within a single aggregate_id. The outbox is ordered per aggregate at the source, but the relay holds up to 4 requests in flight per destination and backs off each retry independently, so a version 6 delivery that needs one retry lands after a version 7 that succeeded on its first attempt. Publishing per-aggregate ordering we do not enforce is worse than publishing none, because a receiver that trusts it overwrites new state with old. Receivers deduplicate on event_id and apply conditionally - UPDATE ... WHERE aggregate_version < :incoming, or the same comparison under a row lock on the aggregate - which discards stale redeliveries and stays correct when two deliveries for one aggregate are processed at the same time. A duplicate and an out-of-order delivery then have the same outcome: no-op.
  5. Define the response contract in the receiver's terms. 2xx means durably accepted, and accepting means writing it down - a receiver that does its processing inline inside our 10 second timeout will be retried while it is still working. 4xx is permanent and we stop. 429, 5xx and timeouts are retried with capped exponential backoff and full jitter within the per-destination concurrency cap, and a circuit opens after a run of consecutive failures and half-opens on a schedule, so a permanently dead endpoint costs one probe per interval instead of a growing queue.
  6. Give them a recovery path that does not involve us holding a queue for everyone: a visible parked state, and a replay endpoint keyed by event_id or time range, so a receiver that was down for two hours asks for the gap on its own schedule.

Worked solution (30 min)

  1. Write the envelope fields and mark which are needed for deduplication and which for discarding a stale delivery.
  2. Write the signature string construction and the receiver's verification steps in order, including the constant-time compare and the tolerance check.
  3. Write the receiver's idempotent apply: dedupe on event_id, then the version-conditional write that discards stale and concurrently delivered redeliveries.
  4. Write the status-to-action table from our side: which codes retry, which stop, and what opens the circuit.
  5. Describe the parked state and the replay call, including what the customer must supply.

Expected result: A self-contained signed envelope, verification over raw bytes with a replay window and a seen-set, an at-least-once guarantee that states delivery is unordered even within one aggregate, a version-conditional apply, a status table that distinguishes permanent from transient, and a customer-driven replay path.

Verify it yourself

  • Delivering the same event twice leaves the receiver's state identical to delivering it once.
  • Delivering version 7 and then version 6 for one aggregate leaves version 7 applied.
  • A destination that always times out costs one probe per circuit interval rather than a retry queue.

Follow-up

  • A customer insists an event never arrived. What do you check, in order, and which table settles the argument?
  • A receiver cannot store a version and asks us to deliver their events in order. What would the relay have to give up to offer that, and what does one failing event then do to the rest of their stream?

Listing latency scales with page size, not with filters

Easy · n+1 · latency · query batching

The tenant listing endpoint reads resource filtered by tenant_id and status, ordered by updated_at DESC, and returns each row plus the owner's display name from app_user and the actor of that resource's latest resource_revision. p99 is 55 ms at 10 rows per page and 1.4 s at 200. Database telemetry shows 401 statements per request, each under 1 ms, and nothing in the slow-query log. Diagnose the cause and give the fix, stating the statement count per request and the p99 you expect afterwards.

Approach

  1. Read the counters before forming a theory. 401 statements for 200 rows is one driver query plus two per row, and sub-millisecond execution with an empty slow-query log rules out a bad plan. The time is round trips, which is why it is invisible in every per-query metric and scales with rows returned rather than with filter selectivity.
  2. Name the two per-row statements from their normalised text: a single-row app_user lookup by user_id, and a resource_revision lookup by resource_id ordered by version DESC LIMIT 1. Confirm by dropping those two response fields and watching the statement count fall to one. That locates the calls in the serialisation layer, not the repository.
  3. Check that the arithmetic accounts for the whole gap. Measure one round trip to the replica in isolation; 400 trips at roughly 3 ms of network plus 0.2 ms of execution is about 1.3 s on top of a 55 ms baseline, which matches. If the multiplication had fallen short, the N+1 would only be part of the story and you would keep looking.
  4. Batch both lookups. Collect owner_user_ids and resource_ids from the driver query, then issue WHERE tenant_id = $1 AND user_id = ANY($2) for the users, and PostgreSQL's SELECT DISTINCT ON (resource_id) ... WHERE resource_id = ANY($2) ORDER BY resource_id, version DESC for the latest revision, which the UNIQUE (resource_id, version) index serves directly. On an engine without DISTINCT ON, use a lateral join or a row_number window. Three statements per request at any page size.
  5. Keep the tenant predicate in the batched query. The per-row version was implicitly scoped because its ids came from tenant-scoped rows; a batched user_id = ANY(...) with no tenant_id is an unscoped read that behaves correctly only as long as the id list is trustworthy.
  6. Re-measure at 10, 50 and 200 rows and confirm the statement count is constant. Latency should now track bytes returned.

Follow-up

  • The page size is capped at 200 today. What breaks first if it is raised to 2,000, and is it still this bug?
  • How do you stop the next N+1 from reaching production, given that no individual query is slow and the endpoint's tests pass?
  • The latest-revision actor is only used to render an avatar. Make the case for denormalising it onto resource, and name the write anomaly that introduces.

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

When faced with conflicting requirements from scientific stakeholders, how do you prioritize development tasks?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Name the disagreement and how you resolved it with evidence.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Can you explain your experience with TANGO Controls or similar distributed control systems?

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • What did you decide not to do, and why?
  • How did you know your change caused the improvement?

Describe a time you had to design a system that required high availability and fault tolerance.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Close with what you would do differently, concretely.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • How did you know your change caused the improvement?
  • What did you decide not to do, and why?