Almac Group Software Engineer Interview Questions & Guide 2026

The 60-second brief

As a Software Engineer at Almac Group, you play a critical role in developing and maintaining the technological infrastructure that supports a global leader in the pharmaceutical and biotech industries. Your work directly impacts the efficiency and reliability of systems that facilitate drug development, clinical trials, and diagnostic services. Because Almac Group operates in a highly regulated, high-stakes environment, the software you build must be robust, scalable, and meticulously tested to meet stringent quality and compliance standards.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

• Make HL7 and X12 ingestion idempotent under replay
• Authorise every identified read against current consent
• Hold accumulator limits under concurrent claim adjudication

Overview

This role is intellectually demanding and offers significant opportunities to work on complex, mission-critical projects. You will collaborate with cross-functional teams, including product managers, quality assurance engineers, and domain experts, to translate business requirements into high-quality code. Whether you are working on internal tools or client-facing applications, your contribution ensures that Almac Group remains at the forefront of scientific innovation. Candidates should expect a professional environment that values precision, technical depth, and a collaborative spirit.

The Almac Group interview process

Almac Group candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Behavioral Screening

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

• Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
• Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
• Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

• Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
• Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
• For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Technical Assessment

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

• In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
• In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
• Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

• Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
• Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
• For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Final Interviews

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

• Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
• Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
• Whether you establish what decision is on the table before proposing anything
• Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

• Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
• Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
• Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Writing the access audit record inside the read transaction, or firing it off after the response with no durability.

Inside the transaction, an audit-store outage blocks clinical reads and turns a logging dependency into a care outage. Fire-and-forget afterwards means the audit trail is incomplete during precisely the incidents it exists to reconstruct, and the gaps are invisible until someone asks for the log. The usual resolution is committing the access decision and its audit row together to a local outbox and shipping asynchronously, which keeps the read path available while preserving durability.

Upserting on the order or result identifier, so a correction overwrites the original row.

It makes the question 'what did the clinician see at 14:02' unanswerable, which is exactly what an incident review or a legal hold asks. It also leaves downstream consumers that already acted on the preliminary value with no correction event to react to, because the state transition was collapsed into a single mutated row and never emitted.

Reading the constraints as preamble rather than as part of the problem

The bounds are usually there to eliminate the obvious approach: n up to 10^5 makes an O(n^2) scan roughly 10^10 operations, far outside any per-test time budget, and an input larger than memory rules out loading it at all. When a bound is not given, ask for it, then say out loud which approach it kills.

Assuming the bug is in the framework

Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.

Questions & practice

Sum surviving claim versions in one pass over unordered lines

Easy · hash map · versioning · streaming aggregation

You are streamed up to 50 million claim_line records in arbitrary order: claim_id, claim_version, line_number, frequency_code (original, replacement, void), enterprise_person_id, allowed_amount_cents. Every version of a claim shares its claim_id, and a replacement arrives as a higher claim_version. Return total allowed_amount_cents per enterprise_person_id, counting each claim once at its highest version and contributing zero when that version is a void. Amounts are non-negative int64 minor units. Target O(N) time and O(C) space for C distinct claims, single pass, no sort.

Approach

1. Say the two grains out loud before writing anything: the version lives at claim grain, the money lives at line grain. Every wrong answer here comes from applying a claim-level rule with a line-level filter.
2. Keep one hash map from claim_id to a small record (best_version, sum_cents, person_id, is_void). Per line: if claim_version is greater than best_version, reset sum_cents to this line's amount and overwrite the person and void flag; if equal, add to it; if lower, discard the line. That reset is what makes the pass order-independent, so a replacement arriving before its original still wins.
3. Resolve the void at the end, not at ingest. A void only nullifies the claim if it is the surviving version, and zeroing on sight would let an earlier replacement that arrives later resurrect the money.
4. Fold the C claim records into a person-keyed map in a second phase, O(C). Do not accumulate into the person total during the stream: you cannot subtract a superseded version you have already forgotten.
5. Cost is O(N) time, O(C) space. The sort-based alternative, group by (claim_id, claim_version) and keep the max, is O(N log N) and needs the set resident; prefer it only when C approaches N and the map will not fit.
6. Keep cents in int64 throughout. Fifty million lines at a realistic per-line ceiling stay four orders of magnitude below 2^63, so the integer type costs nothing and removes the float drift class entirely.

Worked solution (20 min)

1. Define the per-claim record and the three-way comparison on claim_version: greater resets, equal accumulates, lower discards.
2. Run the stream once, touching only the claim map.
3. Walk the claim map, skipping entries whose surviving version is a void, and add each remaining sum into a person map.
4. Return the person map sorted descending by total.

Expected result: Claim A: version 1 has lines of 12000 and 8000 cents, version 2 (replacement) has lines of 9000, 8000 and 500. Claim B: version 1 has a single line of 5000, version 2 is a void carrying one zero-amount line. Both claims belong to one person. Your answer is 17500 cents, from claim A's version 2 only. The naive sum over all seven lines is 42500.

Verify it yourself

• The number of claims contributing equals the number of distinct claim_ids whose highest version is not a void.
• For a multi-line claim, the addends in its sum equal that surviving version's line count, not one.
• Shuffling the input and re-running produces an identical result, which is the property the reset-on-higher-version rule buys you.
• The all-rows sum is greater than or equal to your total, equal only when no claim has a version above 1 and none is a void.

Follow-up

• You shard the stream across eight workers. Sharding by hash of claim_id works; what exactly breaks if you shard by enterprise_person_id instead?
• A replacement arrives for a claim_id whose original never appears in the batch. What does your map produce, and is that the financially correct answer or a reconciliation break?
• The same claim_id is reused by two different submitting organisations. How does the key have to change, and when would you have noticed?

Resolve enterprise identity from a link log that supports unmerge

Hard · union-find · rollback · identity resolution

person_identity_link holds link_id, enterprise_person_id, assigning_authority, source_person_id, match_score, link_status (auto_linked, manual_linked, potential_duplicate, unlinked, rejected), version, superseded_by_link_id, decided_by, decided_at. You have up to 40 million source identities and 60 million decisions applied in decided_at order. Build a structure answering which enterprise identity a given (assigning_authority, source_person_id) resolves to after any prefix of the log, and supporting a revert of the k most recent merges at O(1) each. State the query complexity, and say why path compression is unavailable to you.

Approach

1. Intern the node key as the pair (assigning_authority, source_person_id) into a dense integer index. Never key on source_person_id alone: two facilities in one network routinely issue the same medical record number to different people, so a bare-value key merges two patients before any matching logic has run, and a single-source test fixture will never show it.
2. Classify the log rows before touching the structure. auto_linked and manual_linked are unions; potential_duplicate and rejected are recorded decisions that must not union anything; unlinked is a revert of the link it supersedes, not a new edge.
3. Use union by size with an explicit undo stack, pushing (attached_root, its previous parent, the previous size of the absorbing root) on every union. Find walks parent pointers to the root: union by size bounds tree height at log2(n), so a query is O(log n) worst case, a union is O(log n), and a revert pops two words and is O(1).
4. Path compression is the thing you have to give up, and the reason is specific: it rewrites parent pointers of nodes that were never named in the union being recorded, so the undo entry no longer describes the mutation that happened and a revert restores a forest that is quietly wrong. Near-constant find is only available to an append-only structure.
5. Hold enterprise_person_id as a property of the root slot rather than a value copied onto every member. A merge then relabels one slot instead of n rows, and an unmerge restores two labels instead of reconstructing n.
6. State the price plainly: every read path gains a resolution hop and no downstream table can carry a plain patient_id column. That cost is what buys reversibility, and it belongs in the design discussion rather than being discovered later by whoever writes the first join.

Follow-up

• A merge is found wrong three weeks and 200,000 unions later, and your stack only reverts a suffix. What do you do instead, and what does that cost?
• A claim was posted under the losing identity before the merge. After the unmerge, which identity owns it, and what in your design let you answer that?
• Two matcher instances submit unions concurrently. What is the smallest change that keeps both the structure and the undo stack consistent?

Reconstruct what the chart displayed at five million past instants

Hard · external sort · as-of join · io bound

observation_result holds 200 million versions: observation_id, enterprise_person_id, filler_order_id, loinc_code, value_numeric, result_status, collected_ts, issued_ts, version, supersedes_observation_id. An incident review hands you 5 million queries of (enterprise_person_id, filler_order_id, loinc_code, as_of_instant). For each, return the version that was visible at that instant, meaning the one with the greatest issued_ts at or before it. Neither side fits in memory. The per-query scan is correct. Explain precisely why it is too slow, then give a plan with its complexity.

Approach

1. Name the clock before naming an algorithm. Visibility is issued_ts, the release time. collected_ts is when the specimen was drawn and can precede release by hours, so ordering on it reports a correction as visible long before anyone could have seen it. No amount of index work rescues the wrong column.
2. Be exact about why the naive plan fails, because the interviewer is testing whether you can tell arithmetic cost from I/O cost. Per query the chain scan is O(V_k) and the arithmetic is trivial, but 5 million independent lookups into a 200-million-row structure that does not fit in RAM is 5 million random reads. The job is bounded by seeks per query, not by comparisons, and buying a faster comparison changes nothing.
3. Convert random access into sequential access. Hash-partition both sides on the chain key (enterprise_person_id, filler_order_id, loinc_code) into P shards sized to fit memory, sort each shard's versions by (chain key, issued_ts) and its queries by (chain key, as_of_instant), and sweep the pair in lockstep. Total O((V + Q) log(V + Q)) with external sort, replacing Q seeks with two sequential passes.
4. Inside a chain the sweep is linear, not logarithmic, because queries are visited in ascending as_of order and the version pointer only moves forward: O(V_k + Q_k) per chain. Binary search per query is the better shape only when Q is small relative to V and the versions are already indexed and resident.
5. Return the empty answer as a distinct outcome. A query whose as_of precedes the first issued_ts means nothing was displayed, which is not the same as the earliest value, and is frequently the exact fact the review is chasing.
6. Return a version later marked entered_in_error if it was live at the instant asked about. Reconstructing the past means reporting what was on the screen, including what was wrong, and quietly substituting today's truth defeats the purpose of the exercise.

Follow-up

• Read replicas lagged 40 seconds at the time. Does your answer describe what the clinician actually saw, and how would you bound the difference?
• Serve the same question online for a single chart at a p99 under 50ms. What changes?
• One partner changed its filler_order_id format mid-year, so the chain key is not stable. How does that appear in your output, and how do you detect it rather than returning empty answers?

Denormalise consent into a decision table with a stated staleness bound

Hard · denormalisation · cache invalidation · access control

Access governance answers 'may this requester read this person's data for this purpose' on every identified read — tens of thousands per second at a 5ms p99 — from consent_directive: consent_id, enterprise_person_id, version, scope, purpose_of_use text[], grantee_org_id (NULL meaning all organisations), permit boolean, effective_ts, expires_ts, revoked_ts, status. Evaluating the normalised version history per read misses the budget. Design the denormalised decision store and its key, state the revocation staleness bound in seconds and the mechanism that actually enforces it, and say which normalised rows remain the system of record and why.

Approach

1. Derive the shape from the read, not from the source table. The read is a point lookup on (person, grantee organisation, purpose), so fan the purpose_of_use array out into one row per purpose and make those three columns the primary key. A GIN index on the array would serve containment queries nobody issues.
2. Handle the wildcard concretely. grantee_org_id NULL means 'all organisations', and NULL never equals anything, so a nullable column in the key breaks both the primary key and the equality lookup. Store a sentinel org id of 0 for the wildcard and resolve specific-before-wildcard in the lookup, or the wildcard row is silently unreachable.
3. Make refusal a row, not an absence. An explicit permit = false must outrank a permit for the same key, so carry a precedence rank (specific grantee beats wildcard, refusal beats permit at equal specificity) and resolve with ORDER BY rank LIMIT 1. Default deny when nothing matches, so a rebuild failure fails closed.
4. Separate the two staleness mechanisms and be precise about which one is the bound. The in-process decision cache has a TTL; a revocation also publishes an invalidation. The publish shortens the typical case to milliseconds, but it can be lost, so the guaranteed bound is the TTL alone — quote that number, for example 30 seconds, and justify it against the cache hit rate you need to hold 5ms p99.
5. Evaluate expires_ts at read time against the request clock rather than baking the decision. A cached permit whose expires_ts has passed is wrong regardless of invalidation, because no event fires when a timestamp simply goes by.
6. Keep consent_directive as the system of record: audit must reconstruct the directive in force at any past instant, the decision table holds only current state, and a corrupted decision table must be rebuildable by replaying the directives. Bound the write amplification by keeping grantees at organisation granularity and purposes a closed code set, so one directive expands to a known small number of rows rather than an unbounded cross product.

Worked solution (40 min)

1. Write the decision table DDL with the three-column primary key, the sentinel wildcard, the precedence rank and a monotonic decision_version.
2. Write the lookup query: equality on the key with the wildcard row unioned in, ORDER BY rank, LIMIT 1, and an expires_ts predicate evaluated against the request timestamp.
3. Write the projection that turns one consent_directive version into its decision rows, and compute the expansion factor for a directive covering four purposes and one organisation.
4. State the TTL, then justify it: at N reads per second and M people, compute the cache hit rate the TTL yields and check it against the 5ms p99 budget.
5. Write the failure narrative for a lost invalidation and confirm the TTL, not the bus, is what caps exposure.
6. Write the rebuild procedure from consent_directive and the assertion that proves the rebuild is complete before it is swapped in.

Expected result: consent_decision (enterprise_person_id bigint, grantee_org_id int NOT NULL DEFAULT 0, purpose_of_use text, scope text, permit boolean NOT NULL, precedence smallint NOT NULL, expires_ts timestamptz NULL, decision_version bigint NOT NULL, computed_at timestamptz NOT NULL, PRIMARY KEY (enterprise_person_id, grantee_org_id, purpose_of_use)), where grantee_org_id = 0 is the wildcard. Lookup: SELECT permit FROM consent_decision WHERE enterprise_person_id = :p AND purpose_of_use = :pou AND grantee_org_id IN (:org, 0) AND (expires_ts IS NULL OR expires_ts > :now) ORDER BY precedence LIMIT 1; no row means deny. The guaranteed revocation bound is the local cache TTL — 30 seconds — because the invalidation publish is best-effort and a lost message must not extend exposure. consent_directive stays the system of record: it is what audit replays to reconstruct the directive in force at a past instant, and what the decision table is rebuilt from.

Verify it yourself

• A person with an explicit refusal for one organisation and a wildcard permit resolves to deny for that organisation and permit for every other — verified by two lookups against one fixture.
• Truncating consent_decision yields deny for every request rather than permit, confirming the store fails closed.
• With the invalidation bus stopped, a revoked directive stops being served no later than TTL seconds after the revocation commits — measured, not assumed.
• A decision row whose expires_ts has passed is never returned even when the cache entry is still within its TTL.

Follow-up

• Break-glass must always succeed and must be unmistakably marked. Where does it sit relative to this lookup, and what stops it from becoming the quiet default path?
• The invalidation bus is down for thirty minutes. Walk through what a revoked person's data exposure looks like minute by minute, and what the backlog does when the bus returns.
• Where does the access audit record get written so that an audit-store outage degrades neither the log's completeness nor the read's availability?

Add a backfilled NOT NULL column to a live encounter table

Hard · online migration · ddl locks · backfill

encounter holds 900 million rows and is written continuously by the admit/discharge/transfer consumer. You must add admission_source_code text, backfill it per row from a lookup on the source message archive, constrain it to a fixed code set, make it NOT NULL, and add an index on (facility_id, admit_ts) — with no write outage and no statement blocked longer than one second. Give the ordered steps, the lock level each takes, why the obvious single ALTER is unavailable here, and the session setting that stops a DDL statement from stalling everything behind it.

Approach

1. Say why the shortcut does not apply. Since PostgreSQL 11, ADD COLUMN with a constant default is metadata-only and cheap, but this value is derived per row from an archive lookup, so there is no constant to store and the table must be written row by row regardless.
2. Add the column nullable first: catalog-only, ACCESS EXCLUSIVE for microseconds. The danger is not the statement's duration but the lock queue — a pending ACCESS EXCLUSIVE blocks every reader that arrives behind it, so a long-running report turns a microsecond DDL into a multi-minute outage. SET lock_timeout = '1s' before the ALTER and retry on failure; that converts a potential outage into a retried statement.
3. Deploy dual-write before backfilling. The consumer must populate the column on every insert and transfer from that moment, or the backfill chases a moving tail forever.
4. Backfill in bounded batches over primary-key ranges, a few thousand rows per transaction with a pause between, restricted to WHERE admission_source_code IS NULL so it is resumable and idempotent. One giant UPDATE holds a transaction open for hours, pins the xmin horizon so autovacuum cannot clean anything, and bloats the table by a full row version per updated row.
5. Add the value constraint as CHECK (...) NOT VALID first — ACCESS EXCLUSIVE, no scan — then ALTER TABLE ... VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE and scans without blocking reads or writes.
6. Reach NOT NULL without a blocking scan: add CHECK (admission_source_code IS NOT NULL) NOT VALID, VALIDATE it, then SET NOT NULL, which on PostgreSQL 12 and later uses the validated constraint as proof and skips the full scan; drop the now-redundant CHECK afterwards. Build the index with CREATE INDEX CONCURRENTLY, which takes SHARE UPDATE EXCLUSIVE, makes two passes plus a wait for concurrent transactions, cannot run inside a transaction block, and leaves an INVALID index behind on failure that must be dropped and rebuilt.

Follow-up

• CREATE INDEX CONCURRENTLY has been running for four hours and you need to cancel it. What state is the index left in, how do you detect it, and what do you run next?
• Your backfill is at 40% and the replica is 90 seconds behind. What do you change, and what do you measure to decide the new batch size?
• The code set gains a value two weeks later. Does your CHECK constraint or a lookup table with a foreign key make that change cheaper, and what does each cost on the write path?

Explain the difference between IEnumerable and IQueryable.

Medium

Approach

1. Clarify what is being asked and what a complete answer contains.
2. Say what you would check first and why it is the highest-information step.
3. State your assumptions explicitly before working the problem.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

What are the four pillars of Object-Oriented Programming (OOP), and can you provide practical examples for each?

Medium

Approach

1. Work from the requirement backwards to the design.
2. Clarify what is being asked and what a complete answer contains.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Model an order placement so a lost response stays recoverable

Hard · idempotency · partial failure · reconciliation

An ordering system POSTs a medication or lab order to /orders. It assigns a placer identifier; the performing system later assigns a filler identifier, and neither side controls both namespaces. The caller times out after 3s and cannot distinguish an order that never arrived, one that was accepted with the acknowledgement lost, and one still in flight. Duplicating an order is a patient-safety event and silently abandoning one is worse. Design the contract so the caller can always determine the true outcome without guessing, and specify its behaviour at the timeout, on retry, and after repeated failure.

Approach

1. Remove the ambiguity at its source by letting the caller name the resource before it exists, so the identity of the attempt never depends on our response arriving. The placer identifier is already caller-assigned and unique within the caller's namespace, so the server key is (placer_namespace, placer_order_id) under a unique constraint. A timeout stops being an unknown and becomes a question with a stable key.
2. Prefer the shape that makes the retry trivially safe. PUT /orders/{placer_namespace}/{placer_order_id} is idempotent by HTTP semantics and needs no key header; POST /orders needs an Idempotency-Key to reach the same place. Under either, the write is one INSERT ... ON CONFLICT DO NOTHING, because a check-then-insert lets two retries through under READ COMMITTED. A repeat with an identical canonical body returns the existing resource; a repeat with a different body is 409 rather than a silent replace, since an order is not a document to overwrite.
3. Give the caller a read that resolves a timeout without writing: GET on the same key returns accepted, routed, filled with its filler_order_id, or rejected with a reason, and 404 means we genuinely never saw it. Without that read, the caller's only instrument is another write, which is precisely the behaviour being designed out.
4. Be honest about what acceptance means. Accept is durably persisted and queued for the performing system, not performed, so return 202 with the order resource and a status the caller polls or subscribes to. Returning 201 for an order that is not yet routed makes the caller believe a stronger fact than is true, and the filler identifier arrives later on the performing system's own timeline.
5. Write the caller's behaviour explicitly, because the contract is only half the design: at timeout, GET the key; on 404, retry the write with the same key; on 5xx, exponential backoff with full jitter up to a bounded attempt count; after five failures, stop and raise a human-visible alert carrying the placer identifier. A queued-for-human state is a better outcome than either a duplicate order or a silent drop, and this domain cannot absorb the drop.

Worked solution (40 min)

1. Draw the three timelines a 3s timeout can hide - request lost, work done and response lost, work still in flight - and mark what the caller can distinguish in each, with and without a caller-assigned key.
2. Write the resource path, the unique constraint, and the single atomic statement that performs the write.
3. Write the state machine the GET exposes, state what 404 means, and name the one state that must never be inferred from a timeout.
4. Write the caller's pseudocode for timeout, retry, backoff and give-up, with the attempt count and the alert payload.
5. Run two concurrent retries of the same placer identifier against a real database and confirm one order exists and both callers observe the same state.

Expected result: PUT /orders/{placer_namespace}/{placer_order_id} under UNIQUE (placer_namespace, placer_order_id), written by one INSERT ... ON CONFLICT DO NOTHING; a repeat with the same canonical body returns the existing resource, a repeat with a different body returns 409. GET on the same key returns {status in accepted | routed | filled | rejected, filler_order_id when known, reason when rejected}, with 404 meaning never seen. Acceptance is 202 - durably queued, not performed. Caller: on timeout GET the key; on 404 retry the same key; on 5xx back off with full jitter for five attempts, then stop and alert with the placer identifier rather than resubmitting.

Verify it yourself

• No outcome requires the caller to guess; every timeout resolves into a read.
• Two concurrent retries of one placer identifier leave exactly one order.
• A 202 is never returned for an order that is not durably persisted.
• The give-up path terminates in a human-visible state rather than a dropped order.

Follow-up

• The performing system returns a filler identifier for an order we have no record of. What do you do with it?
• The caller reissues an old placer identifier for a genuinely different order after a counter rollover. What breaks, and how would you detect it?
• Where does the record of the failed attempts live, and what must it carry for an incident review?

Nightly adjudication throughput collapses with deadlocks on accumulators

Medium · deadlock · lock ordering · isolation levels

During the nightly claims batch, throughput collapses from 4,000 to about 300 claim lines a second for minutes at a time. pg_stat_database.deadlocks climbs, the PostgreSQL log carries deadlock detected with SQLSTATE 40P01, and the application retries the aborted transactions. Benefit application and the reversal handler both take SELECT ... FOR UPDATE on the member's deductible accumulator row and on the out-of-pocket accumulator row. Diagnose in order, then give a fix that removes the cycle by construction rather than by retrying faster.

Approach

1. Read the deadlock report before theorising. PostgreSQL logs both process IDs, the statement each was running and the relation and tuple each was waiting on, which names the two lock-acquisition orders directly. There is no need to guess which code paths collide or to reproduce it first.
2. Separate deadlock from ordinary contention. pg_stat_database.deadlocks counts cycles, while log_lock_waits reports waits longer than deadlock_timeout that never form a cycle. Collapsing throughput with few recorded deadlocks means a hot row — one family plan's shared accumulator — is serialising the batch, which has a different remedy from a cycle.
3. Account for the detector's cost. A cycle is only detected after deadlock_timeout, one second by default, so every deadlock burns at least that on both sides plus the retried work. Raising the timeout lengthens each stall; lowering it raises the frequency of the detector's checks. Neither is a fix, and both are common first answers.
4. Reject the two reflex answers explicitly. SERIALIZABLE does not prevent deadlocks in PostgreSQL — row-level lock cycles still occur and serialization failures with SQLSTATE 40001 are added on top — and a larger connection pool simply puts more transactions in contention for the same rows.
5. Remove the cycle by construction: make each member's benefit application take exactly one lock. Either fold both accumulators into a single row keyed (enterprise_person_id, plan_id, plan_year) and update it in one statement, or take pg_advisory_xact_lock over a hash of that key as the first statement of both paths. One lock cannot form a cycle with itself. Deterministic ordering via ORDER BY ... FOR UPDATE also works but depends on the chosen plan preserving that order, so it is a weaker guarantee; hash collisions in the advisory-lock variant only serialise two unrelated members, which is safe but costs throughput.
6. Then bound the transaction. No external call may sit between acquiring the lock and committing, or the row is held for a network round trip and the hot member serialises the batch regardless of which deadlock fix you chose.

Follow-up

• A reversal arrives for a claim adjudicated under a plan design that has since changed. What amount does it subtract, where is that amount recorded, and is that still atomic under your single-lock design?
• Remove the FOR UPDATE entirely and show the exact two-session interleaving that produces a lost update under READ COMMITTED, then say why a single UPDATE ... SET remaining = remaining - $1 does not have the same problem.

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

• Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
• Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
• Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

• Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
• Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
• Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

• Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
• Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
• Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

• For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
• Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
• Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
• Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

• Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
• Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
• Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
• Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

• Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
• Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
• Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

• Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
• Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
• Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

Tell me about a time you worked in a team and had to overcome internal difficulties or conflict.

Medium · behavioural and engineering judgement

Approach

1. Close with what you would do differently, concretely.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that again?

What are your professional strengths and weaknesses, and how are you working to improve?

Medium · behavioural and engineering judgement

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Name the disagreement and how you resolved it with evidence.
3. Pick a story where you made the decision, not one where you watched it.

Follow-up

• What would you do differently if you ran that again?
• What did you decide not to do, and why?

How do you handle stressful situations or competing deadlines?

Medium · behavioural and engineering judgement

Approach

1. Name the disagreement and how you resolved it with evidence.
2. Pick a story where you made the decision, not one where you watched it.
3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that again?

Explain the stages of the SDLC and your experience within each phase.

Medium · behavioural and engineering judgement

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Name the disagreement and how you resolved it with evidence.
3. Close with what you would do differently, concretely.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?