Interview guides · Erith Group
Erith Group Software Engineer Interview Questions & Guide 2026
Erith Group software engineer prep: algorithm drills where every complexity claim has to name the assumption under it.
The 60-second brief
As a Software Engineer at Erith Group, you occupy a vital position at the intersection of traditional engineering excellence and modern technical implementation. You are responsible for bridging the gap between complex structural challenges and the digital solutions that support the firm’s renowned work in temporary works and geotechnical engineering. Your contributions directly impact the safety, efficiency, and precision of large-scale construction projects.
Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.
Focus areas
- Bound every outbound call with a timeout
- Trace a symptom to a mechanism under load
- Choose indexes from the query's access path
Overview
This role is not merely about writing code; it is about applying rigorous engineering logic to real-world problems. You will work within specialized departments, such as the Temporary Works team, where your ability to translate technical concepts into practical, software-driven workflows is critical. Expect a fast-paced environment where your technical proficiency is matched by your ability to communicate complex ideas to a multidisciplinary team.
The Erith Group interview process
Erith Group candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Initial Introduction
An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.
What to demonstrate
- Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
- Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
- Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating
How to prepare
- Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
- Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
- Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes
Technical Assessment
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Possible Follow-Up
You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.
What to demonstrate
- Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
- Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
- Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
- Whether your part in a team effort is stated accurately, including what other people did
How to prepare
- Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
- Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
- Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.
Where candidates lose points
Letting a slow dependency consume unbounded concurrency
The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.
Running a schema change as though the lock lasts as long as the statement
In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.
Not asking what the system looks like if it dies halfway through
For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.
Writing code before the input contract is pinned down
Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.
Questions & practice
Diff a projection against the primary without per-row point reads
Hard · reconciliation · range hashing · throttling
The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.
Approach
- Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
- Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
- For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
- Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
- Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
- Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.
Follow-up
- The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
- Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
- How would you run this continuously at low cost instead of only as incident response?
Collapse a redelivered event batch into per-aggregate high-water marks
Easy · hashing · at-least-once · aggregation
You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.
Approach
- One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
- Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
- Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
- If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
- Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
- Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).
Follow-up
- The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
- How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
- Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?
Canonicalise a request body into a stable idempotency fingerprint
Medium · parsing · canonicalisation · hashing
idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.
Approach
- Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
- Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
- Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
- Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
- Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
- Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.
Worked solution (25 min)
- Write the serialiser: recursive emit with a depth counter, objects sorted by UTF-8 key bytes, arrays in order, strings escaped by one fixed rule, numbers emitted as their original token.
- Run it over three bodies: the same object with keys reordered, the same object with \u0041 written as A, and one with a nested array reversed. The first two must produce identical bytes and the third must not.
- Take the id 9007199254740993, round-trip it through a double, show it returns as 9007199254740992, then state the rule that prevents this.
- Define the hash preimage explicitly with its delimiters, and construct a pair of (path, body) inputs that would collide without them.
Expected result: A canonicaliser that is O(n log n) in body size with an enforced depth cap, sorts keys by UTF-8 byte order, preserves array order, keeps number literals verbatim, rejects duplicate keys, and feeds a delimited preimage to SHA-256, together with a stated list of normalisations deliberately not performed and the failure each would cause.
Verify it yourself
- Key reordering and whitespace changes produce identical fingerprints while reordering an array does not.
- A body containing an integer above 2^53 fingerprints identically after a parse and re-serialise round trip.
- Two requests differing only in a field the endpoint ignores produce different fingerprints, and you can say why that is the safe direction to err.
Follow-up
- A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
- Where does the fingerprint get computed relative to request decompression and the body-size limit?
- The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?
Hold a per-tenant active cap against concurrent creates
Hard · write skew · isolation levels · constraints
A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.
Approach
- Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
- Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
- Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
- State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
- For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
- Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.
Worked solution (35 min)
- Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
- Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
- Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
- Implement the partial unique index for the job case and race 20 enqueues of the same export.
Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.
Verify it yourself
- count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
- Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
- At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
- After each run, the counter equals the real count of active rows.
Follow-up
- A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
- The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
- How do you detect after the fact that the counter drifted, without locking the table?
Denormalise tenant onto revisions and backfill it live
Hard · denormalisation · online migration · backfill
resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.
Approach
- Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
- Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
- Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
- Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
- Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
- Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.
Follow-up
- The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
- How do you verify the backfill actually finished, given rows are still being inserted while it runs?
- A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?
Explain your approach to solving structural load problems.
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
- State your assumptions explicitly before working the problem.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Describe your process for validating technical calculations within a digital tool.
Medium
Approach
- State your assumptions explicitly before working the problem.
- Work from the requirement backwards to the design.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
Cache the tenant listing feed with a bounded staleness window
Easy · caching · cache invalidation · keyset pagination
GET /v1/resources returns one tenant's resources ordered by updated_at DESC, 20 per page, at 14k requests/second peak against a 120 ms p99. The table carries the index (tenant_id, status, updated_at DESC, resource_id DESC) and writes land on the primary at 1.2k/second. Design the read path: the pagination contract, the cache key and value, what a write invalidates, and the staleness a user can observe. State the request rate that actually reaches the database, and the one repopulation race that deleting on write does not close.
Approach
- Settle the pagination contract first, because it decides what is cacheable. OFFSET makes the database produce and discard the skipped rows, so page 500 costs five hundred pages of work, and rows inserted between two fetches shift across the boundary and are skipped or repeated with nothing in the response to reveal it. The cursor is the row value of the last row returned: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($3, $4) ORDER BY updated_at DESC, resource_id DESC LIMIT 21. That is a row-value comparison, not updated_at < $3 AND resource_id < $4, which is a different and wrong predicate.
- Confirm the index actually serves it: equality on the two leading columns, then a range on the pair that follows in exactly the index's sort order, so the plan is an index scan that touches 21 entries with no sort node. Requesting 21 to return 20 is how has_more is answered without a count. resource_id is not decoration - updated_at is not unique, and without the tie-break two rows sharing a timestamp at a page boundary are the skip that keyset pagination was adopted to remove.
- Key the cache on every value the predicate reads: tenant_id, status, cursor and page size. A key that omits tenant_id is a cross-tenant disclosure, and no test running against a single tenant's data will show it.
- Be honest that a write does not invalidate one key. An update moves its row to the head of the ordering, so it invalidates the first page and every cursor page whose range spans the row's old and new position, which is not enumerable. Cache the first page per (tenant_id, status) - that is where the traffic is - with a short TTL, invalidate it on write, and serve deep cursor pages uncached from a replica, since each is already a 21-row index scan and they are rare.
- Name the residual race and the real bound. A reader that loaded rows before the write can populate the cache after the invalidation deleted the key, so the delete is not a staleness bound; the TTL is. Choose the TTL as the staleness a listing can tolerate, and jitter expiries so a busy tenant's keys do not all expire together and stampede the replica. Then do the arithmetic: at an 85% hit rate, 14k requests/second is about 2.1k database reads/second across two replicas, and that is the number capacity planning uses.
Worked solution (20 min)
- Write the keyset predicate and match it column by column against the index, marking which columns are equality, which is the range, and which satisfies the ORDER BY.
- Compare rows examined for page 1 and page 500 under OFFSET and under keyset, and state both numbers.
- Write the cache key template and the first-page invalidation the write path performs.
- Write the interleaving in which a stale value is written into the cache after the invalidation, and identify what bounds it.
Expected result: Keyset pagination on the row value (updated_at, resource_id) served as a 21-entry index scan with no sort at any page depth. The first page per (tenant_id, status) is cached under a short jittered TTL and deleted on write; deeper cursor pages go to a replica uncached. Observable staleness is the TTL, not the invalidation, because a concurrent reader can repopulate the key afterwards. At an 85% hit rate roughly 2.1k reads/second reach the two replicas.
Verify it yourself
- The query plan shows an index scan with no Sort node, and rows examined equal to the limit, at page 1 and at page 500.
- Two rows sharing an updated_at value across a page boundary are returned exactly once across the two pages.
- Every cache key contains tenant_id.
- Removing the TTL leaves staleness unbounded even though every write still invalidates.
Follow-up
- The tenant writes and immediately lists. What does it see, and what is the smallest change that makes its own write visible without sending all 14k requests/second to the primary?
- A tenant has 4 million resources and a client walks every page nightly. What does that do to the cache hit rate, and should that traffic share this path at all?
- Sort order becomes configurable - by title, by created_at. What happens to the index set and to the cache key space?
One customer endpoint stalls deliveries to every other destination
Medium · bulkhead · timeouts · saturation · retry amplification
The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.
Approach
- Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
- Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
- Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
- Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
- Contain in order: park the offending host so the shared pool drains, add a per-resolved-host concurrency cap alongside the per-destination one, give slow hosts their own queue so they cannot occupy the general pool, derive the timeout from the delivery deadline rather than a round number, and check the retry policy is not tripling load on a host that is already slow. Use backoff with full jitter so retries do not resynchronise on recovery.
- State the invariant you are restoring: one tenant's endpoints degrade only that tenant's deliveries. That is a property to load-test for, not to assume from a config value.
Follow-up
- The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
- Where should the 10-second timeout number actually come from?
- If that tenant had one destination row instead of 900, would the cap of 4 have saved you? What would you measure to be sure?
Suggested practice checklist
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Day 1 — Measure before reasoning
- Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
- Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
- Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.
Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.
Day 2 — References, copies, and the bugs they produce
- Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
- Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
- Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.
Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.
Day 3 — Types, once, in a language that checks them
- Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
- Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
- Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.
Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.
Day 4 — Concurrency, starting with what actually runs at the same time
- Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
- Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
- Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.
Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.
Day 5 — Debugging as a procedure rather than an instinct
- Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
- Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
- Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.
Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.
Day 6 — Tests that catch the bug you are about to write
- Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
- Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
- Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.
Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.
Day 7 — Debug something broken, out loud
- Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
- Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
- Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.
Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.
'We rewrote it in a better framework' is not a story
A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.
Why are you interested in applying your engineering background to software development at Erith Group?
Medium · behavioural and engineering judgement
Approach
- Give the blast radius: what could have broken, and what you measured.
- Pick a story where you made the decision, not one where you watched it.
- Close with what you would do differently, concretely.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?
What are the most important qualities of a successful engineer in this industry?
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
- State the situation in two sentences and spend the rest on the reasoning.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Talk about your knowledge and experience.
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What did you decide not to do, and why?
- What would you do differently if you ran that again?
Can you describe a time you had to explain a complex technical problem to a non-technical stakeholder?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Pick a story where you made the decision, not one where you watched it.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What did you decide not to do, and why?
- How did you know your change caused the improvement?