Interview guides · AMCS Group
AMCS Group Software Engineer Interview Questions & Guide 2026
AMCS Group software engineer prep on operations: decide what you would need to see to know a release is going wrong.
The 60-second brief
As a Software Engineer at AMCS Group, you are at the heart of driving digital transformation for the environmental, waste, and recycling industries. Your work directly impacts the efficiency and sustainability of essential global infrastructure. By developing and maintaining sophisticated software solutions, you help organizations optimize complex logistics, automate manual processes, and leverage data to make smarter operational decisions.
Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.
Focus areas
- Evolve APIs without breaking pinned SDK clients
- Make every write idempotent under client retries
- Scope every query and cache key by tenant
Overview
This role is both technically demanding and strategically significant. You will often work within cross-functional, global teams to build scalable, high-performance applications that handle real-world challenges. Whether you are refining backend architectures, improving frontend user experiences, or troubleshooting critical production issues, your contributions ensure that AMCS Group clients can manage their resources more effectively in a rapidly changing market.
Expect to engage with a modern technology stack and a culture that values both technical precision and collaborative problem-solving. Success in this role requires a balance of deep engineering expertise and the ability to communicate complex ideas to stakeholders, making it an ideal environment for engineers who are passionate about building software that delivers tangible, real-world utility.
The AMCS Group interview process
AMCS Group candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Recruiter Screen
Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.
What to demonstrate
- Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
- Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
- Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not
How to prepare
- Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
- Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
- If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window
Technical Deep Dives
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Team or Leadership Interview
When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.
What to demonstrate
- Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
- Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
- Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
- Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened
How to prepare
- Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
- Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
- Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure
Where candidates lose points
Treating a timed-out write as a failed write
A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.
Paginating a growing table with limit and offset
Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.
Starting work without saying what you are about to spend time on
State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.
A cache with no invalidation story
Say how an entry goes stale, how long you can serve it stale, and what happens when many requests miss the same key at the same instant. One popular key expiring under load sends every concurrent request to the origin together; single-flight coalescing, jittered expiry, or serving stale while revalidating are the standard answers.
Questions & practice
Order a job dependency graph and find its critical path
Medium · topological-sort · dag-longest-path · cycle-detection · critical-path
A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.
Approach
- Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
- Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
- Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order:
earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E). - Second pass in reverse topological order for
latest_finish, thenslack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain.slack[v] = 0is exactly the statement that some longest path runs through v; equivalently, the longest path through v has lengthT - slack[v]. - The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by
min(d, T - L_avoid(v)), whereL_avoid(v)is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, whilemin(10, 15 - 14)predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan ismax(T - d, L_avoid(v)). - Compute
L_avoid(v)the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, sinceL_avoid(v)only ever matters through that max: setduration[v] := 0, recompute the makespan asT0(v) = max(T - duration[v], L_avoid(v)), and the gain ismin(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.
Follow-up
- Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
- Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
- Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?
Fold a deduplicated usage stream into hourly rollups
Easy · aggregation · deduplication · watermarks · exact-arithmetic
You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.
Approach
- Bucket on
occurred_at, neveringested_at:hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions.occurred_atsays which hour the customer is billed for;ingested_atsays how current the fold is. Using the second for the first makes late data invisible instead of correctable. - The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over
(tenant_id, idempotency_key)at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning byhash(tenant_id) % Pso each shard holds 1/P of the set and no tenant's keys straddle shards. - Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
- Accumulate in scaled integers, not binary floating point.
numeric(20,6)admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree. - Carry
source_max_ingested_at = max(ingested_at)over the events folded into each cell, and countevent_countover accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks. - State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes
stagingbills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.
Worked solution (25 min)
- Write both key tuples down before any code: dedup key
(tenant_id, idempotency_key), cell key(tenant_id, workspace_id, sku, hour_start), withhour_startderived fromoccurred_atin UTC. - Build a 10,000-row fixture containing one event duplicated three times under the same
idempotency_key, two events sharing anidempotency_keyacross differenttenant_idvalues, one event whoseoccurred_atis two hours before itsingested_at, and onestagingevent inside an otherwise production cell. - Fold it and assert each of those four expectations separately rather than eyeballing a grand total.
- Re-run with the input shuffled and diff the output files.
- Size the dedup set for 250M keys using the load-factor arithmetic and write the number down next to the fixture.
Expected result: The triplicate contributes one event and its quantity once. The two same-key, different-tenant events both count, because the dedup key is the pair. The late event lands in the hour of its occurred_at while that cell's source_max_ingested_at advances to the later timestamp. The staging event is included or excluded per the stated filter and never silently.
Verify it yourself
- Shuffled input produces byte-identical cells, including
event_countandsource_max_ingested_at. - The sum of
quantity_sumacross all cells equals the sum of quantity over the distinct dedup keys, computed independently. - Swapping the accumulator to binary64 makes a fixture of 0.1-valued quantities disagree with the exact sum. That is a failing test, not a rounding note.
Follow-up
- A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
- The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
- What makes a re-run over the same day produce byte-identical rollups?
Schedule ordered webhook retries with a heap of subscription queues
Medium · heap · scheduling · backoff · head-of-line-blocking
Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.
Approach
- Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's
next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, incrementattempt_countand setnext_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.- Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
- Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
- All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on
(subscription_id, next_attempt_at) where status in ('pending','failed_retryable')claimed withFOR UPDATE SKIP LOCKED, and the write-back must be fenced onlease_tokenso a worker that stalled and resumed cannot overwrite a newer attempt.
Follow-up
- One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
- A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
- The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?
Enforce a concurrent-run quota that survives simultaneous requests
Medium · isolation levels · write skew · quotas · lock contention
A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.
Approach
- Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
- Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as
insert ... select ... where (select count(*) ...) < 20narrows the race to the statement's snapshot but does not close it under read committed. - Give the version that holds at read committed: serialise on a row both transactions must touch.
update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning runningupdates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract. - State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
- Close the leak the status enum implies: a run can end as
lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule againststatus = 'running' and leased_until < now(), and treat the counter as a fast path overjob_run, which stays the system of record.
Follow-up
- Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
- Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
- The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?
Find the join that inflates every invoice total
Medium · join fan-out · aggregation grain · sql correctness
invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.
Approach
- Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row:
sum(l.amount_minor)is the true total multiplied by the attempt count, andcount(p.attempt_id)is attempts times lines. Three lines and two attempts report double the money and six attempts. - Reject the reflex repair.
count(distinct p.attempt_id)does fix the count, because attempt_id is unique.sum(distinct l.amount_minor)does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match. - Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
- Keep invoices with no attempts by making the attempt branch a LEFT JOIN with
coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about. - Push each filter to its own grain:
where l.voided_at is nullbelongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants. - Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.
Worked solution (25 min)
- Create one invoice with three lines of 1000, 1000 and 500 minor units and two payment attempts, then run the original query.
- Confirm it reports 5000 and 6 rather than 2500 and 2.
- Apply
sum(distinct l.amount_minor)and confirm the total becomes 1500, which is worse rather than better. - Write the two-CTE version with a LEFT JOIN and coalesce, and confirm 2500 and 2.
- Add a second invoice with lines and no attempts and confirm it still appears.
Expected result: The original sum equals the true line total multiplied by the number of payment attempts and the count equals attempts times lines; the corrected query pre-aggregates each branch to invoice grain in separate CTEs and LEFT JOINs them with coalesce, with voided lines excluded inside the line CTE and the tenant predicate applied to both branches.
Verify it yourself
- The money total is unchanged by adding or removing payment attempts.
- An invoice with zero attempts appears with its total and a zero count.
- Two lines with identical amounts both contribute to the total.
- Voiding a line changes the money figure and leaves the attempt count alone.
Follow-up
- Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with
filter (where ...)be better? - Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
- The same report is needed per tenant per month. What index makes the line CTE cheap?
How many basketballs could fit in this room?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
How would you approach a situation where you need to solve a problem with no clear documentation?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
How do you evaluate code fragments for maintainability and efficiency?
Medium
Approach
- Work from the requirement backwards to the design.
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Write the delivery guarantee and replay API for outbound events
Medium · webhooks · at-least-once · ordering
The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.
Approach
- Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
- Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
- Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
- Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
- Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
- Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.
Worked solution (30 min)
- Write the guarantee sentence and the one-sentence reason exactly-once is not available over HTTP.
- List the headers a customer needs to build their own dedup table: event id, delivery id, attempt number, subscription id, signature and timestamp.
- Write the response-code table with four classes and the action for each, including the 410 case.
- Write the retry schedule with concrete numbers, the terminal condition, and the breaker's threshold and its effect on the delivery row's status.
- Specify the redelivery request and response, and say which columns change and which do not.
Expected result: At-least-once with a stable event id, ordering offered only as an opt-in single-in-flight mode with head-of-line blocking stated, a four-class response table, a jittered capped schedule with a terminal state, and redelivery implemented as new rows at an incremented sequence carrying identical bytes.
Verify it yourself
- The customer can deduplicate using headers alone, without parsing the payload.
- Nothing in the contract requires the sender to know whether the customer committed.
- A day-long outage has a defined end state rather than unbounded retries.
- Redelivery leaves the original delivery rows readable as history.
Follow-up
- A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
- You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?
Gateway p99 spikes on a five-minute cadence
Medium · cache stampede · ttl · invalidation
edge-gateway caches each credential-to-authorisation-context decision for five minutes. p99 sits at 6 ms except for a spike to 900 ms roughly every five minutes, worst in the region with the most pods, and control-plane CPU and read latency rise in step with it. The error rate stays near zero. Customers are told a revoked credential stops authorising within 60 seconds. Give the ordered checklist that identifies the mechanism, and a fix that removes the spike without weakening the 60-second bound.
Approach
- Test periodicity before anything else: take the spike timestamps modulo the TTL in seconds. A tight cluster at a fixed offset means expiry phase, while traffic-driven spikes scatter.
- Overlay pod start times. Entries filled at first request inherit the phase of the pod that filled them, so a cohort of pods deployed together expires together and the amplitude should track cohort size rather than tenant count.
- Separate a herd from a capacity shortfall by measuring control-plane requests per second during a spike against baseline. A stampede shows a step of roughly (pods x hot keys) for one interval with hit rate collapsing to near zero, not a gradual climb that would indicate the dependency is simply undersized.
- Apply three independent controls: randomise each key's TTL by a factor drawn uniformly from something like 0.8 to 1.0 so cohorts de-phase; coalesce concurrent misses per key per pod so exactly one refresh is in flight; and serve the stale value while that refresh runs so a miss costs the stale read rather than the dependency's queue.
- Bound staleness against the published contract rather than against comfort: serve-stale is admissible only up to the 60-second revocation bound, so the TTL floor and the stale window together must stay inside it, and the published invalidation must delete the entry rather than schedule a refresh.
- Decide in advance what a miss does when the control plane is unreachable, because that is now the only uncached path: failing closed converts a dependency outage into a total outage, while extending stale service past the bound breaks the revocation promise. Pick one and configure it explicitly.
Follow-up
- Publish-subscribe invalidation is lossy under a partition. Given that, what actually enforces the 60-second bound, and what number would you put in the contract if asked to defend it?
- One tenant's key is hot enough that a single pod's coalesced refresh still matters. What changes?
- Would a shared cache tier in front of the control plane help or shift the problem, and what new failure does it add?
Suggested practice checklist
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Day 1 — Measure before reasoning
- Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
- Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
- Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.
Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.
Day 2 — References, copies, and the bugs they produce
- Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
- Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
- Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.
Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.
Day 3 — Types, once, in a language that checks them
- Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
- Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
- Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.
Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.
Day 4 — Concurrency, starting with what actually runs at the same time
- Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
- Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
- Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.
Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.
Day 5 — Debugging as a procedure rather than an instinct
- Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
- Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
- Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.
Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.
Day 6 — Tests that catch the bug you are about to write
- Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
- Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
- Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.
Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.
Day 7 — Debug something broken, out loud
- Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
- Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
- Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.
Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.
Outage answers are graded on instruments
Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.
Tell us about a project you felt proud of—what was your specific role and the outcome?
Medium · behavioural and engineering judgement
Approach
- Give the blast radius: what could have broken, and what you measured.
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
How do you handle a frustrated customer experiencing a recurring software issue?
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Give the blast radius: what could have broken, and what you measured.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?
Describe a time you worked with a team across different time zones. How did you ensure effective communication?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What did you decide not to do, and why?
- What would you do differently if you ran that again?