cswork / libraryOpen practice workspace

Interview guides · American Jereh International

American Jereh International Software Engineer Interview Questions & Guide 2026

American Jereh International software engineer prep: algorithm drills where every complexity claim has to name the assumption under it.

The 60-second brief

At American Jereh International, the Software Engineer role sits within the company's specialized engineering divisions. You will develop, maintain, and optimize the control systems and software architectures that drive the company's industrial equipment, including gas turbine packages, microgrid solutions, and advanced pumping systems.

Ask how many rounds there are and what each one is before you plan your weeks, because the composition is what your hours should follow. A loop with three coding rounds and one short design conversation deserves a different split from the reverse, and whoever schedules it will usually say plainly which it is.

Focus areas

  • Bound every outbound call with a timeout
  • Paginate large result sets with keyset cursors
  • Trace a symptom to a mechanism under load

Overview

This role sits at the intersection of traditional mechanical/electrical engineering and modern digital control. Your work affects the reliability, efficiency, and safety of the company's energy and industrial products. You will be expected to tackle complex technical challenges, making sure the company's software integrates with physical hardware to meet the demands of its global clients.

The American Jereh International interview process

American Jereh International candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

  • Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
  • Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
  • Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

  • Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
  • Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
  • Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Interviews

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Team Discussions

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Treating a network call as though it were a local function call

A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.

Questions & practice

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

  1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
  2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
  3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
  4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
  5. State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
  6. Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Worked solution (30 min)

  1. Write the interval derivation with the COALESCE and state in one line which of the two end sources is observed and which is assumed.
  2. Write the concurrency sweep: the endpoint tuples, the sort key including the end-before-start tie-break, and the per-job_type counter.
  3. Hand-trace four attempts of one job, two disjoint and two overlapping by three seconds, and confirm the overlap detector fires exactly once.
  4. Add the skew filter as a minimum overlap duration, state the value you chose, and justify it from how the timestamps are written.

Expected result: One global sort by (job_run_id, started_at) driving a running-maximum-end check for overlaps, plus a sweep over 2n endpoints with ends ordered before starts for per-job_type peak concurrency, both O(n log n), with half-open intervals, NULL finished_at falling back to lease_expires_at, and overlaps shorter than the clock-skew bound excluded.

Verify it yourself

  • Two attempts touching at exactly the same instant report concurrency 1, not 2.
  • A job with a single attempt never appears in the overlap output regardless of its lease.
  • The reported peak timestamp for a job_type lies inside at least one of that type's intervals.

Follow-up

  • A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
  • Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
  • How would you compute both answers incrementally as records arrive rather than in a daily batch?

Canonicalise a request body into a stable idempotency fingerprint

Medium · parsing · canonicalisation · hashing

idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.

Approach

  1. Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
  2. Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
  3. Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
  4. Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
  5. Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
  6. Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.

Follow-up

  • A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
  • Where does the fingerprint get computed relative to request decompression and the body-size limit?
  • The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

  1. Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
  2. Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
  3. State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
  4. Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
  5. Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
  6. Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Follow-up

  • The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
  • A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
  • How would you make the window survive a process restart, and is it worth the cost?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

  1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
  2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
  3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
  4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
  5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
  6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Worked solution (35 min)

  1. Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
  2. Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
  3. Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
  4. Implement the partial unique index for the job case and race 20 enqueues of the same export.

Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.

Verify it yourself

  • count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
  • Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
  • At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
  • After each run, the counter equals the real count of active rows.

Follow-up

  • A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
  • The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
  • How do you detect after the fact that the counter drifted, without locking the table?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

  1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
  2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
  3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
  4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
  5. Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
  6. Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Follow-up

  • The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
  • A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
  • Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Explain your approach to debugging a system where the issue could be either hardware-related or software-related.

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Clarify what is being asked and what a complete answer contains.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How do you approach the documentation of complex codebases for long-term maintenance?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Say what you would check first and why it is the highest-information step.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Keep one unresponsive destination from stalling all webhook delivery

Medium · fan-out · circuit breaker · timeouts · bulkheads

Egress delivery sends about 1.5k webhooks/second to 40k destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. One destination begins accepting connections and never responding; within the hour 150 destinations behave the same way. Design the delivery path so unrelated destinations are unaffected: the pool structure, the timeouts, the retry policy, the per-destination circuit, and what is recorded so a retry is not a second effect at the receiver. State how many in-flight slots the degraded destinations hold and why that number decides the design.

Approach

  1. Start with the number, and with the law that produces it. In-flight work is arrival rate times time in service, so 1.5k/second against a healthy 200 ms response needs about 300 concurrent slots. Per destination the same product applies, ceilinged by the concurrency cap: at the fleet average of 0.0375 deliveries/second per destination (1.5k spread over 40k) a 10-second timeout is 0.375 slots. A destination that has queued retries behind it is a different regime - every slot refills the instant an attempt expires, so it sits pinned at its cap of 4 - and 150 of those hold 600 slots, more than a pool sized for healthy traffic, entirely consumed by endpoints that will never answer. The per-destination cap bounds one endpoint and says nothing about the aggregate, which is exactly why it alone is not containment.
  2. Contain with bulkheads and an admission bound rather than a larger pool. Cap total in-flight per pool and shard destinations across pools by a hash of destination id, so a correlated group - one provider, one region - cannot exceed its pool's share. A delivery refused admission and re-queued with backoff is strictly better than one holding a slot on behalf of a receiver that is not listening.
  3. Treat the timeout as two timeouts, and be exact about what shortening one buys. Connect and read are separate failures and both must be shorter than the budget of whatever is waiting. Occupancy is min(cap, arrival rate x timeout), so dropping the read ceiling from 10 seconds to 3 cuts a merely slow destination's occupancy proportionally, 0.375 slots to 0.11 at the fleet average. It does not cut the 4 slots held by one of the 150: a destination with a retry backlog arrives far above cap/timeout - 0.4/second at a 10-second timeout, 1.33/second at 3 - so it stays pinned at the cap either way and only the slot-seconds per attempt fall. What that does buy is detection rate: 3.3x more failures observed per second on the same four slots, which is how fast the circuit reaches its threshold. Pick the value from the measured latency distribution of successful deliveries, with their high percentile as the floor, not from a round number.
  4. Add a circuit per destination, counting a timeout as a failure. Once open, fail fast without taking a slot - that is the whole point, converting 4 held slots into zero. Half-open on a schedule with exactly one probe and close only if the probe succeeds, so a permanently dead endpoint costs one request per interval instead of a growing retry queue.
  5. Make retries safe and non-synchronising. Back off with full jitter, sleeping a random value in [0, min(cap, base x 2^attempt)], because a fixed delay re-synchronises every failed delivery to one destination into a simultaneous burst. Delivery is at-least-once, so the payload carries the event id under the signature and the receiver deduplicates on it; record the attempt against (destination, event id) rather than a bare success flag, so a lost response does not become a second business effect on the other side.

Worked solution (25 min)

  1. Compute healthy in-flight from rate times latency, then slots held by 150 destinations at the cap and the full timeout, and compare both against one pool size.
  2. Write the pool sharding rule and the admission bound, and state what a refused delivery does next.
  3. Pick connect and read timeouts from the success-latency distribution, then compute min(cap, arrival rate x timeout) for an average destination and for one with a retry backlog, and say which of the two the shorter timeout actually moves.
  4. Write the circuit's state machine with its open threshold, probe interval and close condition, and the backoff formula with full jitter.

Expected result: Destinations sharded across bounded pools with a total in-flight cap per pool, separate connect and read timeouts chosen from the success-latency distribution, a per-destination circuit that fails fast while open and probes once per interval, and full-jitter backoff on retry. A shorter read timeout lowers occupancy only for destinations arriving below cap/timeout; a dead destination with a retry backlog stays pinned at its cap of 4 whatever the timeout, and the circuit opening is what takes it to zero. Deliveries carry a signed event id the receiver deduplicates on, and attempts are recorded per (destination, event id). Degraded destinations hold at most their pool's share rather than 600 slots of a shared pool.

Verify it yourself

  • Slots held by degraded destinations are bounded by the pool shard, not by how many destinations degrade.
  • An open circuit reduces slots held by that destination to zero between probes.
  • Lowering the read timeout does not reduce successful deliveries, verified against the measured success-latency distribution, and does not by itself reduce the slots a backlogged dead destination holds.
  • A redelivered event is discarded by the receiver on its event id rather than applied twice.

Follow-up

  • The destination is not dead - it answers in 9.5 seconds with a 200. Does a failure-rate circuit open? Should anything shed that traffic, and on what signal?
  • One destination requires deliveries in order. What does a per-destination concurrency of 4 do to that guarantee, and what would you change to offer it?
  • A destination has been parked six hours with 900k undelivered events. What does resuming look like, and is delivering the whole backlog the right call?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles · tail latency

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

  1. Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
  2. Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
  3. Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
  4. Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.
  5. Price the fix. This is a fourth index on a table taking about 1.2k writes/second, and every insert and version bump maintains it. Justify it against the query it serves, and check whether it makes an existing index redundant, which here it does not, since the original still serves the status-filtered default listing.
  6. Re-measure per tenant-size bucket rather than in aggregate. A fleet-wide p99 can improve while the largest tenant is still on the old plan.

Follow-up

  • The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
  • How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?
  • If a fourth index were unacceptable on write grounds, what else could serve this query?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

  • Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
  • Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
  • Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

  • Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
  • Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
  • Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

  • Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
  • Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
  • Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

  • Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
  • Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
  • Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

  • Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
  • Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
  • Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

  • Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
  • Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
  • Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

  • Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
  • Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
  • Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

Give an example of a time you took initiative to improve a process or workflow within your team.

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Name the disagreement and how you resolved it with evidence.
  3. Close with what you would do differently, concretely.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

How do you handle feedback from stakeholders when a proposed technical solution is not feasible?

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. Pick a story where you made the decision, not one where you watched it.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Describe a project where you had to balance aggressive deadlines with the need for high-quality, bug-free software.

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Give the blast radius: what could have broken, and what you measured.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

Tell us about a time you had to coordinate with mechanical or electrical engineers to resolve a system conflict.

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Close with what you would do differently, concretely.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?