Interview guides · ARRO Consulting, Inc.

ARRO Consulting, Inc. Software Engineer Interview Questions & Guide 2026

ARRO Consulting, Inc. software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

At ARRO Consulting, Inc., the Software Engineer role is a unique intersection of traditional engineering disciplines and modern technical problem-solving. You are not just writing code; you are building the digital infrastructure that enables the firm's teams to deliver complex environmental, civil, and water/wastewater projects. Your work affects how the firm manages critical community resources, ensuring accuracy, efficiency, and compliance across its portfolio.

Getting the code to run is the floor. What usually separates answers is the case checked without prompting: empty input, a single element, duplicate keys, or a value that overflows the integer type you chose.

Focus areas

Bound credential lifetime by the engagement end date
Scope every query by tenant below application code
Version an API you cannot force clients to upgrade

Overview

This role is highly collaborative and requires a bridge-builder mentality. You will work closely with civil and environmental engineers to translate real-world field challenges into scalable software solutions. By automating workflows and developing specialized internal tools, you help ARRO's project teams solve infrastructure problems more effectively. It is a position for someone who thrives on technical complexity and wants to see their code have a tangible, positive impact on public infrastructure.

The ARRO Consulting, Inc. interview process

ARRO Consulting, Inc. candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Discussions

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Team Meetings

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Final Decision

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

Tenant context leaking across a connection pool

Setting the tenant on a connection and relying on it for the rest of the request looks correct in every test that runs one request at a time. Under transaction pooling the connection goes back to the pool with that session state still attached, and the next checkout, serving a different client, inherits it; row-level security then enforces the previous tenant's policy flawlessly and returns the wrong rows. It reproduces only under concurrency, which is the load profile your integration tests do not have. Use a transaction-scoped setting, and assert that the setting matches the request's tenant immediately before the first query rather than trusting that it was set correctly upstream.

Validating against a client sandbox and assuming production parity

Sandboxes typically carry smaller data, looser or absent rate limits, a schema version behind production, and sometimes synchronous behaviour where production is asynchronous. Code that passes there fails first in the client's production, during a change window you do not control and often cannot get a second one of. The mitigations are specific: assert the observed schema version at the boundary of every run, measure the production rate limit empirically rather than reading it from a document, and design the first production run to be a bounded, reversible slice rather than a full backfill.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Trusting input because it came from your own front end

Anything crossing a trust boundary is hostile: parameterise queries instead of building SQL by concatenation, validate against an allow-list rather than a deny-list, and bound the size of anything you allocate from a request. Raising this unprompted in an API or design question is a cheap and unusually strong signal.

Questions & practice

Resolve effective-dated engagement versions for two million access checks

Hard · binary search · effective dating · complexity budget

The engagement table holds 500,000 effective-dated rows: (engagement_id, version, effective_from, effective_to which is NULL for the open version). A database exclusion constraint guarantees that per engagement_id the tstzrange(effective_from, effective_to) values never overlap, and the ranges are half-open. You must answer 2,000,000 access checks of the form (engagement_id, at_time) in one process with no database, returning the version open at that instant or none. Also report, per engagement, any coverage gap between adjacent versions. Give the complexity of the scan-per-check approach and say why it is not an option.

Approach

State the naive cost first: scanning the table per check is 2e6 x 5e5 = 1e12 range comparisons. Even at 1e9 simple comparisons per second that is roughly fifteen minutes of pure compare work, against a budget measured in seconds. It is correct, just unaffordable, and the fix is an index built once rather than a faster comparison.
Build a hash map engagement_id -> array of versions sorted by effective_from, once. O(V log V) to sort, O(V) memory. The build is amortised over 2e6 lookups, which is what makes it worth doing at all.
Answer each check by binary-searching for the last effective_from <= at_time, then confirming effective_to IS NULL OR at_time < effective_to. If the confirmation fails, the instant lies in a gap and the answer is none, not the neighbouring version. O(log v) per check, O(V log V + Q log v) total.
Name the precondition: predecessor search is only valid because the ranges per engagement do not overlap, which the exclusion constraint enforces with btree_gist for the equality operator. Without that guarantee you need an interval tree returning all stabbed intervals, at O(log n + k). Use the same half-open convention as the database, where at_time equal to effective_to belongs to the next version.
Find gaps with one O(V) pass over each sorted array comparing prev.effective_to against next.effective_from. The exclusion constraint forbids overlaps, not holes, so a hole is entirely possible and is where an access check fails closed for a reason nobody can see in the contract.
If the checks can be buffered, sort them by (engagement_id, at_time) and merge against the version arrays: O(Q log Q + V log V), usually a large constant-factor win over 2e6 random binary searches because the merge walks memory in order.

Follow-up

The open version has effective_to NULL. How do you keep both the sort and the comparison total without a sentinel that overflows your timestamp type?
A gap exists for one engagement and a check lands in it. What should the access path return, and what should it emit so someone fixes the amendment?
Amendments are being written while you answer checks. What read model keeps an answer self-consistent, and what does a check mean if the snapshot is a second stale?

Compare release versions and report the live estate spread

Easy · parsing · version skew · ordering

Each environment row carries reported_release as TEXT: either NULL, or a version of the form MAJOR.MINOR.PATCH with an optional -rc.N suffix. Up to 400 environments. Ordering rule: compare major, then minor, then patch numerically; a -rc.N build precedes the same MAJOR.MINOR.PATCH with no suffix, and rc numbers compare numerically. Return the lowest and highest reported versions, the minor-version spread between them, and the count of environments whose reported_release is NULL. NULL means the environment's version is unknown, not that it converged.

Approach

Parse each string once into a comparable tuple: (major, minor, patch, is_final, rc), where is_final is 1 for a plain release and 0 for a pre-release, and rc is 0 when absent. Tuple comparison then yields the stated order for free, including 3.10.0-rc.1 < 3.10.0. O(L) per string, O(n*L) total.
Track min and max in one linear scan rather than sorting: O(n) time, O(1) extra space, and n is only a few hundred so the parse dominates either way.
Fail loudly on a string that does not match the grammar instead of defaulting it to 0.0.0 or to the newest release. An unparseable reported_release is a reconciliation-ingest bug, and either default hides it in the direction that looks safe.
Compute the minor spread only when min and max share a major version. Across majors the difference of minor numbers is meaningless, so return the pair of versions and flag the condition rather than a number.
Report the NULL count as its own output line. Folding unknowns into 'converged' is how an install ends up running a release nobody knew was still live.

Follow-up

Two majors are live at once. What does 'spread' mean now, and what single number, if any, would you still publish?
reported_release is a claim from the environment, and last_heartbeat_at may be days old for a client-managed install. At what staleness does this figure stop being usable?
How would you enforce the supported window when the rollout plan is built, instead of reporting a violation after it already exists?

Measure credential exposure past engagement close without double counting

Medium · interval merge · sweep line · credential ttl

For one engagement you have credential_grant rows: (grant_id, principal_id, target_system_id, issued_at, not_after, revoked_at which may be NULL). The cutoff instant T is the engagement's ends_on plus access_grace_days. A grant is live over the half-open interval [issued_at, min(not_after, revoked_at)). There are up to 200,000 grants across target systems. For each target_system_id, return the total wall-clock time after T during which at least one grant was live, plus the disjoint segments that make it up. Grants that overlap must be counted once.

Approach

Clip each grant to [max(issued_at, T), min(not_after, coalesce(revoked_at, +inf))) and discard any interval whose start is not strictly before its end. O(n), and it removes every grant that already expired inside the engagement window.
Bucket the survivors by target_system_id and sort each bucket by start. O(n log n) overall, and sorting dominates the whole algorithm.
Sweep each bucket once, holding one open segment [s,e): if the next start is greater than e, emit [s,e) and open a new segment, otherwise set e = max(e, next_end). O(n) after the sort, O(1) working state, O(number of emitted segments) output.
Total exposure is the sum of emitted segment lengths, which is the measure of the union. Summing per-grant durations instead answers a different question and is unbounded above by the elapsed wall clock.
Produce a second, conservative figure that ignores revoked_at and uses not_after alone. revoked_at records that you asked for revocation; whether the client's identity provider honoured it is not something this table knows, and a self-contained token validated offline stays valid to its own expiry regardless.
If a bucket does not fit in memory, sort externally and sweep the stream, or push end timestamps into a min-heap keyed by end and pop those below the current start. Same O(n log n), memory proportional to the maximum number of concurrently live grants.

Worked solution (25 min)

Set T = 2026-06-30T00:00Z for one target system with four grants: g1 not_after 2026-06-29T12:00Z; g2 not_after 2026-07-01T00:00Z, revoked_at NULL; g3 not_after 2026-07-01T06:00Z, revoked_at 2026-06-30T18:00Z; g4 issued 2026-07-01T06:00Z, not_after 2026-07-02T06:00Z.
Clip: g1 becomes empty and drops out. g2 -> [Jun30 00:00, Jul1 00:00). g3 -> [Jun30 00:00, Jun30 18:00). g4 -> [Jul1 06:00, Jul2 06:00).
Sort by start and sweep: g2 and g3 merge into [Jun30 00:00, Jul1 00:00); g4 starts after that end, so it is emitted separately.
Union = 24h + 24h = 48h = 2,880 minutes, in two segments with a 6-hour uncovered gap between them. Summing the same three clipped intervals per grant instead gives 24 + 18 + 24 = 66h; the 18h excess is exactly the [Jun30 00:00, Jun30 18:00) stretch where g2 and g3 overlap.
Recompute ignoring revoked_at: g3 now runs to Jul1 06:00, which bridges the gap, and the union becomes the single segment [Jun30 00:00, Jul2 06:00) = 54h.

Expected result: Exposure after T = 2,880 minutes in segments [Jun30 00:00, Jul1 00:00) and [Jul1 06:00, Jul2 06:00). The conservative figure that ignores revoked_at is 3,240 minutes in one segment. Naively summing the clipped per-grant durations (24 + 18 + 24) would have reported 66 hours, more than the 54 hours that elapsed between T and the last expiry.

Verify it yourself

Union (48h) <= the clipped per-grant sum (66h) always, with equality only when no two clipped intervals overlap; here the 18h difference is the single g2/g3 overlap.
Union (48h) <= last expiry minus T (54h), since the union cannot exceed the elapsed span it lives in.
The emitted segments are strictly increasing and pairwise disjoint, and their count is 2, less than the 3 surviving grants.
g1 contributes nothing, confirming the clip ran before the sort rather than after it.

Follow-up

Three grants overlap and one of them belongs to an offboarded contractor. What must the sweep emit so exposure can be attributed per principal?
Which of your two numbers is the real bound on access if the client system validates tokens offline, and what does that imply about issuing TTLs in the first place?
Run this across 20,000 engagements as a nightly job with a fixed memory budget. What changes in the shape of the computation?

Soft-delete environments without breaking uniqueness or audit history

Easy · soft deletes · partial indexes · audit history

environment holds one row per provisioned deployment: environment_id, client_id, engagement_id, env_name, region, status, destroyed_at TIMESTAMPTZ NULL. Decommissioned environments stay readable for seven years, and env_name must be unique per client among live environments only, because a client may reuse a retired name. Design the constraint set and the history table, write the query that lists a client's live environments, and say what your design does about connector_run rows whose foreign key points at a destroyed environment.

Approach

Keep the tombstone as a nullable timestamp rather than a boolean: destroyed_at carries when as well as whether, and enforce live-only uniqueness with a partial index, CREATE UNIQUE INDEX env_live_name ON environment (client_id, env_name) WHERE destroyed_at IS NULL, which indexes only the live subset and so stays small.
Get the NULL semantics right before you pick a spelling. In a B-tree unique index two NULLs compare distinct by default, so plain UNIQUE (client_id, env_name, destroyed_at) enforces nothing at all among live rows — every live row is unique on the third column purely by virtue of being NULL. PostgreSQL 15's UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) does satisfy the requirement: the NULL live rows now compare equal and collide with each other, while retired rows carry distinct destroyed_at timestamps and stay distinct, so a name can be retired and reused any number of times.
Prefer the partial index anyway, and be able to say why rather than asserting it. It indexes only the live subset — tens of rows per client — instead of seven years of tombstones; it needs no version floor; and it keeps destroyed_at out of the key, so it cannot be defeated by two rows retired at the same instant. That last case is reachable: now() is transaction-start time, so retiring a name, recreating it and retiring it again inside one transaction writes two identical destroyed_at values and raises a duplicate-key error under the constraint form. clock_timestamp() avoids it if you take that route.
Put history in a separate append-only table written by an AFTER INSERT OR UPDATE OR DELETE trigger holding environment_id, changed_at, changed_by, op and the before and after row images. PostgreSQL has no system-versioned tables, so this is a trigger or a logical-decoding consumer, not a DDL flag, and the soft delete arrives at that trigger as an UPDATE rather than a DELETE.
State the foreign-key consequence rather than leaving it implicit: a soft delete leaves every referencing connector_run row resolvable, which is exactly what the audit requirement wants, but it means no read is protected by the database. Give the predicate one home, a view or an RLS-backed view that every caller uses, instead of repeating destroyed_at IS NULL across hundreds of call sites.
Size the retention: seven years of history plus tombstoned rows in the hot table. Partition the history table by month so expiry is a DETACH PARTITION rather than a DELETE that has to be vacuumed, and decide separately whether a residency-driven erasure request can be honoured against an append-only store at all.

Worked solution (20 min)

Write the DDL with destroyed_at TIMESTAMPTZ NULL plus the partial unique index on (client_id, env_name) WHERE destroyed_at IS NULL.
Insert two live rows sharing (client_id, env_name) and confirm the second is rejected; set destroyed_at on the first and confirm the insert now succeeds.
Build the same table a second time under UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) and confirm it passes the same two tests, so you are choosing between two working designs rather than rejecting a broken one.
Add environment_history (history_id, environment_id, changed_at, changed_by, op, row_before JSONB, row_after JSONB) and the AFTER trigger that populates it.
Write the live-listing query: SELECT environment_id, env_name, region, status FROM environment WHERE client_id = $1 AND destroyed_at IS NULL ORDER BY env_name.

Expected result: A partial unique index covering only live rows, a separate append-only history table with its own partitioned retention, and a listing query whose predicate matches the index predicate exactly so the planner can use it.

Verify it yourself

Run retire-and-recreate three times for the same name: three tombstoned rows and one live row coexist without violating the index.
On the NULLS NOT DISTINCT variant, retire and recreate the same name twice inside one transaction and watch it fail on duplicate destroyed_at, since now() does not advance within a transaction; the partial index has no such failure mode.
EXPLAIN the listing query and see the partial index; widen the predicate to destroyed_at IS NULL OR destroyed_at > now() and watch the index drop out, because that predicate no longer implies the index's.
Soft-delete a row and confirm the history trigger fired; a DELETE-only trigger records nothing, which is the common way this audit trail ends up empty.

Follow-up

What stops the next endpoint from forgetting the destroyed_at IS NULL predicate, and does that mechanism survive a developer writing raw SQL?
The history table stores row images as JSONB and the base table gains a column. What do old rows now mean, and how do you read them six years later?
A residency erasure request requires a genuine hard delete for one client. What breaks, and what do you keep?

Return the latest attempt per connector idempotency key

Easy · distinct-on · indexing · idempotency

connector_run stores one row per attempt: run_id, connector_id, idempotency_key, attempt_no SMALLINT, status (queued, running, succeeded, failed, ambiguous, cancelled, abandoned), failure_class, watermark_from, watermark_to, started_at, finished_at, unique on (connector_id, idempotency_key, attempt_no). For one connector_id, write the query returning the highest-numbered attempt per idempotency_key, keeping only those keys whose latest attempt is not succeeded, ordered by watermark_from. Give the index that serves it, and say what the plan does without that index.

Approach

Use DISTINCT ON (idempotency_key) with ORDER BY idempotency_key, attempt_no DESC. PostgreSQL requires the ORDER BY to lead with exactly the DISTINCT ON expressions, and that requirement is what makes the row picked deterministic rather than whichever the scan happened to reach first.
Apply the not-succeeded filter outside, in a wrapping SELECT. Putting it in the inner WHERE changes the question from 'keys whose latest attempt failed' to 'keys that have any non-succeeded attempt', which reports a key that failed twice and then succeeded, and is the single most common wrong answer here.
Know the portable alternative: row_number() OVER (PARTITION BY idempotency_key ORDER BY attempt_no DESC) in a subquery, filtered = 1 outside. A window function cannot be filtered in WHERE because WHERE is evaluated before the window, and PostgreSQL has no QUALIFY clause, so the subquery is mandatory rather than stylistic.
Index (connector_id, idempotency_key, attempt_no DESC). With connector_id fixed by equality, the index supplies the exact order DISTINCT ON needs and the Sort node disappears. It does not reduce rows read: through PostgreSQL 17 there is no loose index scan, so the executor still walks every attempt in range rather than skipping to each next distinct key.
Give the complexity honestly: without the index the plan is a scan plus an O(n log n) sort over the connector's attempts; with it, O(n) ordered reads and constant working memory, where n is attempts in range, not distinct keys. If the distinct keys are a tiny fraction of attempts, a recursive skip-scan is the next step.

Follow-up

attempt_no is not monotonic in time because a retry was queued out of order. Does your query still pick the right row, and which column should it actually order by?
How would you return the latest attempt per key across all connectors in one environment without the query degrading to a full sort?
An ambiguous latest attempt means the target may or may not have been written. What does this query feed, and should ambiguous be treated as failed here?

Explain a time you had to troubleshoot a technical issue that involved both hardware or field data and software components.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Walk us through a project where you had to design a system from scratch to address an operational bottleneck.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Share one client's rate budget across every job you run

Medium · rate-limiting · backoff · concurrency-control · connectors

A connector_binding carries an endpoint, an auth method and a rate budget. Several bindings, an ad-hoc backfill and an operator's debugging script all hit the same client-owned system, whose quota is per client account and shared across every one of them. Your workers are horizontally scaled and stateless. The client returns 429 with Retry-After, and sometimes 503 with nothing. Design the internal contract governing outbound calls: where the limit is enforced, what a worker asks for and gets back, how a 429 propagates, and what a run does when it cannot get budget.

Approach

Key the limiter on the resource actually being consumed: (client_id, target_system_id), never on worker, process or binding. A per-worker limit multiplies by the worker count, which is exactly wrong when the quota is one shared number, and it loosens silently the moment someone scales the deployment.
Make budget an explicit acquisition with a deadline rather than an implicit sleep. A worker calls acquire(key, cost, deadline) against a shared token bucket — a Redis Lua script keeps the read-modify-write atomic — and receives granted, a wait duration, or refused. Return the refusal rather than blocking a worker for minutes; a blocked worker is a worker not doing another client's work.
Add a concurrency bound alongside the rate bound, because they catch different failures. A token bucket happily permits a burst of fifty simultaneous in-flight requests that satisfies the per-second rate and still exhausts the client's connection pool. Hold a bounded lease per (client_id, target_system_id) with a TTL so a crashed worker's lease expires instead of wedging the binding.
Treat the client's own signal as authoritative and shared. A 429 with Retry-After must pause every caller on that key, not just the run that received it, so the limiter takes a cooldown on the key itself. A 503 with no Retry-After gets exponential backoff with full jitter — sleep = random(0, min(cap, base·2^attempt)) — because synchronised retries from a dozen bindings after a client outage are a self-inflicted herd against the quota you are about to need.
Adapt downward on evidence and upward only slowly. Cut the effective rate on repeated 429s, recover gradually, and record the measured sustained rate per binding: the documented limit and the enforced limit differ often enough that treating the client's documentation as fact is the specific error here.
Define what a refused run is. It is not a failure: record it as queued or abandoned with its own reason, leave the watermark untouched, and let the scheduler re-claim the same range later. Counting budget exhaustion as a failure pollutes the per-binding failure rate you alert on and buries the client-owned problems underneath it.

Worked solution (30 min)

Write the limiter's key and list every caller that shares it; name one that is easy to forget.
Write acquire()'s signature and its three outcomes, and say what the worker does with each.
Compute the worst-case in-flight request count under a rate limit alone at your burst size, then add the concurrency bound that fixes it.
Trace a 429 arriving at one of five concurrent runs on the same key and say what the other four do next.
Choose the run status and failure_class for budget exhaustion, then check your alerting query still isolates client-owned failures.

Expected result: A shared token bucket keyed on (client_id, target_system_id) with an atomic acquire, a separate leased concurrency bound with a TTL, a key-wide cooldown driven by the client's Retry-After, full-jitter backoff where no Retry-After is supplied, an empirically measured sustained rate per binding, and budget exhaustion recorded as a non-failure that leaves the watermark where it was.

Verify it yourself

Doubling the worker count leaves the observed request rate against the client unchanged.
One run receiving 429 stops the other runs on the same key within one acquisition interval.
A worker killed mid-request releases its concurrency lease with no operator involvement.

Follow-up

An operator's debugging script drains the whole budget at 09:00 and the scheduled runs starve. What in your design prevents that, and what priority scheme would you accept?
How do you measure the client's real limit without tripping it during their business hours?

Entitlement snapshot expires in lockstep across the fleet

Medium · cache stampede · ttl jitter · single-flight

Delivery requests across the estate return 403 for roughly 200 ms every five seconds. The entitlement service publishes one versioned snapshot; every caller caches it for five seconds and fails closed on a miss. Its request rate is near zero for most of each window and about 4,000 in a 300 ms burst, spread across the 800 application processes a rolling deploy restarted this morning. Its own handler service time is unchanged. Diagnose the shape, then give a fix that preserves fail-closed behaviour.

Approach

Establish the period before the cause. Plot the origin's request rate at one-second resolution: a sawtooth whose period equals the cache TTL, with the mass inside a few hundred milliseconds, says the misses are synchronised, not that traffic grew.
Measure two things next, in this order: key cardinality and in-flight concurrency per process. One key across the whole fleet means every process misses at the same instant. Four thousand requests from 800 processes means roughly five concurrent misses inside each process, so there is no single-flight and each in-flight request for the same key issues its own fetch.
Explain why the phases are locked: the rolling deploy populated every cache within the same few seconds, so 800 independent five-second TTLs expire together and nothing in the system ever scatters them again.
Fix three independent defects rather than one. Single-flight per key per process, so a process has at most one outstanding fetch for a key. A per-process randomised TTL, for example uniform over four to six seconds, so expiry decorrelates after any synchronising event. Refresh-ahead at about 80 percent of the TTL, so the hot key is replaced before it expires and a request never waits on a miss at all.
Keep fail-closed honest about what staleness costs. Serving a stale snapshot on a fetch error is safe for the engagement end-date case, because ends_on travels inside the snapshot and the caller evaluates it locally. It is not safe for a mid-engagement suspension, which only arrives as an invalidation event. So bound the stale window explicitly at tens of seconds, fail closed beyond it, and state the resulting suspension propagation delay as the accepted price.
Name the new steady state so nobody is surprised by it: compute it from the refresh interval rather than the TTL, because refresh-ahead is what now sets the period. Refreshing at 80 percent of a 5-second TTL means each process fetches every 4 seconds, assuming the refreshed entry restarts the TTL clock, so the floor is fleet size divided by the refresh interval: 800 / 4 = 200 requests per second, not the 800 / 5 = 160 that the TTL alone would suggest. Jitter nudges it up rather than down, because the rate averages 1/T and not 1/mean(T): with TTL uniform over 4 to 6 seconds the floor is about 203 requests per second. A process that stops asking for the key stops refreshing it, so 200 is the floor for a uniformly hot fleet and an upper bound for any other traffic mix. Verify by replaying 800 synchronised clients and confirming the origin sees a flat rate at that level with no burst above a stated ceiling.

Follow-up

At 8,000 processes the steady refresh floor becomes 8,000 / 4 = 2,000 requests per second, set by process count and refresh interval alone and independent of user traffic. What changes in the design?
A client is suspended for non-payment at 09:00. Walk the worst-case time until every process stops serving them, with your stale window in place.
How would you detect the next stampede automatically, given that the origin's average rate barely moves when one occurs?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

How do you handle feedback from senior engineers who may not have a software background?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

How do you prioritize features when you are working with stakeholders who have competing technical needs?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Tell us about a time you had to explain a complex technical concept to a non-technical stakeholder.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

Describe a time you worked on a project that required cross-functional collaboration. How did you ensure everyone stayed aligned?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?