Interview guides · Ayla Networks

Ayla Networks Software Engineer Interview Questions & Guide 2026

Code reading practice for Ayla Networks software engineer prep: work out what unfamiliar code does before judging it.

The 60-second brief

As a Software Engineer at Ayla Networks, you are at the heart of the Internet of Things (IoT) revolution. This role is critical to building and maintaining the scalable, cloud-based platforms that enable manufacturers to connect their products to the internet. You will be working on complex, distributed systems that bridge the gap between hardware devices and cloud services, ensuring reliability, security, and seamless data flow.

Ask whether any round happens inside an existing repository instead of a blank file. Reading unfamiliar code, isolating a fault and making the smallest correct change is a different skill from writing a function from scratch, and it needs its own practice.

Focus areas

Bound every outbound call with a timeout
Trace a symptom to a mechanism under load
Detect concurrent edits instead of losing writes

Overview

Your work will directly influence how millions of devices interact with the Ayla Networks ecosystem. You will engage with challenging problem spaces, including high-concurrency environments, data management, and system architecture. This is a role for engineers who thrive on technical rigor and are eager to solve the unique challenges posed by massive-scale device connectivity.

The Ayla Networks interview process

Ayla Networks candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Phone Screens

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Rounds

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Where candidates lose points

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Assuming the bug is in the framework

Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Questions & practice

Describe bitwise operations and when they are most efficient.

Medium · data structures and algorithms

Approach

Walk one small example through your approach before writing the whole thing.
State the target complexity and say which constraint rules the naive version out.
Choose the data structure from the access pattern, not from familiarity.

Follow-up

Which test case would catch an off-by-one here?
How does this change if the input no longer fits in memory?

Explain the implementation and use cases for linked lists.

Medium · data structures and algorithms

Approach

Restate the input: its shape, its size, and what is guaranteed about it.
State the target complexity and say which constraint rules the naive version out.
Walk one small example through your approach before writing the whole thing.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

What are the fundamental differences between a switch and a router?

Medium · data structures and algorithms

Approach

Restate the input: its shape, its size, and what is guaranteed about it.
State the target complexity and say which constraint rules the naive version out.
Choose the data structure from the access pattern, not from familiarity.

Follow-up

Which test case would catch an off-by-one here?
What is the worst case, and how likely is it on real data?

Explain the core concepts of multithreading in a high-concurrency environment.

Medium · languages, concurrency and fundamentals

Approach

Name what is shared across threads and what owns each piece of state.
Reach for the cheapest primitive that closes the race, not the broadest lock.
Say what the runtime actually does before reasoning about the code.

Follow-up

Where could this allocate more than you expect?
What happens if two callers reach this at the same time?

What strategies do you use for managing state across multiple threads?

Medium · languages, concurrency and fundamentals

Approach

Identify the window where an invariant is briefly untrue.
Distinguish a value from a reference to it, and say which one you handed out.
Reach for the cheapest primitive that closes the race, not the broadest lock.

Follow-up

How would you prove the race exists rather than suspect it?
Where could this allocate more than you expect?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.

Worked solution (25 min)

Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.

Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.

Verify it yourself

The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
The heap result matches a full sort of the tenant sums on a sample.
The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).

Follow-up

The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Worked solution (25 min)

Seed one tenant with 500k active resources, 2% of them sharing an identical updated_at.
Time LIMIT 50 OFFSET 0 against OFFSET 20000 and record rows-read from EXPLAIN (ANALYZE, BUFFERS) for each.
Page the whole set with the keyset query while an insert-only writer adds 100 rows/second, collecting resource_ids, and repeat the run with OFFSET.
Repeat both runs under a second writer profile that also deletes 20 rows/second from pages already returned and bumps updated_at on 20 more, and diff each collected id set against the rows that existed for the whole run.
Remove resource_id from the cursor so the seek degrades to updated_at < $2, and re-run the tie-heavy section of the feed.

Expected result: Keyset reads about 50 index entries per page and its latency is flat from page 1 to page 10,000; under both writer profiles the collected ids contain no duplicate and omit no row that existed for the whole scroll. OFFSET at page 400 reads 20,050 rows to return 50. Under the insert-only writer it repeats rows and skips none — an insert can only push a row down past the boundary — and only the second profile, whose deletes and updated_at bumps move rows up past the cursor, also makes it skip. Without the tie-break, the seek is updated_at < $2, which drops every still-unread row sharing the boundary timestamp.

Verify it yourself

EXPLAIN shows an Index Scan with no Sort node — a Sort means the ORDER BY is not satisfied by the index.
Buffers read per page stays constant as page depth grows.
Ids collected across the scroll are unique, and complete for rows untouched during the run.

Follow-up

The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
What does the cursor do when the row it points at has since been deleted?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.

Follow-up

Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
The consumer claims it never received event 4,812,006. What do you look at, in what order?

Convert the listing endpoint from offset pages to stable cursors

Medium · pagination · cursors · index access path

GET /v1/resources returns a tenant's resources newest-updated first, today with page and per_page, backed by index (tenant_id, status, updated_at DESC, resource_id DESC). Callers are a browser feed and a nightly sync job that walks every page. Users report items appearing twice or vanishing between pages, and page 400 is slow. Design the cursor contract: what the cursor contains and how it is encoded, the exact WHERE and ORDER BY, what happens when a row's updated_at changes mid-walk, how a client detects the end, and what the sync job does when a cursor is rejected.

Approach

Separate the two defects, because they have different fixes. Cost: OFFSET n makes the engine produce and discard n rows, so price grows with page depth rather than page size and page 400 pays for 400 pages of work. Correctness: while the set shifts, rows cross the offset boundary and are skipped or repeated, and nothing in the response lets the client detect it.
Write the seek: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($k, $id) ORDER BY updated_at DESC, resource_id DESC LIMIT n. The tie-break is not decoration - updated_at is not unique, and two rows sharing a timestamp across a page boundary reintroduce exactly the skip this was adopted to remove. PostgreSQL seeks the composite index on the row-value comparison directly; on an engine that does not optimise a row constructor, expand it into the equivalent OR form or the plan quietly degrades to a scan.
Encode the cursor as an opaque token carrying the sort key, the id, and a fingerprint of the filter and sort order, signed or at minimum validated. A cursor replayed against a different sort or filter must be a 400 with its own code, not a silently wrong page - the sync job cannot notice the difference otherwise.
State the guarantee precisely rather than generously. Keyset is stable against inserts and deletes elsewhere in the set, because the position is a value and not a count. It is not a snapshot: updated_at is mutable, so a row that is edited during the walk re-sorts and may be seen twice or not at all. If the sync job needs exactly-once coverage, order on an immutable key such as (created_at, resource_id), or walk resource_revision by revision_id and treat updated_at as data.
Define termination and limits in the response, not in the client's inference: fetch n+1 rows, return n, and emit next_cursor only when the extra row existed. Absence of next_cursor is the sole end signal, because a page can legitimately come back short when rows are filtered after retrieval. Cap n and document the cap rather than honouring per_page=10000.
Migrate without a flag day: keep page and per_page working, add the cursor, count usage per credential, and remove the offset path only once the sync job's traffic on it is zero.

Follow-up

The client wants a total count and the ability to jump to page 400. What can you honestly offer instead, and what does each option cost?
How do you paginate backwards, and what does that require of the index?

Keep one tenant's bulk export from starving projection updates

Medium · backpressure · fair scheduling · head-of-line blocking · leases

The worker fleet runs about 600 jobs/second out of job_run, claimed under a lease and heartbeated while running. Most handlers finish under 200 ms; exports run for minutes. One tenant enqueues 50,000 exports. Design the scheduling so that tenant's exports slow down and nothing else does: the queue split, the concurrency caps and where they are enforced, the claim query, and the backpressure signal that stops accepting new work. State the in-flight worker count each class needs at its arrival rate, and what the cap costs the tenant that hits it.

Approach

Do the concurrency arithmetic first, because it determines the pool shapes. Required in-flight work is arrival rate times service time: 600/second of 200 ms handlers needs about 120 workers, while one export per second at 180 seconds needs 180 on its own. The long tail dominates any pool it shares, so a single pool sized from the mean is consumed by exports while 50 ms projections queue behind them. That is head-of-line blocking, and adding capacity does not fix it because the ratio is what is wrong.
Split by job class rather than by priority. Priority inside one pool still lets a running export hold its worker for the next four minutes - there is no preemption for a handler already executing. Separate queues and separate worker processes give each class a floor the other cannot take.
Enforce the per-tenant cap inside the claim, not at enqueue. Claim with SELECT ... WHERE status='queued' AND run_after <= now() AND job_type=$1 ORDER BY run_after, job_run_id FOR UPDATE SKIP LOCKED LIMIT 1, restricted to tenants currently under their cap. Derive the running count from rows in status 'running' with a live lease rather than from a counter incremented at claim and decremented at completion: the derived count self-heals when a worker dies, and the counter leaks a slot every time one does.
Make the ordering fair over tenants instead of over jobs. Strict FIFO with a cap still forces the claim to scan past 50,000 rows belonging to a capped tenant before it finds anyone else's work, so cost grows with backlog depth. Pick the tenant first from a small per-tenant queue-depth summary, then claim within that tenant, which makes the scan proportional to the number of active tenants instead.
Close the loop with backpressure and deduplication. Reject or defer enqueues once a class's depth passes a bound, and rely on UNIQUE (job_type, dedupe_key) WHERE status IN ('queued','running') so a retrying producer collapses into one row instead of multiplying the backlog. Watch oldest-queued-age per class rather than depth: depth means nothing without a service rate, while age is the user-visible latency directly.

Worked solution (25 min)

Compute required in-flight workers per class from arrival rate times service time, and show what one shared pool does when both classes contend.
Write the claim statement with the per-tenant restriction, and state how the running count is derived so it survives a worker being killed.
Trace what the claim scans when one tenant holds 50,000 queued rows, under FIFO and under tenant-first selection.
Choose the backpressure threshold and the signal it watches, then state what the producer receives when it trips.

Expected result: Separate queues and worker pools per job class, sized from rate times service time (about 120 for 200 ms work, about 180 for exports). A per-(job_type, tenant) cap enforced inside the claim with FOR UPDATE SKIP LOCKED, with the running count derived from live leases so it self-heals. Tenant-first selection keeps claim cost proportional to active tenants rather than backlog depth. Enqueue is rejected on class depth, duplicates collapse on dedupe_key, and the alarm is oldest-queued-age per class.

Verify it yourself

A tenant with 50,000 queued exports never occupies more than its cap's worth of workers.
Claim cost does not grow as that tenant's backlog grows.
Killing a worker mid-job returns its cap slot within the lease interval with no manual step.
Projection p99 is unchanged while the export backlog drains.

Follow-up

An export outruns its lease, so two copies run. What must the handler do to make the second copy harmless, and what does it write to guarantee that?
With a cap of 4, the tenant's 50,000 exports now take hours. What do you tell them, and does the answer change if they are the largest tenant on the platform?
Queue depth is flat but oldest-queued-age is climbing. Name the mechanisms that produce that exact pair of signals.

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation · retry amplification

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
Contain in order: park the offending host so the shared pool drains, add a per-resolved-host concurrency cap alongside the per-destination one, give slow hosts their own queue so they cannot occupy the general pool, derive the timeout from the delivery deadline rather than a round number, and check the retry policy is not tripling load on a host that is already slow. Use backoff with full jitter so retries do not resynchronise on recovery.
State the invariant you are restoring: one tenant's endpoints degrade only that tenant's deliveries. That is a property to load-test for, not to assume from a config value.

Follow-up

The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
Where should the 10-second timeout number actually come from?
If that tenant had one destination row instead of 900, would the cap of 4 have saved you? What would you measure to be sure?

Suggested practice checklist

Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.

Day 1 — Fix the scope and take a cold baseline

Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
Set the week's rule: one thing finished every night, including the night you only have forty minutes.

Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One pattern, written three times from blank

Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.

Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.

Day 3 — One design, only to the depth you can defend

Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
Write the one tradeoff you are choosing against and the observation that would make you reverse it.

Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.

Day 4 — Only the fundamentals you will have to defend

Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.

Deliverable: Three written answers, each under two hundred words and each timed aloud.

Day 5 — Your own work, timed

Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in the weekend block

Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
Re-read only your own notes from this week and open no new material.
Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.

The bug you decided not to fix

Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.

How do you handle concurrency issues in a distributed system?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.

Follow-up

How would you detect a consumer that reads the field only during a monthly export?
One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
After removal, what makes the change irreversible, and how long before you cross that line?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

What in that decision was irreversible, and did you know it was irreversible when you made it?
How did you tell the people who had already built on top of the original decision?
What do you now measure before committing to a change of this size?