Interview guides · Kaztronix

Kaztronix Software Engineer Interview Questions & Guide 2026

Software engineer prep for Kaztronix: design exercises weighted toward the constraints of product and business analytics work.

The 60-second brief

A Software Engineer at Kaztronix operates at the intersection of high-level technical architecture and mission-critical implementation. Whether you are working on communications network engineering or specialized programming roles, your work serves as the backbone for complex, large-scale systems that require both precision and scalability. This role is not just about writing code; it is about architecting solutions that solve real-world infrastructure and data challenges.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

Make every write idempotent under retry
Trace a symptom to a mechanism under load
Choose indexes from the query's access path

Overview

By joining Kaztronix, you become a vital contributor to projects that demand high reliability and technical rigor. You will frequently collaborate with cross-functional teams to translate abstract requirements into robust, functional software. The environment is fast-paced and intellectually demanding, offering you the opportunity to influence the trajectory of significant technical initiatives while honing your skills in a professional, mission-driven organization.

Be prepared to discuss how your technical background directly applies to the specific domain of the team you are joining, whether that is network communications or statistical programming.

The Kaztronix interview process

Kaztronix candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Assessments

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Assessments

This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.

What to demonstrate

Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage

How to prepare

For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend

Final Decision

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Questions & practice

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).

Follow-up

The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.
Choose the late-event policy from what the projection is keyed on. The projection upserts on (aggregate_id, aggregate_version) and discards a version it has already applied, so a late event is safe to apply out of order and correctness never depended on the merge at all. Apply it, recompute the affected feed page, and count lateness so the 30-second budget can be re-derived from data rather than folklore.
Say what the merge does not buy: ordering is guaranteed within one aggregate by the log's partitioning, and no watermark makes the cross-aggregate order authoritative. Two events from different aggregates in the same millisecond have no true order, so the feed's order is a presentation choice that must be stable rather than correct.

Follow-up

The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?
One partition is ten minutes behind because its producer is slow. Do you stall the feed or emit without it?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.

Worked solution (25 min)

Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.

Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.

Verify it yourself

The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
The heap result matches a full sort of the tenant sums on a sample.
The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).

Follow-up

The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Worked solution (25 min)

Seed one tenant with 500k active resources, 2% of them sharing an identical updated_at.
Time LIMIT 50 OFFSET 0 against OFFSET 20000 and record rows-read from EXPLAIN (ANALYZE, BUFFERS) for each.
Page the whole set with the keyset query while an insert-only writer adds 100 rows/second, collecting resource_ids, and repeat the run with OFFSET.
Repeat both runs under a second writer profile that also deletes 20 rows/second from pages already returned and bumps updated_at on 20 more, and diff each collected id set against the rows that existed for the whole run.
Remove resource_id from the cursor so the seek degrades to updated_at < $2, and re-run the tie-heavy section of the feed.

Expected result: Keyset reads about 50 index entries per page and its latency is flat from page 1 to page 10,000; under both writer profiles the collected ids contain no duplicate and omit no row that existed for the whole scroll. OFFSET at page 400 reads 20,050 rows to return 50. Under the insert-only writer it repeats rows and skips none — an insert can only push a row down past the boundary — and only the second profile, whose deletes and updated_at bumps move rows up past the cursor, also makes it skip. Without the tie-break, the seek is updated_at < $2, which drops every still-unread row sharing the boundary timestamp.

Verify it yourself

EXPLAIN shows an Index Scan with no Sort node — a Sort means the ORDER BY is not satisfied by the index.
Buffers read per page stays constant as page depth grows.
Ids collected across the scroll are unique, and complete for rows untouched during the run.

Follow-up

The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
What does the cursor do when the row it points at has since been deleted?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Follow-up

The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

How do you ensure your code remains maintainable and scalable as project requirements evolve?

Medium

Approach

State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Describe your process for debugging a complex system failure in a production environment.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Shard by tenant when one tenant outgrows a single shard

Hard · sharding · hot partition · routing · online migration

One primary holds resource, resource_revision, outbox_event and idempotency_key for every tenant and is at its write ceiling at 1.2k writes/second. tenant_id leads every index. Shard across eight primaries. One tenant holds 22% of all rows and by itself exceeds a single shard's write capacity. Design the routing, the split of that tenant, and the online move of a tenant between shards with writes continuing. State what breaks for queries that are tenant-scoped today, and exactly what a write must do when it arrives at the old shard after the cutover.

Approach

Route on a unit smaller than a tenant from the start. Make the routing key (tenant_id, bucket) with a fixed bucket count - 64 over eight shards - and keep a directory mapping each (tenant_id, bucket) to a shard, carrying a version and cached in every service. An ordinary tenant has all 64 buckets pointing at one shard and behaves exactly as it does today; only the hot tenant has its buckets spread. Hashing tenant_id alone spreads tenants evenly, gives you no way to move one, and has no answer at all for a tenant larger than a node. The bucket count is the part you cannot change later without rehashing rows, so pick it well above the shard count and rebalance by moving buckets, not by re-bucketing.
For the tenant that exceeds one node, its buckets must land on different shards - that is the whole point of bucketing it, and buckets confined to its own shard would rename the rows while leaving every write on the node whose ceiling it already exceeds. Size it from measured numbers rather than from its row share: 22% of rows says nothing about write rate. The current primary tops out near 1.2k writes/second on this hardware and workload, so a tenant peaking at W writes/second needs its buckets spread over at least ceil(W / headroom-per-shard) shards, where the divisor is the share of each shard's ceiling you are willing to give it while that shard still serves other tenants - not the full 1.2k. Size on its peak, not its mean.
Fix the co-location invariant at the right grain. What must commit in one transaction is a resource, its resource_revision row and its outbox_event row, so the bucket is a property of the resource: derive it once at creation and stamp it into resource_id, and every later revision and event routes with its parent for free. Per-tenant co-location was never the requirement, and mistaking it for one is what makes a tenant look unsplittable. What genuinely breaks is an invariant spanning two resources of one tenant - a per-tenant counter, uniqueness across its resources - which now needs either a home-shard table or two-phase commit, and 2PC at this write rate is not a serious option.
Keep the idempotency constraint arbitrating, because it is now enforced per shard. PRIMARY KEY (tenant_id, idempotency_key) only continues to reject a retry if the same key always lands on the same shard, so derive the create-path bucket from hash(tenant_id, idempotency_key) and mint the new resource_id inside that bucket, which also puts the key row and the resource it guards in one transaction. A bucket chosen from anything that differs between a request and its retry - a timestamp, the worker id, a client-supplied resource id - splits one key across two shards, both inserts succeed, and the write endpoint's retry safety is silently gone.
State what the split costs the hot tenant's reads. Its listing, one 21-entry index scan today, becomes a scatter-gather: every bucket-shard returns 21 rows, a coordinator merges and discards the surplus, latency becomes the slowest shard's rather than the median's, and the keyset cursor has to carry a position per bucket instead of one (updated_at, resource_id) pair. Counts over that tenant fan out the same way. The relay becomes one leader per shard; consumers are unaffected because their ordering guarantee was always per aggregate and a resource's events never leave its bucket.
Move one bucket at a time, reversibly, and fence the straggler at the shard rather than at the caller. Copy from a snapshot while the bucket stays read-write, tail changes until the remaining delta is a few seconds of writes, fence writes for that (tenant_id, bucket) alone with a retryable status, apply the final delta, bump the routing version. Scoping the fence to a bucket is what makes a seconds-long freeze affordable. Then have each shard store the routing epoch it believes it holds for each (tenant_id, bucket) and reject any write carrying an older one: without that token, a service on a stale map commits successfully to a database nothing will ever read again, and the loss stays invisible for days. Outside the data path, anything that aggregated across tenants in one query - admin reporting, the A-Z index, global counters - becomes a fan-out across eight shards with a merge, and per-tenant uniqueness survives only on tables that stay whole on the tenant's home shard.

Worked solution (40 min)

Write the routing lookup keyed by (tenant_id, bucket), its version field, where it is cached and invalidated, and the request-path cost.
From the tenant's measured peak write rate and the per-shard headroom you will grant it, compute how many shards its buckets must span, assign them, and show no single shard carries its whole write rate.
Trace one create end to end: which value picks the bucket, where resource_id gets it stamped, and why the revision, outbox and idempotency rows land on the same shard.
Write the move steps for one bucket, then the epoch check the shard performs on every write, and trace a stale-map write through it.

Expected result: A versioned (tenant_id, bucket) to shard directory cached in every service, with ordinary tenants pointing all their buckets at one shard. The hot tenant's buckets are deliberately assigned to different shards, enough of them that its peak write rate divided across them fits the headroom granted on each, because no single node can carry it. The bucket is derived once and stamped into resource_id, so a resource, its revisions and its outbox rows commit together on one shard while the tenant spans several; only cross-resource invariants within the tenant lose their single-shard transaction. The create-path bucket comes from hash(tenant_id, idempotency_key) so the per-shard unique constraint still arbitrates retries. The tenant's listing becomes a scatter-gather with a per-bucket cursor. Migration moves one bucket at a time - snapshot copy, tail, seconds-long fence for that bucket, final delta, routing-version bump - and a routing epoch per (tenant_id, bucket) is checked by the shard so a stale caller's write is rejected rather than committed where nothing will read it.

Verify it yourself

The hot tenant's buckets resolve to more than one shard, and its peak write rate split across them is inside the headroom each shard has after its other tenants.
A resource's rows, its revisions, its outbox rows and its idempotency key row always resolve to a single shard, so no one write spans two.
The same (tenant_id, idempotency_key) routes to the same bucket on every retry, so the per-shard unique constraint still rejects the duplicate.
A write carrying an old routing epoch is rejected by the shard rather than merely misrouted by the caller.

Follow-up

The fence lasts 90 seconds instead of 4 because the final delta keeps growing. What is happening, and what do you do while the tenant is fenced?
Two tenants must merge into one account. What does that cost under this scheme, and which step is not reversible?
A shard is lost entirely. Which tenants are affected, and what is the source of truth for rebuilding them?

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation · retry amplification

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
Contain in order: park the offending host so the shared pool drains, add a per-resolved-host concurrency cap alongside the per-destination one, give slow hosts their own queue so they cannot occupy the general pool, derive the timeout from the delivery deadline rather than a round number, and check the retry policy is not tripling load on a host that is already slow. Use backoff with full jitter so retries do not resynchronise on recovery.
State the invariant you are restoring: one tenant's endpoints degrade only that tenant's deliveries. That is a property to load-test for, not to assume from a config value.

Follow-up

The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
Where should the 10-second timeout number actually come from?
If that tenant had one destination row instead of 900, would the cap of 4 have saved you? What would you measure to be sure?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

Describe a situation where you disagreed with a team member’s technical approach. How did you resolve it?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

What steps do you take to mentor junior engineers or improve team processes?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

Tell me about a time you had to explain a complex technical issue to a non-technical stakeholder.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Give an example of a time you took ownership of a project that was failing or behind schedule.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?