Interview guides · KGS

KGS Software Engineer Interview Questions & Guide 2026

System design practice for KGS software engineer candidates: reach a data model and an API contract, not a box diagram.

The 60-second brief

At KGS Group, the Software Engineer role supports the firm's multidisciplinary engineering consulting work. KGS Group is fundamentally an engineering firm, and it relies on software, automation, and data systems in its hydroelectric, utility, and infrastructure projects. You will be responsible for developing the tools and digital frameworks that enable the firm's structural, electrical, and civil teams to deliver high-quality, complex designs.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

Scope every query by tenant below application code
Diagnose failures inside environments you cannot instrument
Evolve schemas expand-contract across un-upgradable client deployments

Overview

Your work will directly influence the efficiency and precision of KGS Group's project delivery. Whether you are building automation tools for engineering calculations, integrating sensor data from hydroelectric facilities, or developing interfaces for the firm's project management systems, your contributions directly impact the safety and reliability of critical infrastructure. Candidates report that the role calls for problem-solvers, not just coders, who can translate abstract engineering requirements into robust, scalable software solutions.

The KGS interview process

KGS candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Assessments

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Team Meetings

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Final Selection Stages

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Treating an ambiguous failure as a definite one

A timeout, a 502 from an intermediary, or a connection reset after the request bytes were sent all leave the target's state unknown. Classifying those as failures and retrying duplicates the effect; classifying them as successes and advancing the watermark loses data silently. Both wrong answers are common because the ambiguous case is rare in a sandbox and routine in production. The run needs a distinct ambiguous state, a dedupe key that makes the retry safe, and a reconciliation read against the target when the key alone cannot settle it.

Long-lived shared credentials for client systems

One static key used by the connector runtime, a debugging script and three engineers cannot be attributed to a person in an audit, cannot be expired at engagement close, and cannot be rotated without breaking an unknown number of consumers at once. The expensive part of the eventual incident is not the leak; it is that rotation requires finding every holder under time pressure, and nothing recorded who they were. Issue per-principal, per-target, short-lived grants from a broker so rotation is a no-op, attribution is automatic, and closure is a TTL rather than a search.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Starting work without saying what you are about to spend time on

State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.

Questions & practice

Find peak window utilisation and the first rejected request

Medium · sliding window · two pointers · rate limiting

One connector binding's requests against a client endpoint arrive as timestamps t[0..n-1] in milliseconds, sorted non-decreasing, n up to 1,000,000. The endpoint's budget is R requests per rolling window of W milliseconds, and a request at time t is admitted only if strictly fewer than R already-admitted requests fall in the half-open window (t-W, t]. Return (a) the maximum number of requests falling in any W-millisecond window, counting every request regardless of admission, and (b) the index of the first request the limiter would reject. Both parts in O(n) time.

Approach

Part (a) is a two-pointer scan: for each right index, advance left while t[right] - t[left] >= W, which leaves the window exactly (t[right]-W, t[right]], and track max(right-left+1). Left is monotone non-decreasing, so total work is O(n) amortised with O(1) extra space. The maximum over all windows of width W is attained at a window whose right edge is a request timestamp, so scanning right edges is sufficient.
Part (b) is not the same scan. A rejected request consumes no budget, so the window must count admitted requests only. Keep a FIFO deque of admitted timestamps: pop from the front while front <= t - W, admit if the deque holds fewer than R, and return the first index where it does not. O(n) total pushes and pops, O(R) memory.
Pin the boundary convention before writing either loop. With a half-open (t-W, t] window a request exactly W milliseconds after an earlier one does not see it; switching to a closed window changes the answer at every boundary and is where most disagreements with the client's own limiter come from.
Note what fixed buckets of width W buy and cost: O(1) memory, but they admit up to 2R inside a span shorter than W, R at the end of one bucket and R at the start of the next. If buckets are unavoidable, halve the bucket width or use a weighted estimate across two adjacent buckets.
If part (a) already exceeds R, the binding cannot run at this concurrency under any limiter shape. That is a scheduling problem, not a retry-policy problem, and no backoff tuning fixes it.

Follow-up

Three bindings and four workers share this one endpoint's quota. Where does the counter live, and what is the correct behaviour when that store is unreachable?
Add full-jitter backoff to the retry path and state precisely what it prevents that plain exponential backoff does not.
The client documents the limit as 'about 100 per minute'. How would you measure the real one without tripping it in their production?

Top-k longest connector runs from a stream too large to sort

Medium · heap · top-k · streaming

A day of connector_run rows sits in object storage: 200,000,000 rows, unsorted, far larger than memory, and streamable once. Each row has run_id, connector_id, started_at and finished_at, where finished_at is NULL for runs that never reached a terminal state. Return the 200 longest terminated runs by (finished_at - started_at), ties broken in favour of the lower run_id, plus the count of rows with a NULL finished_at. You may not sort the input or hold it in memory. State the time and space complexity you achieve.

Approach

Keep a bounded min-heap of capacity k=200 ordered on (duration, -run_id). The root is then the weakest retained entry: smallest duration, and on a duration tie the largest run_id, which is exactly the one the stated tie-break should evict.
Per row: if finished_at is NULL, increment a counter and skip; otherwise compute the duration and push when the heap holds fewer than k, else compare against the root and replace only if strictly better. O(n log k) worst case, O(k) space, and after the heap warms nearly every row costs a single comparison against the root.
Reject the alternatives explicitly. A full sort is O(n log n) and at 2e8 rows means an external merge sort over hundreds of gigabytes to produce 200 rows. Quickselect is O(n) average but needs the whole array resident, which the constraint forbids.
Handle the NULL as data, not as an edge case: depending on the language, subtracting from NULL either throws or yields a sentinel that lands at the top of the list. Count those rows and report them, because a spike of unterminated runs is its own signal.
To parallelise, give each of p shard readers its own size-k heap and merge the p results. The union of per-shard top-k sets contains the global top-k, because an element beaten by fewer than k rows globally is beaten by fewer than k rows in its own shard, so the merge is exact at O(pk log k).

Worked solution (20 min)

Stream eleven rows as (run_id, duration_seconds): (1,12) (2,405) (3,7) (4,88) (5,405) (6,3) (7,1200) (8,61) (9,NULL) (10,99) (11,405). Use k=3.
Fill the heap with the first three terminated rows, then maintain it: after (2,405) and (5,405) and (7,1200) have been seen, the heap holds those three and the root is (405, -5), which is run 5.
Row 9 has no finished_at, so it increments the unterminated counter and never reaches the heap.
Row 11 has duration 405, equal to the root's duration, but run_id 11 > 5, so under (duration, -run_id) it is not strictly better than the root and is discarded on the tie-break rather than on duration.
Drain the heap in descending order to produce the final list.

Expected result: Top 3 = run 7 (1200s), run 2 (405s), run 5 (405s). Unterminated count = 1 (run 9). Run 11, also at 405s, is excluded by the lower-run_id tie-break.

Verify it yourself

The heap never holds more than k=3 entries at any point in the stream.
The heap root is always the weakest retained entry, so every discarded row compared unfavourably against it.
Brute-force sorting the 10 terminated rows and taking the first 3 gives the same set, including the same tie resolution.
Splitting the same stream into two shards with independent size-3 heaps and merging yields the identical top 3.

Follow-up

Now return the top 20 bindings by total run time instead of the top runs. What changes, and how many distinct keys must you hold to make it exact?
The key space is too large to hold an exact aggregate. What does a Misra-Gries summary with m counters actually guarantee about the counts it returns?
Rows now arrive continuously and you want a rolling one-hour top-k. Does the bounded heap still work, and if not, what breaks?

Compare release versions and report the live estate spread

Easy · parsing · version skew · ordering

Each environment row carries reported_release as TEXT: either NULL, or a version of the form MAJOR.MINOR.PATCH with an optional -rc.N suffix. Up to 400 environments. Ordering rule: compare major, then minor, then patch numerically; a -rc.N build precedes the same MAJOR.MINOR.PATCH with no suffix, and rc numbers compare numerically. Return the lowest and highest reported versions, the minor-version spread between them, and the count of environments whose reported_release is NULL. NULL means the environment's version is unknown, not that it converged.

Approach

Parse each string once into a comparable tuple: (major, minor, patch, is_final, rc), where is_final is 1 for a plain release and 0 for a pre-release, and rc is 0 when absent. Tuple comparison then yields the stated order for free, including 3.10.0-rc.1 < 3.10.0. O(L) per string, O(n*L) total.
Track min and max in one linear scan rather than sorting: O(n) time, O(1) extra space, and n is only a few hundred so the parse dominates either way.
Fail loudly on a string that does not match the grammar instead of defaulting it to 0.0.0 or to the newest release. An unparseable reported_release is a reconciliation-ingest bug, and either default hides it in the direction that looks safe.
Compute the minor spread only when min and max share a major version. Across majors the difference of minor numbers is meaningless, so return the pair of versions and flag the condition rather than a number.
Report the NULL count as its own output line. Folding unknowns into 'converged' is how an install ends up running a release nobody knew was still live.

Follow-up

Two majors are live at once. What does 'spread' mean now, and what single number, if any, would you still publish?
reported_release is a claim from the environment, and last_heartbeat_at may be days old for a client-managed install. At what staleness does this figure stop being usable?
How would you enforce the supported window when the rollout plan is built, instead of reporting a violation after it already exists?

Generate invoice lines once under concurrent month-end runs

Hard · isolation levels · unique constraints · idempotency

Month-end generation reads approved work_record rows for one engagement and period, sums minutes, inserts one invoice_line whose generation_key is UNIQUE over (engagement_id, period_start, period_end, line_type, generator_version), then sets those work records to invoiced with locked_at and invoice_line_id. Two runs execute concurrently for the same engagement while late approvals are still committing. Working in PostgreSQL, name the anomaly at READ COMMITTED and at REPEATABLE READ, and specify the controls that make a retry a no-op rather than a second charge.

Approach

At READ COMMITTED each statement takes a fresh snapshot, so the SELECT that computed the sum and the later UPDATE see different data: an approval committed between them is a phantom the sum missed but the UPDATE can still mark invoiced. Worse, an UPDATE that blocks on a row another transaction is changing re-evaluates its WHERE against the new row version once the lock releases, so WHERE status = 'approved' silently skips rows the other run already moved — fewer rows than you counted, and no error anywhere.
At REPEATABLE READ, which in PostgreSQL is snapshot isolation, the sum and the update agree because the whole transaction shares one snapshot, but a concurrent update to the same row aborts you with serialization_failure, SQLSTATE 40001, which the application must catch and retry. Snapshot isolation still permits write skew across different rows; only SERIALIZABLE with SSI excludes it, at the cost of more 40001s and predicate-lock memory bounded by max_pred_locks_per_transaction.
Make the second run collide instead of race: take a transaction-scoped advisory lock on a hash of (engagement_id, period_start, line_type) at the top of the run, so one generates while the other waits and then finds finished work. Keep the UNIQUE generation_key as the backstop, because the advisory lock is per-database-connection state that a failover or a second database node does not carry.
Treat the unique violation as success. Catch SQLSTATE 23505, re-select the existing invoice_line by generation_key, and return it. INSERT ... ON CONFLICT DO NOTHING ... RETURNING returns zero rows on conflict, so a handler that trusts RETURNING writes a NULL invoice_line_id and reports a failure for work that in fact completed; ON CONFLICT DO UPDATE is worse, because it overwrites a line that has already been issued.
Bound the read deterministically rather than by timing: select the work records FOR UPDATE ordered by work_record_id so both runs acquire row locks in the same order and cannot deadlock, and filter on approved_at < the run's start timestamp so a late approval is out of scope by definition and lands in the next period.
Enforce immutability in the database, not in the service. A BEFORE UPDATE trigger raising when locked_at IS NOT NULL and a billed column changes, plus CHECK (status <> 'invoiced' OR invoice_line_id IS NOT NULL), means the constant and legitimate business pressure to edit an invoiced entry resolves into an appended credit_note row instead of a quiet update.

Worked solution (40 min)

Open two psql sessions and interleave them by hand: A begins and inserts the invoice_line without committing; B begins and attempts the same insert.
Observe B blocking on the unique index, commit A, and record the exact error B receives.
Implement the handler path: catch 23505, re-select by generation_key, return that row, and confirm both sessions report the same invoice_line_id.
Repeat at REPEATABLE READ with both sessions updating the same work_record to force 40001.
Approve a work record after A's start timestamp and follow where it ends up.

Expected result: One invoice_line per (engagement, period, line_type, generator_version); the losing run returns the winner's row as success rather than an error; the late approval remains status approved, unlocked, and is picked up by the next period's run; the 40001 path is exercised by a retry rather than surfaced to the caller.

Verify it yourself

Both sessions report the same invoice_line_id, and the table holds exactly one row for that generation_key.
The late-approved work record still has status approved and locked_at NULL after both runs finish.
An UPDATE against a work record with locked_at set is rejected by the trigger, not merely by the service layer that happened not to issue it.
SELECT generation_key, count() FROM invoice_line GROUP BY 1 HAVING count() > 1 returns nothing — trivially, because the constraint and not the code is what guarantees it.

Follow-up

A line has been issued and then an approval behind it is reversed. What rows do you write, and what does the original line look like afterwards?
How many times does the run retry before giving up, and what state does it leave behind for the operator who picks it up?
After the fact, how do you prove no engagement-period was billed twice, without trusting the application code that wrote it?

Find watermark coverage gaps with a window function

Medium · window functions · gaps and islands · event tables

Each succeeded connector_run claims a half-open watermark range [watermark_from, watermark_to). Ranges for one connector can overlap, and a re-run can be nested entirely inside an earlier range. Write the query that returns, per connector_id, every interval not covered by any succeeded run, as (gap_from, gap_to), ordered by connector then time. State the frame clause you use and why the default one is wrong, and give the index that removes the sort.

Approach

Rule out lag(watermark_to) first and say why: ordered by start time, the immediately preceding row may end earlier than a row before it, so a nested or shorter range makes you report a gap that a wider earlier range already covers. The correct comparison is against the running maximum of every previous watermark_to, not the last one.
Compute max(watermark_to) OVER (PARTITION BY connector_id ORDER BY watermark_from, watermark_to ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). The ROWS frame is load-bearing: the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes the current row, so the running max always covers the current row's own end and the query reports no gaps ever.
Add watermark_to as a tiebreaker in the window ORDER BY so rows sharing a start instant are ordered deterministically under a ROWS frame, then emit a gap where the running max is non-null and watermark_from is strictly greater than it. The first row per connector has an empty frame and a NULL max; that is not a gap unless you define a floor such as the binding's creation time, which you should decide explicitly.
Filter status = 'succeeded' in the window's input, and argue the classification: an ambiguous run may or may not have applied its range, so counting it as covered can drop data silently. Gap detection should treat ambiguous as uncovered and leave resolution to the reconciliation path.
Offer the shorter alternative with its precondition: on PostgreSQL 14 and later, range_agg(tstzrange(watermark_from, watermark_to, '[)')) per connector builds a multirange, and the gaps are its complement against the window you care about. It handles adjacency and nesting without any frame subtlety, at the cost of requiring that version.
Index (connector_id, watermark_from, watermark_to) WHERE status = 'succeeded'. It supplies the window's required ordering directly, so the plan is a WindowAgg over an index scan with no Sort, and the work is O(n) over that connector's succeeded runs.

Follow-up

How do you distinguish a real gap from a period when the connector was deliberately paused?
The connector has run for two years and you only care about the last 30 days. What does adding that predicate do to the correctness of the running maximum?
How would you turn this query into an alert without it firing on the trailing edge, where the newest range is simply not written yet?

Describe a challenging bug you encountered in a mission-critical system. How did you diagnose and resolve it?

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
State the consistency you need, and where you are willing to be stale.
Name the failure you are designing for, then the recovery path.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

If you were tasked with automating a manual calculation process currently used by our engineering teams, how would you start?

Medium · system design and architecture

Approach

Name the read and write paths separately; they rarely have the same bottleneck.
Name the failure you are designing for, then the recovery path.
Choose a partition key and say what query it makes expensive.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

Walk me through the architecture of a project you led from conception to deployment.

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Choose a partition key and say what query it makes expensive.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

How have you designed software to interface with hardware or existing industrial systems?

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

How do you ensure your code remains maintainable and scalable when working in a multidisciplinary team?

Medium

Approach

State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Design the connector work queue under skewed client quotas

Medium · backpressure · rate limiting · queueing · head-of-line blocking

Hundreds of connector bindings run a few thousand jobs a day across a worker fleet. Each binding targets one client endpoint whose quota, say 600 requests a minute, is shared by every job you point at that client, including operator-triggered ones. Volume is skewed: roughly 5% of bindings produce most of the traffic and most of the failures. Design the queueing and admission control. State the key that concurrency and rate limiting are enforced on, what is still admitted while one endpoint returns 429 for an hour, what gets merged or dropped as the backlog grows, and how a small binding still runs on time.

Approach

Key the limiter on (client_id, endpoint), never on the worker or the job. The quota belongs to the client and is spent by every process you aim at them, so a per-worker limit silently multiplies by fleet size and a single global limit starves everyone the moment one client is hot. Implement it as a distributed token bucket in Redis behind one Lua script so the check and the decrement are atomic, with the refill set below the client's real limit, measured rather than read from their documentation.
Give every key its own queue and lease a bounded number of in-flight runs per key. One shared FIFO puts a backed-up large binding in front of a small one - head-of-line blocking, and the actual complaint you will receive. Workers claim with SELECT ... FOR UPDATE SKIP LOCKED over due runs and check the per-key in-flight count before claiming.
Treat 429 as a control signal rather than an error: honour Retry-After when present, otherwise exponential backoff with full jitter, sleep = random(0, min(cap, base * 2^attempt)). Deterministic backoff resynchronises every job against the same client and rebuilds the burst you just backed away from. While a key is in backoff, stop admitting its runs rather than admitting and failing them, so your queue absorbs the pressure instead of the client's endpoint.
Bound the queue instead of letting it grow. By Little's law a key admitted at a rate above its service rate grows without limit, so decide what is dropped before the backlog decides for you: scheduled incremental runs are idempotent and coalescable, so two pending runs over overlapping cursor ranges collapse into one wider range, while an operator-triggered run is not coalescable and needs a small reserved lane.
Name the trade-off. Per-key isolation costs fleet utilisation, because workers idle while a hot key is throttled, and buys predictable latency for the long tail of small bindings. Measure per-key scheduling delay percentiles, not fleet throughput - throughput is dominated by the noisy 5% and hides what the other 95% experience.

Worked solution (30 min)

Model two bindings on one client key, one issuing 500 requests a minute and one issuing 5, under a shared 600 a minute budget, served by a single FIFO queue.
Measure the small binding's scheduling delay, then rerun with per-key queues plus a reserved lane and measure again.
Inject an hour of 429 responses with deterministic backoff across 20 workers and plot the request rate at each retry boundary.
Repeat with full jitter and compare the peak rate and the time to drain.

Expected result: Under one FIFO the small binding's delay tracks the large binding's backlog and grows into hours; per-key queues hold it near its due time regardless of the neighbour. Deterministic backoff produces sharp synchronised spikes at every boundary, each large enough to re-trip the quota, while full jitter spreads the same retries into a roughly flat rate under budget.

Verify it yourself

Requests observed per minute at the client endpoint never exceed the configured budget, measured at the endpoint side rather than at the scheduler.
Scaling the fleet from 5 to 50 workers changes the observed request rate for no key.
A key in backoff admits no new runs, and its pending runs accumulate with a visible age rather than failing.

Follow-up

The client's quota turns out to be per credential rather than per organisation, and you hold three. Does the key change, and does anything else?
A run has sat queued for six hours behind a throttled binding. What does the client-facing status say during those six hours, and when do you abandon it?
How do you establish the real limit without tripping it inside the client's production system?

Credential minting stalls estate-wide from one client's provider

Hard · partial outage · timeouts · bulkheads · queueing

At 14:05 the access broker's p99 rises from 28 ms to 9.4 s. Error rate is only 3 percent. Connector runs and operator sessions for every client begin failing, yet only one client's identity provider is degraded: it accepts TCP connections and never replies. The broker handles five mint calls per second, of which one targets that provider, through a shared outbound worker pool of 50. Handler service time, measured from the start of each call, is unchanged. Diagnose in order and give the changes that bound the blast radius.

Approach

Separate slow from queued before anything else, because they need opposite responses. Split the broker's latency into time waiting for an outbound slot and time executing the call. Unchanged service time alongside a 300-fold p99 means the whole regression is queueing, so the target of the investigation is a saturated resource, not slow code.
Split every metric by target_system_id next. One client's provider owns all the non-completing calls; everyone else's calls are fast once they start. Combined with step one this gives the shape immediately: one dependency's calls occupy the shared pool and every other client waits behind them.
Do the arithmetic and check it against the timeline. One call per second that never completes fills all 50 slots in 50 seconds, which matches an onset measured in under a minute rather than gradual decay. If the numbers had not fit, saturation would not be the whole explanation and you would keep looking.
Confirm at the socket, since a hang and a failure are indistinguishable in an error rate. Connections in ESTABLISHED with bytes sent and nothing received distinguish a silent peer from a refused connection or a reset. Note explicitly that TCP keepalive will not rescue you here: the Linux default sends its first probe after two hours, so only an application-level read timeout bounds this at all, and its absence is the root defect.
Fix by bounding, in this order: a connect timeout and a read timeout on every outbound call, both set below the caller's own latency budget; a per-target concurrency limit so a single provider can occupy at most a small share of the pool; a circuit breaker keyed by target_system_id that opens on consecutive timeouts and half-opens with one probe; and a retry budget with full jitter, because three retries against a hanging peer triple the occupancy rather than improving the odds.
Then reconsider the blast radius rather than only the incident. A fail-closed broker on every request path is a shared dependency whose outbound concurrency must be partitioned per client, and a refusal for one client's target must surface as a typed result the caller can classify, not as a timeout that has already consumed a slot to produce.

Follow-up

Choose the read timeout and justify it with evidence. What breaks for a provider that is legitimately slow, and how do you tell the two apart?
The breaker is open for that client. What does an in-flight connector run observe, and which failure_class should it record?
Could the broker have kept serving every other client with no code change at all? What would that configuration have cost you on an ordinary day?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

Describe a time you had to optimize a piece of code that was performing poorly in a production environment.

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

Can you explain a complex technical concept to a non-technical stakeholder, such as a project manager or a client?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

How do you handle feedback on your code or design decisions from senior staff?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?