Interview guides · By Light Professional IT Services

By Light Professional IT Services Software Engineer Interview Questions & Guide 2026

Concurrency drills for By Light Professional IT Services software engineer prep: races, lock ordering, and what a check-then-act sequence assumes.

The 60-second brief

As a Software Engineer at By Light Professional IT Services, you are a critical contributor to the mission-focused technology solutions that define the company’s reputation in the government contracting space. Whether you are working on complex network operations, infrastructure design, or specialized software development, your work directly supports high-stakes environments where reliability, security, and performance are non-negotiable.

The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.

Focus areas

Bound credential lifetime by the engagement end date
Version an API you cannot force clients to upgrade
Make integration retries idempotent without target-side idempotency keys

Overview

This role requires a blend of technical precision and adaptability. You will be expected to bridge the gap between abstract requirements and functional systems, often working within collaborative, cross-functional teams that include project managers and other engineers. The environment is fast-paced and intellectually demanding, offering you the chance to influence systems that have a tangible impact on the success of the organization's government partners.

The By Light Professional IT Services interview process

By Light Professional IT Services candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Discussions with Leadership

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
Whether you can be redirected in the middle of an answer without restarting it from the beginning
Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Where candidates lose points

Forking the product per client instead of building a configuration or extension point

A fork is the cheapest possible answer for one client and the most expensive possible answer for the estate. Maintenance cost grows as O(clients x changes): a single security patch now needs one port per fork, each with its own conflicts, its own test run and its own release window, and the forks drift further apart with every port. The honest rule is that a required fork is a product gap, so it should either become a supported extension point with a stable interface, or be an explicit, time-boxed and separately priced artefact that is never expected to receive upstream fixes.

Tenant context leaking across a connection pool

Setting the tenant on a connection and relying on it for the rest of the request looks correct in every test that runs one request at a time. Under transaction pooling the connection goes back to the pool with that session state still attached, and the next checkout, serving a different client, inherits it; row-level security then enforces the previous tenant's policy flawlessly and returns the wrong rows. It reproduces only under concurrency, which is the load profile your integration tests do not have. Use a transaction-scoped setting, and assert that the setting matches the request's tenant immediately before the first query rather than trusting that it was set correctly upstream.

A queue or buffer with no bound

Every producer-consumer boundary needs a capacity and a policy for reaching it: block the producer, shed load, or drop the oldest entry. Unbounded buffering converts a temporary slowdown into memory exhaustion and hides the backpressure signal that would have revealed the consumer was falling behind.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Questions & practice

Find peak window utilisation and the first rejected request

Medium · sliding window · two pointers · rate limiting

One connector binding's requests against a client endpoint arrive as timestamps t[0..n-1] in milliseconds, sorted non-decreasing, n up to 1,000,000. The endpoint's budget is R requests per rolling window of W milliseconds, and a request at time t is admitted only if strictly fewer than R already-admitted requests fall in the half-open window (t-W, t]. Return (a) the maximum number of requests falling in any W-millisecond window, counting every request regardless of admission, and (b) the index of the first request the limiter would reject. Both parts in O(n) time.

Approach

Part (a) is a two-pointer scan: for each right index, advance left while t[right] - t[left] >= W, which leaves the window exactly (t[right]-W, t[right]], and track max(right-left+1). Left is monotone non-decreasing, so total work is O(n) amortised with O(1) extra space. The maximum over all windows of width W is attained at a window whose right edge is a request timestamp, so scanning right edges is sufficient.
Part (b) is not the same scan. A rejected request consumes no budget, so the window must count admitted requests only. Keep a FIFO deque of admitted timestamps: pop from the front while front <= t - W, admit if the deque holds fewer than R, and return the first index where it does not. O(n) total pushes and pops, O(R) memory.
Pin the boundary convention before writing either loop. With a half-open (t-W, t] window a request exactly W milliseconds after an earlier one does not see it; switching to a closed window changes the answer at every boundary and is where most disagreements with the client's own limiter come from.
Note what fixed buckets of width W buy and cost: O(1) memory, but they admit up to 2R inside a span shorter than W, R at the end of one bucket and R at the start of the next. If buckets are unavoidable, halve the bucket width or use a weighted estimate across two adjacent buckets.
If part (a) already exceeds R, the binding cannot run at this concurrency under any limiter shape. That is a scheduling problem, not a retry-policy problem, and no backoff tuning fixes it.

Worked solution (25 min)

Take W = 10,000 ms, R = 3, and timestamps in milliseconds: [0, 1000, 2000, 3000, 11000, 11500, 12000, 12500].
Part (a): at right=3 (t=3000) the window (-7000, 3000] holds 4 requests. At right=7 (t=12500) advance left past 0, 1000 and 2000 to index 3, since 12500-3000 = 9500 < 10000; the window holds indices 3..7, which is 5 requests.
Confirm no other right edge beats 5 by inspecting each, then record the peak.
Part (b): admit index 0, 1 and 2 (deque grows to 3 entries, none expiring since t-W is negative). At index 3, t=3000: nothing to pop, deque size is 3, which is not strictly fewer than R=3, so reject.
Check what happens after the rejection if rejected timestamps were wrongly enqueued: indices 4 onwards would also be refused despite the deque having expired entries by t=11000.

Expected result: (a) peak = 5 requests in the window (2500, 12500]. (b) first rejection at index 3, t = 3000 ms. Indices 0, 1 and 2 are admitted; the requests at 11000 onwards are admissible again because the first three have left the rolling window.

Verify it yourself

The left pointer in part (a) never decreases across the whole scan.
Brute force over all 8 right edges reproduces peak = 5.
Peak (5) > R (3) confirms the budget is exceeded at this arrival rate, independent of the limiter's shape.
Every admitted timestamp remaining in the deque satisfies front > t - W at the moment of the check.

Follow-up

Three bindings and four workers share this one endpoint's quota. Where does the counter live, and what is the correct behaviour when that store is unreachable?
Add full-jitter backoff to the retry path and state precisely what it prevents that plain exponential backoff does not.
The client documents the limit as 'about 100 per minute'. How would you measure the real one without tripping it in their production?

Compute the minimal re-run set that preserves idempotency keys

Hard · interval complement · idempotency keys · watermarks

One connector binding must cover [span_start, span_end). You have all its connector_run rows: (idempotency_key, attempt_no, status, watermark_from, watermark_to), with half-open ranges, where idempotency_key is derived deterministically from the connector and the range, so re-running a range reproduces its key. Only status='succeeded' counts as covered. Produce the minimal re-run plan: every instant in the span ends up covered, any sub-interval some prior run already claimed is re-run at exactly that claim's range, and genuinely unclaimed sub-intervals become new claims cut at the binding's chunk size. Explain what merging ranges would cost.

Approach

Merge the succeeded ranges into a maximal disjoint union: sort by start, sweep extending the open segment while the next start is not beyond the current end. O(n log n) time, O(n) space, and this is the only status that contributes coverage.
Take the complement of that union inside [span_start, span_end) to get the uncovered gaps, in O(u) over the merged segments. Clip at both ends so the plan never reaches outside the span.
Sweep the non-succeeded claims (ambiguous, failed, abandoned) against the gap list with a two-pointer over both sorted lists. Any claim intersecting a gap is emitted at its original watermark_from and watermark_to, byte for byte, so the derived idempotency_key reproduces and the retry rejoins the same ledger row. O(n) after sorting.
Subtract the emitted claim ranges from the gaps; whatever survives was never claimed by any attempt and becomes fresh ranges cut at the chunk size. Only these get new keys. O(u + c).
Accept that a re-emitted claim may overlap time that is already succeeded. Range bookkeeping is not what makes the retry safe; the record-level dedupe key is, and re-reading a covered window costs one extra fetch against the client's quota, which is far cheaper than a key you can no longer reproduce.
Exclude anything still in status 'running' with a live lease. Re-claiming it races the in-flight attempt, doubles the quota spend against the client endpoint, and produces two ledger writers for one key. Total plan cost is O(n log n) time and O(n) space.

Follow-up

An ambiguous run may have applied every record before the timeout. What do you read to settle it before re-running, and what is left if the client's schema offers no natural key?
Retention pruned the ledger rows for one of these keys. What does the re-run do now, and what bounds how long the ledger must be kept?
Two planners run concurrently against the same binding. What stops them emitting and executing the same plan twice?

Resolve effective-dated engagement versions for two million access checks

Hard · binary search · effective dating · complexity budget

The engagement table holds 500,000 effective-dated rows: (engagement_id, version, effective_from, effective_to which is NULL for the open version). A database exclusion constraint guarantees that per engagement_id the tstzrange(effective_from, effective_to) values never overlap, and the ranges are half-open. You must answer 2,000,000 access checks of the form (engagement_id, at_time) in one process with no database, returning the version open at that instant or none. Also report, per engagement, any coverage gap between adjacent versions. Give the complexity of the scan-per-check approach and say why it is not an option.

Approach

State the naive cost first: scanning the table per check is 2e6 x 5e5 = 1e12 range comparisons. Even at 1e9 simple comparisons per second that is roughly fifteen minutes of pure compare work, against a budget measured in seconds. It is correct, just unaffordable, and the fix is an index built once rather than a faster comparison.
Build a hash map engagement_id -> array of versions sorted by effective_from, once. O(V log V) to sort, O(V) memory. The build is amortised over 2e6 lookups, which is what makes it worth doing at all.
Answer each check by binary-searching for the last effective_from <= at_time, then confirming effective_to IS NULL OR at_time < effective_to. If the confirmation fails, the instant lies in a gap and the answer is none, not the neighbouring version. O(log v) per check, O(V log V + Q log v) total.
Name the precondition: predecessor search is only valid because the ranges per engagement do not overlap, which the exclusion constraint enforces with btree_gist for the equality operator. Without that guarantee you need an interval tree returning all stabbed intervals, at O(log n + k). Use the same half-open convention as the database, where at_time equal to effective_to belongs to the next version.
Find gaps with one O(V) pass over each sorted array comparing prev.effective_to against next.effective_from. The exclusion constraint forbids overlaps, not holes, so a hole is entirely possible and is where an access check fails closed for a reason nobody can see in the contract.
If the checks can be buffered, sort them by (engagement_id, at_time) and merge against the version arrays: O(Q log Q + V log V), usually a large constant-factor win over 2e6 random binary searches because the merge walks memory in order.

Follow-up

The open version has effective_to NULL. How do you keep both the sort and the comparison total without a sentinel that overflows your timestamp type?
A gap exists for one engagement and a check lands in it. What should the access path return, and what should it emit so someone fixes the amendment?
Amendments are being written while you answer checks. What read model keeps an answer self-consistent, and what does a check mean if the snapshot is a second stale?

Find watermark coverage gaps with a window function

Medium · window functions · gaps and islands · event tables

Each succeeded connector_run claims a half-open watermark range [watermark_from, watermark_to). Ranges for one connector can overlap, and a re-run can be nested entirely inside an earlier range. Write the query that returns, per connector_id, every interval not covered by any succeeded run, as (gap_from, gap_to), ordered by connector then time. State the frame clause you use and why the default one is wrong, and give the index that removes the sort.

Approach

Rule out lag(watermark_to) first and say why: ordered by start time, the immediately preceding row may end earlier than a row before it, so a nested or shorter range makes you report a gap that a wider earlier range already covers. The correct comparison is against the running maximum of every previous watermark_to, not the last one.
Compute max(watermark_to) OVER (PARTITION BY connector_id ORDER BY watermark_from, watermark_to ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). The ROWS frame is load-bearing: the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes the current row, so the running max always covers the current row's own end and the query reports no gaps ever.
Add watermark_to as a tiebreaker in the window ORDER BY so rows sharing a start instant are ordered deterministically under a ROWS frame, then emit a gap where the running max is non-null and watermark_from is strictly greater than it. The first row per connector has an empty frame and a NULL max; that is not a gap unless you define a floor such as the binding's creation time, which you should decide explicitly.
Filter status = 'succeeded' in the window's input, and argue the classification: an ambiguous run may or may not have applied its range, so counting it as covered can drop data silently. Gap detection should treat ambiguous as uncovered and leave resolution to the reconciliation path.
Offer the shorter alternative with its precondition: on PostgreSQL 14 and later, range_agg(tstzrange(watermark_from, watermark_to, '[)')) per connector builds a multirange, and the gaps are its complement against the window you care about. It handles adjacency and nesting without any frame subtlety, at the cost of requiring that version.
Index (connector_id, watermark_from, watermark_to) WHERE status = 'succeeded'. It supplies the window's required ordering directly, so the plan is a WindowAgg over an index scan with no Sort, and the work is O(n) over that connector's succeeded runs.

Follow-up

How do you distinguish a real gap from a period when the connector was deliberately paused?
The connector has run for two years and you only care about the last 30 days. What does adding that predicate do to the correctness of the running maximum?
How would you turn this query into an alert without it firing on the trailing edge, where the newest range is simply not written yet?

Add a non-null column to a live work-record table

Hard · online migration · lock contention · expand-contract

work_record holds roughly nine million rows and absorbs a month-end write burst. You must add currency CHAR(3) NOT NULL, backfill it from the engagement, add a foreign key to rate_card, and eventually drop the superseded hours_decimal column. Several releases are live, including client-managed installs running two-quarter-old code that still writes hours_decimal. Give the ordered plan, the lock each step takes and for roughly how long, how the backfill is chunked and restarted, and the signal that makes the drop safe.

Approach

Sequence it expand, dual-write, backfill, switch reads, contract, and be explicit that only the last step is dangerous. Adding is cheap because an old release ignores a column it does not know about; removing breaks a writer you cannot upgrade and whose failure you may not see for weeks.
ALTER TABLE work_record ADD COLUMN currency CHAR(3) — nullable, no default. That is a catalog-only change on every supported version and takes milliseconds. Refuse NOT NULL DEFAULT 'XXX': PostgreSQL 11 and later store a non-volatile default in the catalog and materialise it on read, so it is just as fast, but it hands all nine million rows a value that is already non-null and already wrong. That destroys the rest of the plan — the NOT NULL you add later is satisfied from the first second, so it can never find a row the backfill missed, and every consumer now has to distinguish a real currency from a sentinel forever. A NULL is the only marker that survives a crashed backfill.
The statement is instant; the lock queue is not. An ACCESS EXCLUSIVE request waits behind every open transaction holding a conflicting lock on the table, and every request arriving after it queues behind that, so one long analytics SELECT turns instant DDL into a total stall. Run it with SET lock_timeout = '2s' inside a retry loop that catches 55P03 and backs off with jitter.
Decide up front what an unconverged writer inserts. Two-quarter-old code omits currency, so it writes NULL, which means SET NOT NULL would start rejecting its inserts outright. Either fill the gap in the database with a BEFORE INSERT trigger that resolves currency from the engagement when NEW.currency IS NULL — one indexed lookup per insert, which you should time against the month-end burst before committing to it — or leave the column nullable and gate SET NOT NULL on the same release-convergence signal that gates the drop. What you must not do is paper over it with a default, which turns a visible NULL from an unconverged writer into an invisible 'XXX' that reconciliation will never flag.
Backfill on a keyset cursor — WHERE work_record_id > $last AND currency IS NULL ORDER BY work_record_id LIMIT 5000 — one transaction per batch, advancing $last to the last work_record_id the batch wrote so filled rows are never revisited. Set statement_timeout, pause between batches so autovacuum and any replicas keep up, and persist the cursor so a crash resumes from the last committed batch. OFFSET n costs O(n) per page because the server scans and discards those rows, which makes the whole backfill quadratic in the table size.
Take NOT NULL without a blocking scan, and only once the backfill reports zero rows remaining: ADD CONSTRAINT work_record_currency_nn CHECK (currency IS NOT NULL) NOT VALID takes a brief ACCESS EXCLUSIVE and scans nothing; VALIDATE CONSTRAINT runs under SHARE UPDATE EXCLUSIVE so reads and writes continue throughout, and raises check_violation, SQLSTATE 23514, naming a row if the backfill missed one — that failure is the entire reason the step exists, and it only exists because the column was added without a default; then SET NOT NULL, which on PostgreSQL 12 and later proves itself from the validated CHECK and skips its own full scan.
Use the same shape for the foreign key — ADD CONSTRAINT ... REFERENCES rate_card(...) NOT VALID, then VALIDATE CONSTRAINT — and build any supporting index with CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block, scans the table twice, and on failure leaves an INVALID index that must be dropped concurrently before you retry. Note what the FK does not prove: a single-column reference is MATCH SIMPLE, so rows with NULL currency pass validation untouched, and validating it before the backfill finishes certifies only the rows that were already correct.
Gate the contract step on observed convergence rather than elapsed time: every non-destroyed environment must report a reported_release_id at or past the release that stopped writing hours_decimal, and reported_release_id IS NULL counts as unknown and therefore blocks. DROP COLUMN itself is a catalog-only update, instant under ACCESS EXCLUSIVE, reclaiming no space until a rewrite — the cost is entirely in the writer you did not know was still running.

Worked solution (45 min)

Write out each DDL statement in order, annotated with its lock mode and whether it is safe during the month-end burst.
Implement the retry wrapper: SET lock_timeout, attempt, catch 55P03, sleep with jitter, retry a bounded number of times.
Write the keyset backfill loop with the cursor persisted after each committed batch, and time one batch.
Leave one row un-backfilled on purpose, run VALIDATE CONSTRAINT, and read the 23514 it raises — then fix that row and re-run. A plan in which this step cannot fail is a plan that is not checking anything.
Write the convergence query over environment that returns the rows still blocking the drop.
Ship the DROP COLUMN as its own migration whose precondition is that query returning zero rows.

Expected result: An ordered plan: ADD COLUMN currency CHAR(3) nullable with no default, under a short ACCESS EXCLUSIVE with lock_timeout and retry; a BEFORE INSERT trigger or a convergence gate covering writers that do not yet set it; application dual-write; keyset backfill in bounded, restartable batches until no NULLs remain; CHECK ... NOT VALID then VALIDATE, which fails loudly on a missed row, then SET NOT NULL; FK NOT VALID then VALIDATE with any index built CONCURRENTLY; reads switched; DROP COLUMN withheld until every non-destroyed environment reports a converged release.

Verify it yourself

Hold an open transaction with a SELECT on work_record, issue the ALTER, and watch the queue form in pg_locks and pg_stat_activity; then confirm lock_timeout aborts the ALTER instead of everything else waiting on it.
SELECT count(*) FROM work_record WHERE currency IS NULL falls batch by batch and must read zero before SET NOT NULL; under a sentinel default it would have read zero from the first minute and proved nothing.
Time one batch and multiply: nine million rows at 5,000 per batch is 1,800 batches, so the inter-batch pause, not the query, sets the wall-clock duration.
Run concurrent inserts throughout VALIDATE CONSTRAINT and confirm none of them blocked.
With the trigger disabled, an insert from a simulated old release lands with currency NULL and is caught by the NULL count rather than passing as a plausible value.
The convergence query returns non-zero while any environment has reported_release_id IS NULL, so a silent client-managed install blocks the drop rather than being assumed converged.

Follow-up

A client-managed install reports a release you no longer support and has not converged for two quarters. Do you drop, and what do you owe that client first?
During the dual-write window, which column does the month-end reconciliation read, and what does a mismatch between the two mean?
The backfill dies at batch 900 of 1,800. What exactly does the restart re-read, and how do you know it did not skip a range?

They explained the tech stack they used; do you have experience with any of the mentioned technology or anything similar to it?

Medium

Approach

State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

What specific technical skills do you bring to the team, and how have you applied them in past projects?

Medium

Approach

State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

How do you stay updated with changes in [relevant technology/stack] to ensure your work remains efficient?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Design the connector work queue under skewed client quotas

Medium · backpressure · rate limiting · queueing · head-of-line blocking

Hundreds of connector bindings run a few thousand jobs a day across a worker fleet. Each binding targets one client endpoint whose quota, say 600 requests a minute, is shared by every job you point at that client, including operator-triggered ones. Volume is skewed: roughly 5% of bindings produce most of the traffic and most of the failures. Design the queueing and admission control. State the key that concurrency and rate limiting are enforced on, what is still admitted while one endpoint returns 429 for an hour, what gets merged or dropped as the backlog grows, and how a small binding still runs on time.

Approach

Key the limiter on (client_id, endpoint), never on the worker or the job. The quota belongs to the client and is spent by every process you aim at them, so a per-worker limit silently multiplies by fleet size and a single global limit starves everyone the moment one client is hot. Implement it as a distributed token bucket in Redis behind one Lua script so the check and the decrement are atomic, with the refill set below the client's real limit, measured rather than read from their documentation.
Give every key its own queue and lease a bounded number of in-flight runs per key. One shared FIFO puts a backed-up large binding in front of a small one - head-of-line blocking, and the actual complaint you will receive. Workers claim with SELECT ... FOR UPDATE SKIP LOCKED over due runs and check the per-key in-flight count before claiming.
Treat 429 as a control signal rather than an error: honour Retry-After when present, otherwise exponential backoff with full jitter, sleep = random(0, min(cap, base * 2^attempt)). Deterministic backoff resynchronises every job against the same client and rebuilds the burst you just backed away from. While a key is in backoff, stop admitting its runs rather than admitting and failing them, so your queue absorbs the pressure instead of the client's endpoint.
Bound the queue instead of letting it grow. By Little's law a key admitted at a rate above its service rate grows without limit, so decide what is dropped before the backlog decides for you: scheduled incremental runs are idempotent and coalescable, so two pending runs over overlapping cursor ranges collapse into one wider range, while an operator-triggered run is not coalescable and needs a small reserved lane.
Name the trade-off. Per-key isolation costs fleet utilisation, because workers idle while a hot key is throttled, and buys predictable latency for the long tail of small bindings. Measure per-key scheduling delay percentiles, not fleet throughput - throughput is dominated by the noisy 5% and hides what the other 95% experience.

Worked solution (30 min)

Model two bindings on one client key, one issuing 500 requests a minute and one issuing 5, under a shared 600 a minute budget, served by a single FIFO queue.
Measure the small binding's scheduling delay, then rerun with per-key queues plus a reserved lane and measure again.
Inject an hour of 429 responses with deterministic backoff across 20 workers and plot the request rate at each retry boundary.
Repeat with full jitter and compare the peak rate and the time to drain.

Expected result: Under one FIFO the small binding's delay tracks the large binding's backlog and grows into hours; per-key queues hold it near its due time regardless of the neighbour. Deterministic backoff produces sharp synchronised spikes at every boundary, each large enough to re-trip the quota, while full jitter spreads the same retries into a roughly flat rate under budget.

Verify it yourself

Requests observed per minute at the client endpoint never exceed the configured budget, measured at the endpoint side rather than at the scheduler.
Scaling the fleet from 5 to 50 workers changes the observed request rate for no key.
A key in backoff admits no new runs, and its pending runs accumulate with a visible age rather than failing.

Follow-up

The client's quota turns out to be per credential rather than per organisation, and you hold three. Does the key change, and does anything else?
A run has sat queued for six hours behind a throttled binding. What does the client-facing status say during those six hours, and when do you abandon it?
How do you establish the real limit without tripping it inside the client's production system?

Connector workers are OOM-killed every day and a half

Medium · memory leak · caching · heap analysis

Connector runtime workers are OOM-killed after 18 to 40 hours and a restart clears it. RSS grows monotonically at about 1.9 MB per completed run, flat regardless of how many records that run moved, and workers serving more distinct bindings do not grow faster than workers serving one. Live heap measured immediately after a forced full collection rises with the same slope as RSS. Traffic is unchanged week over week. Give the ordered diagnosis, the specific object you expect at the root, and the fix.

Approach

Prove retention before hunting for it. RSS and the post-collection live set rising with the same slope means the objects are reachable, which is a leak in the ordinary sense. RSS rising while the live set stays flat would be allocator fragmentation or native buffers, and that is a different investigation with different tools.
Find the unit the growth is keyed on by division rather than by guessing. 1.9 MB per completed run, flat across record counts and flat across distinct bindings, points at something retained exactly once per run. Per-record growth would implicate an accumulating buffer; per-binding growth would implicate a legitimate cache that is merely unbounded.
Diff two heap snapshots an hour apart by retained size and walk the dominator tree to a single root. Expect a module-level dictionary whose entry count equals the number of runs completed since process start, which is the observation that confirms step two.
Read the key and the value separately, because they are two different bugs. The key contains something run-scoped, an idempotency key or a correlation id, which is why the hit rate is zero and the map only ever grows. The value is megabytes rather than kilobytes because the compiled schema validator closes over the first parsed page, so every entry pins a client payload as well.
Fix both halves. Re-key on (connector_id, source_schema_version), whose cardinality is in the hundreds and stable. Cap the map with a fixed-size LRU so an unforeseen key can never make it unbounded again. Stop the closure capturing the page, so an entry costs a compiled schema rather than a schema plus a payload.
Make the class of bug visible next time: export entry count and hit rate for every process-level cache. A zero hit rate is the unmistakable signature of a run-scoped key and appears within minutes, whereas this one took 18 hours to present itself as an OOM kill.

Follow-up

Those retained pages are client payloads sitting in a long-lived shared process. What does that mean beyond memory?
A worker killed mid-run leaves connector_run rows in 'running'. Who reconciles them, and how do you avoid applying that range twice?
Same symptom, but the post-collection live set is flat and only RSS climbs. What are you looking at instead, and which tool do you reach for?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

What are your primary professional goals for the next stage of your career?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Where do you see yourself in 5 years, and how does this role fit into that trajectory?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Can you describe how you typically interact with team members to resolve technical challenges?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?