Interview guides · Cerebral Staffing

Cerebral Staffing Software Engineer Interview Questions & Guide 2026

Cerebral Staffing software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

The Software Engineer role at Cerebral Staffing is a specialized position that bridges the gap between high-level technical development and direct industrial application. As a partner to major industry leaders like John Deere, Cerebral Staffing places engineers into critical, high-impact environments where software and systems engineering intersect with physical product development and manufacturing excellence.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

Make integration retries idempotent without target-side idempotency keys
Bound credential lifetime by the engagement end date
Version an API you cannot force clients to upgrade

Overview

In this role, you are not just writing code in a vacuum; you are contributing to the operational efficiency, quality, and product development lifecycles of world-class machinery. Whether you are working on tactical quality engineering, supplier-side software systems, or direct product development, your work directly influences the reliability of the products that move the world. This position requires a unique blend of technical rigor, attention to detail, and a deep understanding of the manufacturing ecosystem.

The Cerebral Staffing interview process

Cerebral Staffing candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Where candidates lose points

Validating against a client sandbox and assuming production parity

Sandboxes typically carry smaller data, looser or absent rate limits, a schema version behind production, and sometimes synchronous behaviour where production is asynchronous. Code that passes there fails first in the client's production, during a change window you do not control and often cannot get a second one of. The mitigations are specific: assert the observed schema version at the boundary of every run, measure the production rate limit empirically rather than reading it from a document, and design the first production run to be a bounded, reversible slice rather than a full backfill.

Treating an ambiguous failure as a definite one

A timeout, a 502 from an intermediary, or a connection reset after the request bytes were sent all leave the target's state unknown. Classifying those as failures and retrying duplicates the effect; classifying them as successes and advancing the watermark loses data silently. Both wrong answers are common because the ambiguous case is rare in a sandbox and routine in production. The run needs a distinct ambiguous state, a dedupe key that makes the retry safe, and a reconciliation read against the target when the key alone cannot settle it.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Reading the constraints as preamble rather than as part of the problem

The bounds are usually there to eliminate the obvious approach: n up to 10^5 makes an O(n^2) scan roughly 10^10 operations, far outside any per-test time budget, and an input larger than memory rules out loading it at all. When a bound is not given, ask for it, then say out loud which approach it kills.

Questions & practice

Measure credential exposure past engagement close without double counting

Medium · interval merge · sweep line · credential ttl

For one engagement you have credential_grant rows: (grant_id, principal_id, target_system_id, issued_at, not_after, revoked_at which may be NULL). The cutoff instant T is the engagement's ends_on plus access_grace_days. A grant is live over the half-open interval [issued_at, min(not_after, revoked_at)). There are up to 200,000 grants across target systems. For each target_system_id, return the total wall-clock time after T during which at least one grant was live, plus the disjoint segments that make it up. Grants that overlap must be counted once.

Approach

Clip each grant to [max(issued_at, T), min(not_after, coalesce(revoked_at, +inf))) and discard any interval whose start is not strictly before its end. O(n), and it removes every grant that already expired inside the engagement window.
Bucket the survivors by target_system_id and sort each bucket by start. O(n log n) overall, and sorting dominates the whole algorithm.
Sweep each bucket once, holding one open segment [s,e): if the next start is greater than e, emit [s,e) and open a new segment, otherwise set e = max(e, next_end). O(n) after the sort, O(1) working state, O(number of emitted segments) output.
Total exposure is the sum of emitted segment lengths, which is the measure of the union. Summing per-grant durations instead answers a different question and is unbounded above by the elapsed wall clock.
Produce a second, conservative figure that ignores revoked_at and uses not_after alone. revoked_at records that you asked for revocation; whether the client's identity provider honoured it is not something this table knows, and a self-contained token validated offline stays valid to its own expiry regardless.
If a bucket does not fit in memory, sort externally and sweep the stream, or push end timestamps into a min-heap keyed by end and pop those below the current start. Same O(n log n), memory proportional to the maximum number of concurrently live grants.

Follow-up

Three grants overlap and one of them belongs to an offboarded contractor. What must the sweep emit so exposure can be attributed per principal?
Which of your two numbers is the real bound on access if the client system validates tokens offline, and what does that imply about issuing TTLs in the first place?
Run this across 20,000 engagements as a nightly job with a fixed memory budget. What changes in the shape of the computation?

Assign connector bindings to rollout waves and name the cycle

Medium · topological sort · cycle detection · dependency graph

Connector bindings declare ordering dependencies: an edge u -> v means v must not run in a wave earlier than one after u has succeeded. You have up to 50,000 bindings and 200,000 edges, and a misconfigured dependency may have created a cycle. Assign each binding the earliest wave number such that every predecessor sits in a strictly earlier wave, return the total wave count, and if a cycle exists return the bindings on one concrete cycle rather than a boolean. The implementation must be iterative: a dependency chain can be 50,000 deep.

Approach

Build an adjacency list and an indegree array, then run Kahn's algorithm from every indegree-zero node. Bindings with no predecessors get wave 0. O(V+E) time, O(V+E) space.
Relax waves as you decrement: when popping u, set wave[v] = max(wave[v], wave[u]+1) for each successor before enqueueing v at indegree zero. Because v is only enqueued after its last predecessor is popped, wave[v] is 1 + max over predecessors, which is the earliest legal wave. Total wave count is max(wave)+1, the longest path in nodes.
If Kahn emits fewer than V nodes, the residual subgraph is exactly the nodes lying on or downstream of a cycle. Every residual node still has an in-edge inside the residual, otherwise it would have been popped, so walking backwards along in-edges cannot dead-end and must revisit a node within |residual| steps; the segment between the two visits is a concrete cycle. O(V) with a visited-position map.
Use an explicit stack or a queue rather than recursion. Recursive DFS on a 50,000-node chain exceeds the default stack in most runtimes, and the failure looks like a crash rather than a cycle.
At this size, recompute the whole plan whenever an edge changes. Incremental topological order maintenance costs far more code than the O(V+E) rebuild it saves.

Follow-up

Two bindings land in the same wave but hit the same client endpoint, whose rate budget is shared. How does the wave plan have to change, and what problem class does that turn into?
One dependency holds only for overlapping watermark ranges rather than globally. Is the dependency graph still a DAG over bindings, and what is the right vertex if not?
Report the critical path so an operator can see which chain sets the wave count, not just that it is five.

Collapse connector retry attempts into logical jobs and failure counts

Easy · hash aggregation · idempotency · retries

You are given one UTC day of connector_run rows: (connector_id, idempotency_key, attempt_no, status, failure_class, records_read, records_applied), unique on (connector_id, idempotency_key, attempt_no), up to 5,000,000 rows in arbitrary order. Retries of one logical job share the idempotency_key; the job's outcome is the status of its highest attempt_no. Return, per connector_id: the number of logical jobs, a count of terminally failed jobs by failure_class, and the dedupe hit total, meaning sum(records_read - records_applied) over terminal-succeeded attempts only. One pass over the input; state your memory bound.

Approach

Key a hash map on (connector_id, idempotency_key) and keep only the highest attempt_no seen so far plus that row's status, failure_class and record counts. One pass, O(n) time, O(d) space where d is the number of distinct logical jobs, not O(n).
Treat a repeated (key, attempt_no) as corrupt input and raise, since the table's uniqueness constraint says it cannot happen; silently overwriting hides a double-insert in the runtime.
Fold the d surviving entries into per-connector counters in a second pass over the map, not over the rows. The failure_class histogram is a small fixed-width map per connector because failure_class is an enum.
Compute dedupe hits only from terminal-succeeded attempts. A failed attempt's records_read is work attempted, not work deduplicated, and adding it counts the same source records once per retry.
If d does not fit in memory, partition the input by hash(connector_id, idempotency_key) into p files and aggregate each partition independently. Same O(n) total work, p sequential passes, memory traded for I/O; the partition function must use the full key or a job's attempts split across files.

Worked solution (20 min)

Take six rows: (c=7,K=A,1,failed,client_429,0,0), (c=7,K=A,2,succeeded,null,500,480), (c=7,K=B,1,ambiguous,client_timeout,300,300), (c=7,K=B,2,succeeded,null,300,0), (c=9,K=C,1,failed,schema_mismatch,0,0), (c=9,K=C,2,failed,schema_mismatch,0,0).
Build the map: A -> attempt 2 succeeded, B -> attempt 2 succeeded, C -> attempt 2 failed/schema_mismatch. Three entries from six rows.
Fold per connector: connector 7 has 2 logical jobs, 0 terminal failures; connector 9 has 1 logical job, 1 terminal failure classed schema_mismatch.
Dedupe hits from terminal-succeeded attempts only: (500-480) + (300-0) = 320 for connector 7, 0 for connector 9.
Compute the wrong answer deliberately by counting attempt rows: 6 jobs and 3 failures, which is what a naive GROUP BY on run rows returns.

Expected result: connector 7: 2 logical jobs, no terminal failures, 320 dedupe hits. connector 9: 1 logical job, 1 terminal failure with failure_class='schema_mismatch', 0 dedupe hits. Attempt-level counting would have reported 6 jobs and 3 failures.

Verify it yourself

The sum of terminal statuses across all connectors equals the number of distinct (connector_id, idempotency_key) pairs, here 3.
records_applied <= records_read holds on every row, so the dedupe total is non-negative.
Rerunning the aggregation over a shuffled copy of the input gives identical output, since the highest attempt_no wins regardless of arrival order.
Map size equals 3, strictly less than the 6 input rows, confirming collapse actually happened.

Follow-up

A job's highest attempt is 'cancelled' but an earlier attempt succeeded. What is the job's outcome, and what does that say about who writes the cancel?
Produce this incrementally as runs land instead of as a daily batch: what state do you keep per key, and what happens when an attempt arrives out of order?
records_applied is written by your runtime, not by the client. What would you reconcile it against before anyone trusts the dedupe-hit number?

Return the latest attempt per connector idempotency key

Easy · distinct-on · indexing · idempotency

connector_run stores one row per attempt: run_id, connector_id, idempotency_key, attempt_no SMALLINT, status (queued, running, succeeded, failed, ambiguous, cancelled, abandoned), failure_class, watermark_from, watermark_to, started_at, finished_at, unique on (connector_id, idempotency_key, attempt_no). For one connector_id, write the query returning the highest-numbered attempt per idempotency_key, keeping only those keys whose latest attempt is not succeeded, ordered by watermark_from. Give the index that serves it, and say what the plan does without that index.

Approach

Use DISTINCT ON (idempotency_key) with ORDER BY idempotency_key, attempt_no DESC. PostgreSQL requires the ORDER BY to lead with exactly the DISTINCT ON expressions, and that requirement is what makes the row picked deterministic rather than whichever the scan happened to reach first.
Apply the not-succeeded filter outside, in a wrapping SELECT. Putting it in the inner WHERE changes the question from 'keys whose latest attempt failed' to 'keys that have any non-succeeded attempt', which reports a key that failed twice and then succeeded, and is the single most common wrong answer here.
Know the portable alternative: row_number() OVER (PARTITION BY idempotency_key ORDER BY attempt_no DESC) in a subquery, filtered = 1 outside. A window function cannot be filtered in WHERE because WHERE is evaluated before the window, and PostgreSQL has no QUALIFY clause, so the subquery is mandatory rather than stylistic.
Index (connector_id, idempotency_key, attempt_no DESC). With connector_id fixed by equality, the index supplies the exact order DISTINCT ON needs and the Sort node disappears. It does not reduce rows read: through PostgreSQL 17 there is no loose index scan, so the executor still walks every attempt in range rather than skipping to each next distinct key.
Give the complexity honestly: without the index the plan is a scan plus an O(n log n) sort over the connector's attempts; with it, O(n) ordered reads and constant working memory, where n is attempts in range, not distinct keys. If the distinct keys are a tiny fraction of attempts, a recursive skip-scan is the next step.

Worked solution (15 min)

Create the table with a handful of rows across three idempotency keys, including one key with attempts 1 failed, 2 ambiguous, 3 succeeded.
Write the inner DISTINCT ON query alone and confirm it returns one row per key.
Wrap it and add WHERE status <> 'succeeded' plus ORDER BY watermark_from in the outer query.
Create the index and compare EXPLAIN output before and after.

Expected result: SELECT * FROM (SELECT DISTINCT ON (idempotency_key) idempotency_key, attempt_no, status, failure_class, watermark_from, watermark_to FROM connector_run WHERE connector_id = $1 ORDER BY idempotency_key, attempt_no DESC) latest WHERE status <> 'succeeded' ORDER BY watermark_from;

Verify it yourself

The key whose attempts run failed, ambiguous, succeeded is absent from the result, proving the filter runs after the pick.
Move the filter into the inner WHERE and watch that key reappear as attempt 2 — the wrong answer, visible on a four-row fixture.
EXPLAIN shows an Index Scan with no Sort node once (connector_id, idempotency_key, attempt_no DESC) exists.

Follow-up

attempt_no is not monotonic in time because a retry was queued out of order. Does your query still pick the right row, and which column should it actually order by?
How would you return the latest attempt per key across all connectors in one environment without the query degrading to a full sort?
An ambiguous latest attempt means the target may or may not have been written. What does this query feed, and should ambiguous be treated as failed here?

Generate invoice lines once under concurrent month-end runs

Hard · isolation levels · unique constraints · idempotency

Month-end generation reads approved work_record rows for one engagement and period, sums minutes, inserts one invoice_line whose generation_key is UNIQUE over (engagement_id, period_start, period_end, line_type, generator_version), then sets those work records to invoiced with locked_at and invoice_line_id. Two runs execute concurrently for the same engagement while late approvals are still committing. Working in PostgreSQL, name the anomaly at READ COMMITTED and at REPEATABLE READ, and specify the controls that make a retry a no-op rather than a second charge.

Approach

At READ COMMITTED each statement takes a fresh snapshot, so the SELECT that computed the sum and the later UPDATE see different data: an approval committed between them is a phantom the sum missed but the UPDATE can still mark invoiced. Worse, an UPDATE that blocks on a row another transaction is changing re-evaluates its WHERE against the new row version once the lock releases, so WHERE status = 'approved' silently skips rows the other run already moved — fewer rows than you counted, and no error anywhere.
At REPEATABLE READ, which in PostgreSQL is snapshot isolation, the sum and the update agree because the whole transaction shares one snapshot, but a concurrent update to the same row aborts you with serialization_failure, SQLSTATE 40001, which the application must catch and retry. Snapshot isolation still permits write skew across different rows; only SERIALIZABLE with SSI excludes it, at the cost of more 40001s and predicate-lock memory bounded by max_pred_locks_per_transaction.
Make the second run collide instead of race: take a transaction-scoped advisory lock on a hash of (engagement_id, period_start, line_type) at the top of the run, so one generates while the other waits and then finds finished work. Keep the UNIQUE generation_key as the backstop, because the advisory lock is per-database-connection state that a failover or a second database node does not carry.
Treat the unique violation as success. Catch SQLSTATE 23505, re-select the existing invoice_line by generation_key, and return it. INSERT ... ON CONFLICT DO NOTHING ... RETURNING returns zero rows on conflict, so a handler that trusts RETURNING writes a NULL invoice_line_id and reports a failure for work that in fact completed; ON CONFLICT DO UPDATE is worse, because it overwrites a line that has already been issued.
Bound the read deterministically rather than by timing: select the work records FOR UPDATE ordered by work_record_id so both runs acquire row locks in the same order and cannot deadlock, and filter on approved_at < the run's start timestamp so a late approval is out of scope by definition and lands in the next period.
Enforce immutability in the database, not in the service. A BEFORE UPDATE trigger raising when locked_at IS NOT NULL and a billed column changes, plus CHECK (status <> 'invoiced' OR invoice_line_id IS NOT NULL), means the constant and legitimate business pressure to edit an invoiced entry resolves into an appended credit_note row instead of a quiet update.

Follow-up

A line has been issued and then an approval behind it is reversed. What rows do you write, and what does the original line look like afterwards?
How many times does the run retry before giving up, and what state does it leave behind for the operator who picks it up?
After the fact, how do you prove no engagement-period was billed twice, without trusting the application code that wrote it?

Describe a situation where you had to interpret complex data to improve a process or product.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How do you balance the need for rapid development with the stringent quality standards required in industrial settings?

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Make timesheet submission safe under a client's automatic retries

Medium · idempotency · retries · unique-constraints · http-semantics

work_record holds work_record_id, engagement_id, principal_id, work_date, entered_at, minutes, is_billable, status, approved_at, locked_at and invoice_line_id, and entered_at is assigned by the server at insert. A delivery-team app submits entries over mobile networks and retries any request that has not answered within ten seconds. Today POST /engagements/{id}/work-records inserts a row and returns 201, and month-end shows duplicated minutes. Design the write contract so a retried submission cannot double-book: start from whether those columns give you a natural key, then specify the key you actually use, its scope and lifetime, the response on replay, the behaviour when the same key arrives with a different body, and what the app does on each outcome.

Approach

Name the actual fault: POST is not idempotent, and a ten-second timeout leaves the outcome unknown, because the row may have committed and only the response was lost. Retrying is correct client behaviour; making the retry safe is the server's job. Two mechanisms exist and they are not interchangeable — a natural key derived from the row's own grain, or an opaque client-supplied key — so the first question is which one this grain can support.
Test the natural key against the columns you were given, and reject it. The candidate is (engagement_id, principal_id, work_date), and nothing left over separates two legitimate entries: minutes and is_billable repeat honestly, because one person can book two distinct ninety-minute billable blocks on one day, and entered_at is server-assigned so it differs on every retry — it changes at exactly the moment a dedupe key must stay fixed. A unique index on that triple therefore rejects a real second entry, trading a visible bug for an invisible one: duplicated minutes surface at month-end, while silently refused minutes surface as an under-billed engagement nobody reconciles. Prefer a natural key wherever the grain genuinely has one; this grain does not, so the key has to come from the client.
Require an Idempotency-Key header and say so in the contract. With no natural key available, a request that arrives without one cannot be made retry-safe by anything the server knows, so refuse it with its own stable code rather than accepting it and hoping. The IETF header field is still a draft, so document your own semantics rather than pointing at a spec. Scope the key to (authenticated principal, endpoint, key) so one tenant cannot collide with another's, store a fingerprint of the request body beside it, and retain it at least as long as the client's maximum retry horizon; a key pruned before the last retry re-opens the duplicate window it existed to close.
Pin the client's half of the contract, because that is where this design usually fails in practice: the key is generated once per logical entry and reused byte-for-byte on every retry of it. An app that mints a fresh UUID per attempt satisfies the header and still double-books, and no server-side check can tell that apart from two genuine entries. Specify a deterministic construction the app can rebuild after a restart rather than one it holds in memory, or a crash between attempts reintroduces the duplicate.
Same key with a different body is a distinct 4xx with its own code, never a silent replay of the first response. These are two different bugs: a genuine retry must replay, and a key reused for new content is a client-side key-generation defect that stays invisible for months if you answer it with the original body.
Handle the concurrent in-flight duplicate, which is the case a single-threaded test never reaches: two copies arrive forty milliseconds apart, both miss the key lookup, both insert. Resolve it in the database rather than by check-then-act. Put a unique index on (principal_id, endpoint, idempotency_key), insert that row in the same transaction as the work record, and let the loser take the violation. Under READ COMMITTED in PostgreSQL the second insert blocks on the winner's uncommitted key row and then raises 23505 when the winner commits; that error aborts the transaction, so take a SAVEPOINT before the insert if you intend to read the winner back on the same connection. Note also that INSERT ... ON CONFLICT DO NOTHING returns no row on conflict, so the handler needs an explicit SELECT afterwards, or ON CONFLICT DO UPDATE ... RETURNING, to obtain the winning row.
Answer the loser that arrives before the winner has stored its response with a 409 carrying its own code and a Retry-After, not with a guessed body. Store each response with its status code so a later replay is byte-identical, and write the app's branch table: 201 and 200 both mean submitted; the different-body 4xx and the missing-key 4xx stop and surface to the user; a 409 against locked_at means re-read; the in-progress 409 retries after Retry-After; timeouts and 5xx retry with the same key and the same body. No branch may retry a terminal 4xx and no branch may drop a timeout.

Worked solution (30 min)

List the columns that could separate two entries on the same engagement, principal and date, strike out each with the reason it cannot, and write down the legitimate row pair a unique index on that triple would reject.
Write the handler as one transaction containing both the key row and the work record, and mark the commit point explicitly.
Enumerate four arrivals of the same request — first, retry after commit, retry after a crash before commit, and a concurrent duplicate whose winner has not yet stored a response — and give each a status code and body.
Write the different-body case, including how the body fingerprint is computed and which fields are excluded from it.
State the key's scope and retention, and tie the retention number to the client's retry horizon rather than to a round figure.
Write the app's branch table and check that no branch retries a terminal 4xx and none drops a timeout.

Expected result: A written finding that work_record has no natural key at this grain, so the endpoint requires a client-supplied Idempotency-Key scoped to (principal, endpoint, key), unique-indexed and stored with a body fingerprint and the response; replays return the original status and body; same-key-different-body returns a distinct 4xx; a request with no key is refused rather than accepted; the concurrent duplicate is settled by the unique violation rather than a pre-check, with a loser whose winner has not yet responded answered 409 plus Retry-After; and only timeouts, 5xx and that 409 retry.

Verify it yourself

Firing the same request fifty times concurrently leaves exactly one work_record row, one 201 and forty-nine 200s, all naming the same work_record_id.
Killing the process between the insert and the response leaves the retry returning the committed row rather than creating a second one.
Two genuinely distinct entries by the same principal on the same engagement and work_date, submitted under different keys, both persist.

Follow-up

The period has been billed and locked_at is set when a retry finally lands. What status does that retry get, and what does the app show the user?
What would have to be true of work_record for a natural key to be the better answer here, and what would you still keep the header for?
How long do you keep idempotency keys, and what exactly breaks on the day you prune one too early?

Credential minting stalls estate-wide from one client's provider

Hard · partial outage · timeouts · bulkheads · queueing

At 14:05 the access broker's p99 rises from 28 ms to 9.4 s. Error rate is only 3 percent. Connector runs and operator sessions for every client begin failing, yet only one client's identity provider is degraded: it accepts TCP connections and never replies. The broker handles five mint calls per second, of which one targets that provider, through a shared outbound worker pool of 50. Handler service time, measured from the start of each call, is unchanged. Diagnose in order and give the changes that bound the blast radius.

Approach

Separate slow from queued before anything else, because they need opposite responses. Split the broker's latency into time waiting for an outbound slot and time executing the call. Unchanged service time alongside a 300-fold p99 means the whole regression is queueing, so the target of the investigation is a saturated resource, not slow code.
Split every metric by target_system_id next. One client's provider owns all the non-completing calls; everyone else's calls are fast once they start. Combined with step one this gives the shape immediately: one dependency's calls occupy the shared pool and every other client waits behind them.
Do the arithmetic and check it against the timeline. One call per second that never completes fills all 50 slots in 50 seconds, which matches an onset measured in under a minute rather than gradual decay. If the numbers had not fit, saturation would not be the whole explanation and you would keep looking.
Confirm at the socket, since a hang and a failure are indistinguishable in an error rate. Connections in ESTABLISHED with bytes sent and nothing received distinguish a silent peer from a refused connection or a reset. Note explicitly that TCP keepalive will not rescue you here: the Linux default sends its first probe after two hours, so only an application-level read timeout bounds this at all, and its absence is the root defect.
Fix by bounding, in this order: a connect timeout and a read timeout on every outbound call, both set below the caller's own latency budget; a per-target concurrency limit so a single provider can occupy at most a small share of the pool; a circuit breaker keyed by target_system_id that opens on consecutive timeouts and half-opens with one probe; and a retry budget with full jitter, because three retries against a hanging peer triple the occupancy rather than improving the odds.
Then reconsider the blast radius rather than only the incident. A fail-closed broker on every request path is a shared dependency whose outbound concurrency must be partitioned per client, and a refusal for one client's target must surface as a typed result the caller can classify, not as a timeout that has already consumed a slot to produce.

Follow-up

Choose the read timeout and justify it with evidence. What breaks for a provider that is legitimately slow, and how do you tell the two apart?
The breaker is open for that client. What does an in-flight connector run observe, and which failure_class should it record?
Could the broker have kept serving every other client with no code change at all? What would that configuration have cost you on an ordinary day?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Scope is read from what you decided

Team size, service count and tickets closed say very little. Seniority shows in the decision you owned: what you chose not to build, which constraint you traded away, whose objection you had to resolve before anything could move. A large project where you executed someone else's plan is a small story.

How do you handle feedback from cross-functional teams, such as quality assurance or supply chain managers?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

Tell me about a time you identified an inefficiency in a workflow and implemented a technical solution to fix it.

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Describe a time you had to troubleshoot a complex technical issue under a tight deadline.

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

Describe a time you had to explain a technical concept to a non-technical stakeholder.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?