Interview guides · Black & Veatch

Black & Veatch Software Engineer Interview Questions & Guide 2026

Code reading practice for Black & Veatch software engineer prep: work out what unfamiliar code does before judging it.

The 60-second brief

A Software Engineer at Black & Veatch operates at the critical intersection of digital innovation and global infrastructure. Unlike traditional software roles focused solely on consumer applications, this position is deeply integrated into the lifecycle of complex, large-scale engineering projects, including advanced power generation, water systems, and infrastructure asset management. You are not just writing code; you are building the digital backbone that enables the design, monitoring, and optimization of the world’s most essential physical systems.

If the loop includes an asynchronous take-home, treat it as a code review of you rather than as a puzzle: structure, what you chose to test, and what you wrote down about the constraint you were working under. Hold the stated time box and say what you would have done with more of it.

Focus areas

Build at-least-once pipelines with explicit deduplication horizons
Keep money in integer minor units
Make every write idempotent under client retries

Overview

The impact of your work is tangible and high-stakes. Whether you are developing control systems for power plants, automating mechanical system simulations, or creating data-driven insights for asset reliability, your contributions directly influence the safety, efficiency, and sustainability of critical infrastructure. You will work within multidisciplinary teams, bridging the gap between hardware instrumentation, electrical systems, and software solutions to solve real-world engineering challenges.

The Black & Veatch interview process

Black & Veatch candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Deep Dives

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Discussions with Leads

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Questions & practice

Order a job dependency graph and find its critical path

Medium · topological-sort · dag-longest-path · cycle-detection · critical-path

A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.

Approach

Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order: earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E).
Second pass in reverse topological order for latest_finish, then slack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain. slack[v] = 0 is exactly the statement that some longest path runs through v; equivalently, the longest path through v has length T - slack[v].
The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by min(d, T - L_avoid(v)), where L_avoid(v) is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, while min(10, 15 - 14) predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan is max(T - d, L_avoid(v)).
Compute L_avoid(v) the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, since L_avoid(v) only ever matters through that max: set duration[v] := 0, recompute the makespan as T0(v) = max(T - duration[v], L_avoid(v)), and the gain is min(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.

Follow-up

Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?

Hold a tenant to a trailing sixty-second request limit

Medium · sliding-window · two-pointer · rate-limiting · tenant-skew

The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.

Approach

Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request.
Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate, prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact.
Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (tokens, last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual.
Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.

Worked solution (25 min)

Implement the deque version and instrument the per-request pop count, then confirm total pops equal total pushes over a run.
Generate a burst that places R requests in the last 100 ms of one minute and R more in the first 100 ms of the next.
Run that burst through the exact deque, a fixed 60-second counter, and the weighted two-bucket estimate, recording admissions in the trailing 60 seconds at every instant.
Size the memory as R x active tenants x 8 bytes at R = 1,000 and 200,000 tenants, and compare it against what a token bucket would need.

Expected result: The exact deque never admits more than R in any trailing 60-second window. The fixed counter admits close to 2R across the boundary. The weighted estimate lands between the two on this burst and approaches 2R once the previous window's requests are packed at its end.

Verify it yourself

Total pops equal total pushes, which is what makes the amortised O(1) claim true rather than merely asserted.
Sliding the whole burst by 30 seconds changes the fixed counter's result and leaves the deque's unchanged.
Running two processes each limiting to R/2, against a tenant that sends everything to one of them, shows that tenant throttled at half its contractual limit.

Follow-up

One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
Quotas rather than rate limits: the check is select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes.
How do you return an accurate Retry-After from the exact algorithm without a second scan?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
What makes a re-run over the same day produce byte-identical rollups?

Rebuild an hourly rollup with deduplication and late-arrival accounting

Hard · window functions · deduplication · event time vs ingest time · watermarks

From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.

Approach

Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed: row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs.
Bucket on occurred_at and nothing else, and pin the timezone explicitly. date_trunc('hour', timestamptz) truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argument date_trunc('hour', occurred_at, 'UTC') on PostgreSQL 16 or later, or date_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC' before that. Filter environment = 'production' explicitly, since metering covers three environments and billing covers one.
Aggregate to the grain with sum(quantity), count(*) and max(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include.
Compute the late share inside the dedup-and-aggregate step as a conditional aggregate, sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows: sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back.
Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.

Worked solution (40 min)

Generate a day of events for one tenant across three workspaces and two SKUs, inject 2% duplicates sharing an idempotency_key with a later ingested_at, and inject 1% whose ingested_at is six hours past occurred_at.
Write the dedup CTE alone and confirm its row count drops by exactly the injected duplicate count.
Aggregate and compare quantity_sum against a ground truth computed outside SQL from the generator's own records.
Add the two window and filtered aggregates, and confirm the final hour's running total equals the ungrouped day total for each (workspace, sku).
Run the whole query twice over static data and diff the two outputs.
Re-run with the session TimeZone set to a non-UTC zone and confirm hour_start does not move.

Expected result: One query: a dedup CTE using row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id); aggregation by (workspace_id, sku, hour_start) producing quantity_sum, event_count and max(ingested_at) plus a filtered late-quantity aggregate; a running total as a window over the aggregated rows; and an explicit statement that the partition-scoped daily unique index cannot catch a duplicate that crosses the ingest-day boundary or arrives in a later replay.

Verify it yourself

Row count after dedup equals raw count minus injected duplicates, exactly.
The running total at the final hour equals the ungrouped day total for that (workspace, sku).
Two consecutive runs over static data return byte-identical rows.
Every hour_start is hour-aligned in UTC and unchanged by the session's TimeZone setting.

Follow-up

A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?

Explain why the metering dashboard scans every daily partition

Medium · indexing · partition pruning · sargability · tenant skew

usage_event is range-partitioned daily on ingested_at and holds tenant_id, workspace_id, environment, sku, quantity numeric(20,6), occurred_at and ingested_at. The only relevant index is on (occurred_at). A dashboard runs select sku, sum(quantity) from usage_event where tenant_id = $1 and date_trunc('hour', occurred_at) >= $2 and environment = 'production' group by sku, and EXPLAIN shows a sequential scan of every partition. Give each distinct reason, rewrite the predicate so an index can serve it, propose the index, and state the write cost its column order adds.

Approach

Separate the three causes rather than blaming one. First, date_trunc('hour', occurred_at) wraps the column, so the predicate is not sargable against a btree on the bare column. Second, pruning keys off ingested_at while the query constrains occurred_at, so no partition can be excluded. Third, even made sargable, (occurred_at) is not tenant-leading, so for one tenant among thousands the scan reads the whole time range and discards nearly all of it.
Rewrite the bound carefully, because the obvious rewrite is only conditionally equivalent. date_trunc('hour', x) >= $2 equals x >= $2 only when $2 is already hour-aligned; for an arbitrary $2 it means x >= date_trunc('hour', $2) + interval '1 hour'. Normalise the parameter in the caller and leave the column bare.
Restore pruning with a second, redundant predicate on the partition key: ingested_at >= $2 - interval '<late-data horizon>'. State both sides of it. It prunes to a handful of partitions, and it silently omits any event whose ingest lagged past that horizon, which is precisely what a producer replay produces. Either document the horizon as a stated bound, or partition on occurred_at and move the problem into the dedup window instead.
Propose (tenant_id, occurred_at) include (sku, quantity) per partition. A partial index where environment = 'production' mostly saves size rather than selectivity, since production dominates the three environments; take it if non-production is a meaningful share and skip it otherwise.
Price the write path honestly. At roughly 250M rows/day each extra index is another insert plus WAL per row, and a tenant-leading key scatters inserts across one hot leaf per active tenant instead of appending to a single rightmost leaf, so page dirtying and random I/O both rise. An INCLUDE payload widens every leaf entry and enlarges the index accordingly.
Add the index-only-scan caveat before someone reports it as a regression: on a freshly appended table the visibility map is not yet set for recent pages, so the INCLUDE columns still cost heap fetches until autovacuum has been through, and the newest hour is exactly the data the dashboard reads.

Follow-up

CREATE INDEX CONCURRENTLY is not supported on a partitioned parent. Give the sequence that gets this index onto 400 existing partitions without blocking ingest.
One tenant holds 200 times the median row count and the dashboard still times out for them with the index in place. What changes?
Should this read hit usage_rollup_hourly instead? State what that costs in freshness and what the watermark lets you promise.

Describe a situation where you had to learn a new technical standard or software framework under a tight deadline.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Walk us through a time you had to identify a bottleneck in a complex system.

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Make run creation safe when the client times out and retries

Medium · idempotency · retries · outbox

POST /v1/runs inserts a job_run row and dispatches a sandbox. The caller is a customer's CI pipeline with a 10-second timeout and three automatic retries. A run that takes 12 seconds to start returns after the client has already given up, so the retry starts a second sandbox and bills it. Redesign the endpoint: the key the client sends, its scope and storage, what happens when the same key arrives with a different body, what happens when a retry arrives while the first request is still in flight, and how long the key stays honoured.

Approach

Name the defect before designing: the client's timeout is an unknown outcome, and the naive shape gives it no way to ask whether the earlier call landed, so its only choices are to duplicate the run or to drop it.
Require an Idempotency-Key header scoped to (tenant_id, key) rather than to the credential, so a rotated or replaced credential still replays. Store the key with a request fingerprint, SHA-256 over method, path and the canonicalised body, plus state, the stored response status and body, and the resulting run_id.
Make the claim a write rather than a read: insert the key row inside the same transaction as the run under unique (tenant_id, idempotency_key). The loser of a concurrent race takes a unique violation and then reads the winner's row, which is what makes the duplicate case deterministic instead of timing-dependent.
Define three outcomes precisely. Same key, same fingerprint, completed: replay the stored response with a header marking it a replay. Same key, different fingerprint: 422 with an idempotency_key_reuse code, never the stored response. Same key still in flight: 409 with Retry-After, because blocking the second caller holds a connection for the duration of someone else's request.
Close the dispatch gap with an outbox row committed alongside the run and a relay that enqueues it. Commit-then-publish loses the dispatch if the process dies in between; publish-then-commit hands a worker a run_id that does not exist yet.
Set retention from the client, not from storage cost: the key must outlive the caller's worst-case total retry window, so publish a number such as 24 hours and state that a key reused after expiry legitimately creates a second run.

Worked solution (30 min)

Draw the timeline of the failing case: commit at t=11s, client abandons at t=10s, retry at t=10.1s, and mark where the second sandbox starts.
Write the idempotency table DDL with the unique constraint and the fingerprint column, then the single transaction that inserts both rows.
Enumerate the four arrival cases (first, replay, conflicting body, in-flight) and write the exact status, headers and body for each.
Add the outbox row to the same transaction and describe the relay's at-least-once behaviour and why the worker must tolerate a duplicate dispatch.
State the retention horizon and the sentence that goes in the public docs about key reuse after expiry.

Expected result: One transaction inserting job_run plus an idempotency row under unique (tenant_id, idempotency_key) and an outbox row; replay returns the stored response, a conflicting fingerprint returns 422, an in-flight duplicate returns 409 with Retry-After, and the horizon is published as a number.

Verify it yourself

The concurrent-duplicate case is resolved by a constraint violation, not by a prior SELECT.
No path returns the stored response for a different request body.
There is no window where the run is committed but the dispatch is not, or the reverse.
The key scope contains the tenant, and nothing in it is derived from the credential.

Follow-up

The SDK generates a fresh key on every attempt rather than reusing one. What in your response surface makes that visible instead of silently doubling the bill?
How does this interact with job_run.parent_run_id: is an automatic retry of a failed run the same object as a client retry of the request?

A rare job-run overwrite that logging makes disappear

Hard · fencing tokens · leases · concurrency

About one job run in fifty thousand records billable_seconds matching no observed sandbox lifetime, and a few rows show worker_id changing after finished_at was already set. It does not reproduce: debug logging around the terminal write made it vanish for two weeks before it returned. Runs last from 200 ms to 30 minutes, the lease is 60 seconds and is renewed while a run executes. Give an ordered checklist, the mechanism, and a fix that makes the illegal write impossible rather than merely rarer.

Approach

Mine the evidence instead of chasing a repro: select rows where updated_at is later than finished_at, or where a terminal status was written twice, and join them to attempt history to recover both writer identities. The defect has already happened tens of times and the rows are the recording.
State the signature before measuring it. If the cause is a lease that expired while the original worker was stalled, affected runs should cluster where the gap between the last renewal and the terminal write exceeds the lease, and should correlate with worker pause metrics rather than with workload shape.
Read the disappearance honestly. Logging inside the window changed the timing and lowered the probability; it is evidence about how narrow the window is, not a fix. Reproduce by widening the window on purpose, shortening the lease and injecting a pause between sandbox exit and the terminal write, rather than by adding more instrumentation.
Name the mechanism precisely: the lease expires during a stall such as a long garbage-collection pause or a brief partition, the run is re-dispatched, and the original worker then wakes and writes its terminal state over the new attempt's row. A lease alone cannot stop this, because the check and the write are separated by the stall.
Fix by fencing the write itself: UPDATE job_run SET status = $2, finished_at = $3, billable_seconds = $4 WHERE run_id = $1 AND status = 'running' AND lease_token = $5, with zero rows affected interpreted as having been fenced rather than as success. The token lives on the row so the store arbitrates, not the worker's memory.
Keep the state machine honest: a retry inserts a new row pointing at parent_run_id rather than resetting the old one, and a run whose worker vanished terminates as lost with billable_seconds null, because recording failure asserts an outcome nobody observed and then bills and retries on that assertion.

Follow-up

The supervisor also emits a usage event on completion. What does the fenced worker do about the event it already emitted, and how does metering absorb it?
Lease renewal is itself a network call. What happens when a renewal times out, and how does the worker decide whether it still holds the lease?
Why is lengthening the lease past the longest legitimate run the wrong lever, and what breaks if you do it anyway?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

Run the same sub-skill decomposition on the second-largest gap in half the time.
Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

Describe your experience with control systems architecture and how you ensure software-hardware integration.

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Tell us about a time you had to resolve a disagreement regarding a technical design choice within a project team.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

How do you explain complex technical software concepts to a civil or mechanical engineer who lacks a programming background?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

How do you handle project requirements that change due to physical or environmental constraints?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?