Interview guides · Atkore

Atkore Software Engineer Interview Questions & Guide 2026

Concurrency drills for Atkore software engineer prep: races, lock ordering, and what a check-then-act sequence assumes.

The 60-second brief

As a Software Engineer at Atkore, you are a critical contributor to the digital infrastructure that supports one of the nation’s leading manufacturers of electrical, safety, and infrastructure solutions. Your work goes beyond writing code; it involves building the technical tools and systems that enable Atkore to operate with greater efficiency, safety, and precision. You will be expected to bridge the gap between complex engineering challenges and practical, scalable software solutions.

Ask how many rounds there are and what each one is before you plan your weeks, because the composition is what your hours should follow. A loop with three coding rounds and one short design conversation deserves a different split from the reverse, and whoever schedules it will usually say plainly which it is.

Focus areas

Bound every outbound call with a timeout
Choose indexes from the query's access path
Make every write idempotent under retry

Overview

This role is inherently collaborative, often requiring you to work alongside cross-functional teams, including product managers, drafters, and operations leaders. Whether you are optimizing internal workflows or integrating specialized design software—such as Revit—into the company's ecosystem, your impact is felt across the organization. Success in this role requires a balance of technical aptitude and the ability to communicate how your software solutions solve real-world industrial problems.

The Atkore interview process

Atkore candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screens

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Assessments

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Behavioral Assessments

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Panel Interviews

Coding rounds mostly set a floor. They decide whether you clear the bar, not where you land on the ladder. Level tends to come out of the design discussion and the ownership stories, so the question worth auditing beforehand is whether the scope you describe matches the scope of the job. Work that stops at your own service, or a story whose hard part was writing the code rather than getting several people to agree on an interface, reads a level below where you think you are interviewing, and that gap is usually resolved downwards.

What to demonstrate

Whether the largest thing you describe owning ran end to end — the decision, the migration path, the rollout, and what you did when it went wrong — or stopped at the change you merged
Whether design answers include what you would not build, what you would defer, and what you would measure before committing, rather than only what the boxes are
Whether a disagreement in a story was settled with something checkable — a benchmark, a prototype, a written proposal — instead of by seniority or by waiting it out
Whether you can say which calls you made alone and which you escalated, and why the line sat where it did

How to prepare

Write your largest piece of owned work as a timeline of decisions — who decided what, when, and what you did when the plan broke — then delete every sentence whose subject is "we" and see how much survives
Take one system you know well and drill the migration answer: how old and new paths run side by side under live traffic, how you compare their outputs, what the rollback is once writes are going to both, and which step you would not automate
Map each line of the ladder in the job posting to a specific thing you have done, find the line you cannot support, and prepare the closest evidence you have plus an honest account of the gap

Meetings with Management

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Going silent while thinking

Narrate the candidates and why you are discarding them, even in fragments: sorting first would make this a two-pointer scan, but it destroys the original indices, which the output needs. From the other side of the table, a candidate thinking hard and a candidate stuck are indistinguishable until one of them speaks.

Questions & practice

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Follow-up

A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
How would you compute both answers incrementally as records arrive rather than in a daily batch?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Follow-up

The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
How would you run this continuously at low cost instead of only as incident response?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Worked solution (20 min)

Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.

Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.

Verify it yourself

After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.

Follow-up

The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
How would you make the window survive a process restart, and is it worth the cost?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.

Follow-up

A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
How do you honour an erasure request while resource_revision.actor_user_id still references this table?
What changes if a user may hold membership in two tenants?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.

Worked solution (25 min)

Load 2M resource rows across 5k owners in one tenant, run the query under EXPLAIN (ANALYZE, BUFFERS), and record the node reading the most rows plus the Sort Method line.
Create (tenant_id, owner_user_id, created_at DESC) concurrently and re-run, confirming the Sort node is gone and actual rows fall to about 20.
Run the two negative cases and capture the plan for each.
Re-run the original tenant listing query to confirm the new index has not displaced the index that query depends on.

Expected result: With (tenant_id, owner_user_id, created_at DESC) the plan is an Index Scan with no Sort, roughly 20 rows and a handful of buffers, sub-millisecond. owner_user_id alone still scans; ORDER BY updated_at seeks on the pair and then sorts that owner's rows.

Verify it yourself

Actual rows at the leaf node is on the order of the LIMIT, not the tenant's row count.
No Sort node appears above the index scan.
Buffers read stays constant as one owner's row count grows; if it grows with it, the ORDER BY is not being satisfied by the index.
pg_stat_user_indexes shows idx_scan climbing on the new index and not on the listing index for this endpoint.

Follow-up

90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
A dashboard runs this for 40 owners in one page load. What changes about the design?
How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?

Can you describe a complex technical challenge you faced and how you overcame it?

Medium

Approach

Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

What is your history with Revit?

Medium

Approach

Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Choose what to break when replication lag reaches forty seconds

Hard · replication lag · read-after-write · consistency vs availability · load shedding

Reads are served from two replicas: 14k requests/second, about 85% absorbed by cache, so roughly 2.1k reads/second reach the database. Writes go to the primary at 1.2k/second. A tenant's backfill drives replication lag from under 100 ms to 40 seconds and it is still climbing. Sessions that have just written are pinned to the primary. Decide, endpoint class by endpoint class, whether to serve stale, fail, or route to the primary, and justify each choice with the load it adds to the primary. Then state what you would have built beforehand.

Approach

Establish blast radius before cause, because mitigation and diagnosis have different deadlines. The decisive arithmetic is what happens if the database reads move to the primary: 2.1k reads/second on top of 1.2k writes/second roughly triples its operation count, on the node already absorbing the backfill that caused this. Reads and writes are not equal in cost, so treat that as an argument against a blanket move rather than as a capacity model - but it is enough to rule out routing everything to the primary.
Classify endpoints by what staleness costs, not by how important they feel. Reads whose staleness is invisible - listings, search, counters - stay on the replica and return the watermark so the client can tell. Reads that immediately follow that same session's write keep their primary pin, which is a small bounded slice of traffic rather than the whole 2.1k/second. Reads that feed a decision with a side effect - authorisation, quota, the read half of a read-modify-write - must not be stale at all, because a 40-second-old permission row is the stale-permission failure wearing a different costume; those go to the primary or fail.
Shed instead of queueing. If the must-be-fresh class alone exceeds the primary's headroom, refuse its lowest-value slice with 503 and a retry-after. A request queued behind a saturated primary holds a connection for a client that has already given up, and the retry storm that follows is what turns degradation into an outage. Bound the connection pool per role so the read fallback cannot consume the write path's connections - that bulkhead is the single decision that determines whether writes survive the next ten minutes.
Attack the cause in parallel, since it is the one thing that can be stopped. The backfill is the load generator. A backfill that reads replication lag as its throttle signal and pauses above a threshold would have made this a non-event, with batch sizes small enough that each batch's write volume is a fraction of what a replica can apply per second. That is most of the answer to what should have existed beforehand.
Name the mechanism you would prefer over session pinning. Capture the write position at commit and require the read path to be at or past it: compare the primary's pg_current_wal_lsn() at commit time against the replica's pg_last_wal_replay_lsn(), and fall back to the primary only for the specific request that is ahead of the replica. Session pinning is the cheap approximation and it over-pins - every read in the window goes to the primary whether or not it needed to, which is a share of the cost being paid right now.

Worked solution (35 min)

List the endpoints in three buckets - staleness invisible, staleness visible to the writer only, staleness unsafe - and attach the share of the 2.1k reads/second each bucket carries.
Compute the primary's operation count under each routing option and mark which options are arithmetically available.
Write the pin rule and its window, then the shed rule: which slice, what status code, what retry-after.
Write the backfill's throttle predicate against a measured lag value, including its pause threshold and resume condition.

Expected result: Staleness-invisible reads stay on the replica and return a watermark. Reads following the session's own write keep a bounded primary pin. Reads that gate a side effect go to the primary or fail, with the lowest-value slice shed as 503 plus retry-after if headroom runs out. Connection pools are bounded per role so the read fallback cannot starve writes. The backfill is throttled on measured replication lag. The durable fix is a commit-position token compared against the replica's replay position, replacing blanket session pinning.

Verify it yourself

The traffic routed to the primary under the plan is stated as a rate, not as a category.
No element of the plan increases the write path's connection usage.
The backfill throttle references a measured lag value and has an explicit resume condition.
Every endpoint left on a replica returns something with which a client can detect staleness.

Follow-up

Lag returns to normal in nine minutes. Which mitigation do you remove first, and which one stays permanently?
A user reports their change did not save, and the write committed. Trace the path that produces that report and name the signal that would have shown it before the report arrived.
The replica is 40 seconds behind but otherwise healthy. Do you take it out of rotation? What does that do to the other replica's lag?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency · fencing

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.

Follow-up

The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
You now count takeovers. What alert fires on that counter, and at what threshold?
What breaks if you simply raise the lease to five minutes?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

What the behavioural round is actually measuring

Every story you tell gets read for blast radius and judgement: what could have broken, who else it touched, what you knew at the moment you decided. Nobody can audit your code in an hour, so they audit your reasoning instead. Pick work where the call was genuinely yours and the consequences were real enough to remember.

How do you adapt your technical skills to support non-technical stakeholders?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Name a project you worked on with others where there was a disagreement. How did you handle it?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Tell us about a time you had to confront your supervisor about them being wrong. How did you handle it and what was the process you went through?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

I was asked about previous leadership experiences.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?