Interview guides · Apollo Technology Solutions

Apollo Technology Solutions Software Engineer Interview Questions & Guide 2026

Apollo Technology Solutions software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

At Apollo Technology Solutions, the Software Engineer role centers on delivering robust, scalable, and user-centric technical solutions for the company's diverse client base. You will serve as a critical bridge between complex business requirements and high-performance digital infrastructure. Whether you are building custom web applications, managing enterprise platforms like Salesforce or ServiceNow, or maintaining legacy mainframe systems, your work affects the efficiency and success of the organizations Apollo Technology Solutions serves.

Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.

Focus areas

Detect concurrent edits instead of losing writes
Make every write idempotent under retry
Trace a symptom to a mechanism under load

Overview

This role requires a blend of deep technical expertise and a consultative mindset. The role calls for clean code along with attention to accessibility, platform stability, and long-term maintainability. You will operate in a dynamic, project-driven environment where your ability to solve problems, adapt to new frameworks, and collaborate with cross-functional teams will be the primary driver of your success.

The Apollo Technology Solutions interview process

Apollo Technology Solutions candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Discussions

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Final Evaluations

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Saying 'eventually consistent' without naming the anomaly a user would see

Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.

Questions & practice

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Follow-up

The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.

Worked solution (25 min)

Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.

Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.

Verify it yourself

The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
The heap result matches a full sort of the tenant sums on a sample.
The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).

Follow-up

The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Follow-up

A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
How would you compute both answers incrementally as records arrive rather than in a daily batch?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.

Follow-up

The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
How do you verify the backfill actually finished, given rows are still being inserted while it runs?
A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.

Worked solution (20 min)

Create the PostgreSQL partial unique index, insert a live row, soft delete it, and insert the same address again.
Repeat the delete-and-reinsert cycle three times and confirm three deleted rows coexist with exactly one live row.
Write the MySQL form with the discriminator, then deliberately reverse the polarity so live rows carry NULL, and show two live duplicates commit.
Attempt a second live insert on both engines and map the resulting 23505 / ER_DUP_ENTRY to the 409 the handler should return.

Expected result: PostgreSQL rejects only the second live row and admits unlimited deleted ones. MySQL behaves identically with UNIQUE (tenant_id, email, deleted_marker) where live = 0; with the polarity reversed, two live rows commit and nothing afterwards can tell them apart.

Verify it yourself

Group live rows by (tenant_id, lower(email)) after a randomised insert/delete run — no group exceeds one.
Inserting 'User@x.test' after 'user@x.test' is rejected; if it is accepted, the case handling is in application code rather than in the index.
EXPLAIN a login lookup and confirm the partial index is chosen — it is only usable by queries that carry the deleted_at IS NULL predicate, so a query without it falls back to a scan.

Follow-up

A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
How do you honour an erasure request while resource_revision.actor_user_id still references this table?
What changes if a user may hold membership in two tenants?

How do you ensure your code meets Web Accessibility standards during the development lifecycle?

Medium

Approach

Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Walk us through a scenario where you had to choose between two different technical architectures. What factors influenced your decision?

Medium

Approach

Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

What are the key differences between a standard web application and a ServiceNow-based solution?

Medium

Approach

Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Give the resource write endpoints a failure taxonomy clients can act on

Easy · error taxonomy · status codes · retries

Two callers use POST and PATCH /v1/resources: a server-side SDK that retries automatically, and a browser app that shows the user a message. Today every failure is a 500 carrying prose. Define the error contract for a malformed body, a field that fails validation, an expired token, a token whose auth_version no longer matches, a resource_id owned by another tenant, a version mismatch on update, an idempotency key reused with a different body, an exceeded tenant quota, and a read replica that has not caught up. Deliverable: the envelope, the status per case, and what each caller does.

Approach

Split the envelope by audience: a stable code enum for programs, a message documented as human-only and free to change wording, a request_id that joins to resource_revision.request_id and the trace, and a details array of field paths for validation failures. Publish the negative rules too - clients never branch on message, and an unrecognised code falls back to the status class.
Assign status by who has to change something. 400 for bytes that do not parse, 422 for a body that parses and violates a rule, 401 for a token that no longer authenticates - expiry and an auth_version mismatch are the same instruction, re-authenticate - 403 for a scope the principal lacks, 404 rather than 403 for another tenant's resource_id because 403 confirms the id exists, 412 for a failed If-Match (409 if the version travels in the body instead), 422 for a key reused with a different fingerprint, 429 for quota.
Derive retryability from whether the outcome is unknown, not from the status number. A timeout or a 5xx on a write is unknown - the transaction may have committed and the response lost - so the only safe retry is one carrying the same idempotency key. Every 4xx except 429 is deterministic, and retrying it only spends the caller's remaining deadline.
Refuse to model replica lag as an error. Read-after-write is held by pinning the session to the primary for a short window, not by a 404 the SDK retries into a loop; if staleness must be visible, expose it as a watermark on a 200, because a failure code invites a retry that cannot fix it.
Write both caller behaviours into the published contract: the SDK retries only 429, 503 and timeouts, with capped backoff and full jitter bounded by the deadline it was given; the browser stops and shows message, except on 412, where it must re-read the resource and rebase the edit rather than resubmit.

Worked solution (20 min)

Write the envelope as four named fields and state which are present on every error response without exception.
Fill a nine-row table: condition, status, code string, retryable yes/no, and the schedule or the reason a retry cannot help.
For each retryable row, name what makes the retry safe - HTTP method semantics, or an idempotency key.
Write the two caller policies as pseudocode: which codes the SDK retries, and what the browser does on 412 and on 429.

Expected result: An envelope with a code enum, a human message, a request_id and per-field details; nine conditions mapped to distinct statuses; retry advice attached to the outcome rather than to the status class; replica lag handled as consistency, not as an error.

Verify it yourself

Every row marked retryable names the mechanism that makes repeating it safe.
A resource_id from another tenant and a resource_id that does not exist are indistinguishable from outside.
Adding a new code does not change any existing code's meaning or any status already published.

Follow-up

A customer reports two resources created from one click. Which of your status codes could have produced that, and what in the contract permitted the client's reading?
You need a tenth error code next quarter without a version bump. What must v1 already have said for that to be non-breaking?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency · fencing

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.

Follow-up

The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
You now count takeovers. What alert fires on that counter, and at what threshold?
What breaks if you simply raise the lease to five minutes?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Outage answers are graded on instruments

Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.

Describe a time you encountered a significant performance bottleneck. How did you identify the root cause and implement a fix?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Describe a time you had to migrate data or functionality from a legacy system to a modern platform.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

How do you approach gathering requirements from non-technical stakeholders to ensure the final product meets their needs?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?