Interview guides · Australian National University

Australian National University Software Engineer Interview Questions & Guide 2026

Australian National University software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

A Software Engineer at the Australian National University (ANU) plays a pivotal role in bridging the gap between academic research and real-world technical implementation. You are not just writing code; you are building the digital infrastructure that supports world-class scientific inquiry, administrative excellence, and high-impact computational projects. Whether you are developing plugins for emerging AI tools or managing complex national computational infrastructure, your work directly impacts the university’s ability to lead in global innovation.

The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.

Focus areas

Diff a roster snapshot before applying any deletes
Make outbound grade publication idempotent and ordered
Sandbox untrusted learner code with hard limits

Overview

This role is unique because it often requires a blend of rigorous software engineering practices and an appreciation for the academic environment. You will frequently collaborate with professors, researchers, and cross-functional teams to translate abstract scientific requirements into scalable, robust software solutions. Expect to work in an environment where precision, clarity in documentation, and the ability to explain technical concepts to non-technical stakeholders are as important as your coding proficiency.

The Australian National University interview process

Australian National University candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Assessment

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Technical Evaluations

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Discussions

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Where candidates lose points

Treating the launch subject identifier as a global user id, or merging identities on email.

The subject claim is stable only within its issuer, so the same human arriving from a second platform is a different subject and must be a second external identity bound to the same person. Email is worse than useless as a merge key here: privacy settings frequently suppress it from the launch entirely, districts recycle addresses between graduating and incoming students, and younger learners often have none. A merge on email eventually joins two children's work into one account, which is both a grade bug and a privacy incident.

Autoscaling on average utilisation against a bell-shaped step function.

Scale-up latency (scheduling, image pull, process and JIT warm-up, connection pool and cache fill) is typically tens of seconds to minutes, while the demand step is under a minute and repeats on a published schedule. By the time the metric crosses a threshold, the period has started and the queue has already built. The workable answer is a schedule-driven pre-warm derived from the tenant calendar plus a load shedder that protects writes, not a more aggressive threshold.

Abandoning working code to chase the optimal solution

Get the straightforward version correct, state its complexity, and only then optimise, keeping the working version until the faster one passes the same cases. A correct quadratic solution with a stated path to linear beats a half-written optimal one that never ran.

Starting work without saying what you are about to spend time on

State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.

Questions & practice

Diff a roster snapshot against live enrollments in one pass

Easy · hash join · diffing · soft delete

A tenant's bulk roster snapshot has been normalised into N incoming enrollment records carrying section_id, person_id, role (learner, instructor, aide, observer), source, begin_date and end_date. You also hold M stored enrollment rows for that tenant where deleted_at IS NULL. N and M are each up to 1,000,000. In one pass over the incoming records, classify each as an add, an update or a no-op, and produce the set of stored rows the snapshot never asserts, which becomes deletes_proposed on roster_sync_run. State your time and space bounds in bytes.

Approach

Fix the key first, because the schema already decided it: the partial unique index is (tenant_id, section_id, person_id, role) WHERE deleted_at IS NULL, so role is part of the identity. One human legitimately holds two live rows in a section (instructor and observer), and keying on (section_id, person_id) alone collapses them into one, emitting a spurious update and a spurious delete on every run.
Load the M stored rows into an open-addressed hash map from that 3-tuple to (enrollment_id, comparable_fields, seen_flag). Packed, the key is 24 bytes and the payload about 8, so at a 0.7 load factor that is roughly 45-50 MB for a million rows; boxed objects in a JVM or CPython map cost 5-10x that, which is the number that decides whether this fits in a worker.
Stream the incoming records and probe once each: miss is an add, hit with equal comparable fields is a no-op, hit with any difference is an update. Set the seen flag on hit. After the stream, the stored entries still unseen are deletes_proposed; count them and divide by M to get the fraction the gate is evaluated on.
Handle revival explicitly: an incoming record that misses the live map may still match a soft-deleted row. Look that up and clear deleted_at in place rather than inserting a new enrollment_id, because the partial index constrains only live rows and will not catch the duplicate — the history simply fragments into a new id at every term boundary.
Complexity is O(N + M) expected time and O(M) space. State the alternative and its crossover: sorting both sides and merging is O(N log N + M log M) with O(1) working memory beyond the sort, and it wins as soon as M stops fitting in RAM, since an external sort degrades gracefully where a hash map does not.
Guard the semantics before any of this runs. Absence means deletion only for a bulk feed; on a delta feed absence is silence and only an explicit tobedeleted status removes anything. A tenant that switches feed modes mid-term will otherwise have one delta file read as a snapshot, which proposes deleting almost everything.

Follow-up

The upstream system changes its external_sourced_id scheme, so every incoming row looks new and every stored row looks absent. How do you reconcile without creating duplicate persons?
The apply aborts halfway through. What makes the next run finish rather than double-apply, and what does snapshot_digest do for you?
M grows to 50 million for one tenant. Which of your two algorithms do you run, and what changes about the delete detection?

Dispatch a bell-driven autograde burst under per-tenant caps

Hard · scheduling · heap of queues · fairness · np-hard

Thirty thousand autograde jobs enqueue over 90 seconds as a district's periods end. Each carries tenant_id, enqueued_at, a language-dependent wall_limit_seconds between 2 and 15 with a mean near 3, and a deadline equal to its section's period end. You have 400 one-shot workers, jobs cannot be preempted once started, and no tenant may hold more than 20 percent of the workers at once. Choose the dispatch order that maximises jobs returning feedback before their deadline, give the data structure and its per-dispatch cost, and say plainly whether that order is optimal.

Approach

Do the capacity arithmetic first, because it reframes the question. Four hundred workers at a 3-second mean serve about 133 jobs per second; 30,000 arrivals over 90 seconds is 333 per second. Twelve thousand are served during the burst, leaving 18,000 queued, which drains in about 135 more seconds. The system is oversubscribed by roughly two and a half times, so the scheduler is not deciding whether anyone misses a deadline — it is deciding who.
Choose earliest-deadline-first and then state its standing precisely. On a single machine with preemption, EDF meets every deadline whenever any schedule can. This is neither: 400 machines and non-preemptive jobs of unequal length. Minimising the number of late jobs on parallel identical machines is NP-hard — even minimising makespan on two machines reduces from PARTITION — so EDF here is a defensible heuristic, not an optimum. On one machine, Moore-Hodgson would give the exact optimum in O(n log n); that exactness does not survive the move to 400.
Enforce the cap with a heap of queues rather than by scanning. Keep one min-heap per tenant ordered by deadline, and a global min-heap holding at most one entry per eligible tenant keyed on that tenant's head deadline. Dispatch pops the global head, pops that tenant's head job, increments its in-flight count, and re-pushes the tenant only if it is still under 80 workers. On completion, decrement and re-insert the tenant if it was parked with work waiting. Each dispatch is O(log T + log J_t) instead of O(J), and the cap costs a counter comparison rather than a rescan.
Add admission control, which is where most of the win actually is. A job whose now + wall_limit_seconds already exceeds its deadline cannot return feedback in time no matter what, so running it burns a worker slot that a still-feasible job could have used. Route it to a graded-after-the-bell path immediately. This is the Moore-Hodgson instinct — discard the jobs you cannot save — applied as a filter rather than as an exact algorithm.
Break ties on enqueued_at ascending so the order is deterministic and a job cannot be starved by a stream of equal-deadline arrivals. Without a deterministic tiebreak the same burst replays differently, and an operator comparing two incidents cannot tell a scheduling change from noise.
Give the honest ceiling. No dispatch order makes a 2.5x oversubscription feasible; the levers are worker count, wall limits per language, and whether feedback is required before the bell at all. A candidate who presents EDF as the fix, rather than as the allocation of an unavoidable shortfall, has answered a different question.

Worked solution (45 min)

Compute service rate, arrival rate, peak backlog and drain time from the stated numbers, and write the oversubscription factor down before designing anything.
Implement the two-level heap with per-tenant in-flight counters, plus the admission filter that rejects jobs already past feasibility at dispatch time.
Simulate the 90-second burst against 400 workers with a sampled wall-limit distribution, recording on-time count, per-tenant miss rate and the tail completion time.
Re-run the same trace three ways — FIFO, pure EDF with no cap, and EDF with the cap and admission filter — and tabulate the three outcomes.
Re-run the skewed case where one tenant owns two thirds of the jobs, and compare the cap's effect on that tenant against its effect on everyone else.

Expected result: EDF with admission control beats FIFO on on-time count because it stops spending workers on already-doomed jobs; the cap lowers the dominant tenant's on-time count and raises everyone else's, leaving the global total roughly flat or slightly lower — which is the fairness trade being bought, stated as such rather than hidden.

Verify it yourself

No tenant ever exceeds 80 concurrent jobs at any instant in the simulated trace.
Total dispatched plus total admission-rejected equals 30,000 exactly.
Tail completion time lands near the 225-second figure the arithmetic predicted, or the simulation and the model disagree and one of them is wrong.
Replaying the identical trace twice produces an identical dispatch order, which the enqueued_at tiebreak is there to guarantee.

Follow-up

One tenant submits 20,000 of the 30,000 jobs. Show what your cap does to the other tenants' deadline-miss rate versus pure EDF.
Workers become preemptible with a checkpoint. Does EDF's optimality argument come back, and under exactly what conditions?
A job exceeds its wall limit and is killed. Does it re-enter the queue, and what does your answer do to the deadline guarantees of everything behind it?

Detect two-device autosave interleaving in an event stream

Medium · sliding window · event time · watermarks

Autosave events arrive on a stream as (attempt_id, client_instance_id, seq, received_at) at roughly 4,000 events per second, out of event-time order by at most 120 seconds. For each attempt, report the earliest 600-second window in which that attempt's autosaves alternate between client instances — A, then B, then A again — which indicates a second tab or device rather than a reload. A plain handoff, where one instance stops and another starts, must not be reported. Memory must be bounded by concurrency, not by total events. State your bounds.

Approach

Partition by attempt_id so every event for one attempt lands on one worker, then buffer per attempt until the watermark — the maximum received_at seen minus the 120-second lag bound — passes an event's timestamp. Only then is that event's event-time position final. Anything arriving after its watermark goes to a side output and is counted, not silently dropped, because a rising late rate is the signal that the lag bound is wrong.
Inside one attempt, slide two pointers over the event-time-ordered buffer, advancing the right edge one event at a time and evicting from the left while right.t - left.t > 600. Each event enters and leaves exactly once, so the window maintenance is O(1) amortised per event.
Represent the window as a run-length-compressed deque of (client_instance_id, count) rather than a raw list. Pushing on the right either increments the tail run or appends a new one; evicting on the left decrements the head run and pops it at zero. Runs never merge, because adjacent runs have different ids by construction, so both operations stay O(1).
Read the alternation criterion straight off that structure: three or more runs in the window means A, B, A or three distinct instances, which is interleaving. Exactly two runs is a clean handoff — a reload where one instance stops and the next begins — and must not fire. Counting distinct ids instead of runs is what produces a false positive on every browser refresh.
Bound the memory arithmetically. At 4,000 events per second with one autosave per learner every 25 seconds, about 100,000 attempts are active, and a 600-second window holds roughly 24 events each. That is 2.4 million live events; packed at 48 bytes it is about 115 MB, and several times that with boxed objects. The run-compressed deque is smaller still, since a single-instance window collapses to one entry.
Separate this from the write-path rule it resembles. The attempt rejects an autosave whose seq is below last_autosave_seq, which protects the stored state; it does not detect anything, because two tabs sharing a monotonic counter produce strictly increasing sequences and never trip it. Detection and rejection answer different questions and both are needed.

Follow-up

A learner's connection drops for ten minutes and the client replays its buffered autosaves on reconnect. Does your detector fire, and should it?
You must now also report how many seconds of overlap the two instances had. What changes in the window structure?
The lag bound is violated and events arrive 15 minutes late for one tenant. What does your side output let you do that a dropped-event counter would not?

Model external identities so a second platform is not a merge

Easy · identity resolution · normalisation · unique constraints

You have person(person_id, tenant_id, display_name, email_lower NULL). Learners arrive three ways: a signed launch carrying (issuer, audience, subject, deployment_id), a roster feed carrying a sourcedId, and SSO carrying a nameid. The same human can arrive from two platforms. Email is frequently absent, recycled between graduating and incoming learners, and never guaranteed unique. Give the DDL for the identity tables with every uniqueness constraint, the statement a launch runs to resolve a person, and the rule that decides when two external identities may be bound to one person.

Approach

Split the human from its identifiers: person holds the internal id, external_identity holds one row per issued identifier with a FK to person. The relationship is many-to-one by construction, so a second platform is an INSERT, not a reconciliation, and nothing in the schema tempts you to key on a mutable attribute.
Scope the subject correctly. A subject claim is unique only within its issuer, and a platform that issues pairwise subject identifiers emits a different subject per relying party, so the natural key is (issuer, audience, subject); dropping audience collides the first time the same platform is registered twice. Tenancy comes from the deployment mapping you store, never from anything the client sends.
Give roster identifiers their own namespace: UNIQUE (tenant_id, source_system, sourced_id), either as a discriminated row in the same table or its own table. Cross-namespace binding is where duplicate humans are born, so require a corroborating signal — a sourcedId released inside the launch, or an explicit admin action — and never a name or email string match.
Write the resolve as an atomic claim rather than SELECT-then-INSERT: INSERT INTO external_identity (...) VALUES (...) ON CONFLICT (issuer, audience, subject) DO NOTHING RETURNING person_id, and on zero rows re-SELECT. Allocate the person inside the same transaction so losing the race rolls the speculative person back; two simultaneous first launches must leave exactly one person.
Make merges explicit and reversible-by-audit: person_merge(loser_person_id PRIMARY KEY, winner_person_id, merged_at, actor, reason) plus a redirect on lookup, because stored attempts, scores and outbound publications still carry the loser's id and must keep resolving. A merge is one-way; an unmerge is a restore from the audit, not an UPDATE.
Keep email as a searchable attribute, case-folded, with a non-unique index for support lookup only, and state the rule out loud: a support tool may surface candidates, it may not auto-bind them.

Worked solution (20 min)

Write external_identity with UNIQUE (issuer, audience, subject) and a separate UNIQUE (tenant_id, source_system, sourced_id) for feed identifiers, both FK to person.
Write the ON CONFLICT DO NOTHING RETURNING claim plus the zero-row fallback SELECT, inside one transaction with the person insert.
Run two concurrent first launches with the same tuple and assert one person row and two identical resolutions.
Insert a second identity for the same human from a different issuer, bind it to the existing person, and show that no constraint had to be dropped to allow it.

Expected result: One person per human, one external_identity row per issued identifier, uniqueness enforced on (issuer, audience, subject) and on (tenant_id, source_system, sourced_id), and no unique constraint anywhere on email.

Verify it yourself

Two identities differing only in issuer both insert successfully and resolve to the same person_id.
Two concurrent first launches produce exactly one person row, with the loser returning the winner's person_id rather than an error.
Inserting two people with the same email_lower succeeds, and the resolve path never reads that column.
After a merge, a lookup by the loser's person_id still resolves, through the redirect, to the winner.

Follow-up

A district changes platforms. Every learner arrives with a new issuer and subject but the same sourcedId. What does the first launch after the change do, and how many person rows exist afterwards?
Two identities were bound to one person in error and both have graded attempts. What does the unmerge do, and which stored ids can you not change?
The launch releases no name and no email at all. What does the instructor see in the roster, and where does that display name come from?

Explain why the grading queue index stopped serving its ORDER BY

Medium · indexing · partial index · planner statistics

attempt holds 2 billion rows and 99.7% are state='graded'. The grading queue runs SELECT attempt_id FROM attempt WHERE tenant_id = $1 AND state IN ('submitted','grading') ORDER BY submitted_at LIMIT 100 against index ix_attempt_queue (tenant_id, state, submitted_at). At the bell the query takes 30 seconds and EXPLAIN shows a Sort above a bitmap heap scan, even though the queue is only a few thousand rows. Explain why that index cannot serve the ordering, give the replacement definition, and say how the replacement behaves as rows leave the queue.

Approach

Start from what the index actually orders. Its tuples are sorted by (tenant_id, state, submitted_at), so with two state values the scan yields all 'grading' rows in submitted_at order, then all 'submitted' rows in submitted_at order. The query wants one submitted_at order across both, which that physical order does not provide, so the planner must sort the whole matching set and the LIMIT cannot stop the scan early. With a single state constant the same index satisfies the ordering perfectly — the plan changed because of the IN list, not because of the data.
Replace it with a partial index that moves the predicate out of the key: CREATE INDEX CONCURRENTLY ix_attempt_queue ON attempt (tenant_id, submitted_at) WHERE state IN ('submitted','grading'). Ordering is now satisfied directly, the LIMIT stops after 100 index tuples, and the index covers roughly 0.3% of rows so it stays resident in cache instead of competing with the table.
State the precondition, because it is the part people skip: the planner uses a partial index only when it can prove the query's predicate implies the index predicate. A literal IN list matching the definition proves; state = ANY($1) with the list bound as a parameter does not, so the queue's state set has to be literal in the SQL, not passed in.
Fix the estimate separately from the access path. With five states the planner already has each one in its most-common-values list, so single-column selectivity is fine; what it gets wrong is the dependence between tenant_id and state, because a bell means one tenant owns nearly the whole queue while the planner multiplies the two selectivities as if independent. CREATE STATISTICS ON tenant_id, state FROM attempt corrects it, and you ANALYZE explicitly after a bulk transition rather than waiting for autovacuum, whose analyze threshold of 50 + 0.1 x reltuples needs 200 million changed rows on this table.
Account for the lifecycle. A row leaves the queue by UPDATE state='graded', so its new version is not in the partial index at all while the old index entry remains until vacuum reclaims it. The index therefore churns in proportion to throughput rather than to depth: size it and vacuum it against how many attempts pass through per day, not against how many sit in the queue at once.
Keep the alternative in your pocket for when the partial index is not acceptable: two explicit scans merged, (SELECT ... WHERE state='submitted' ORDER BY submitted_at LIMIT 100) UNION ALL (SELECT ... WHERE state='grading' ORDER BY submitted_at LIMIT 100) ORDER BY submitted_at LIMIT 100, which keeps the existing index and pushes the LIMIT into both branches. Confirm whichever you choose with EXPLAIN on your major version rather than from memory, since array-key handling in b-tree scans has changed between releases.

Follow-up

The queue drains and refills eight times a day. What is the vacuum strategy for this index, and what do you monitor to know it is losing?
A second consumer claims rows with FOR UPDATE SKIP LOCKED. Does the plan survive, and what does skipping do to the ordering guarantee?
Support wants oldest-first fairness across tenants rather than within one. Does the same index serve that query?

Can you walk me through your final year project or your most recent technical project?

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Can you explain the difference between specific programming paradigms (e.g., C vs. C++ vs. JavaScript)?

Medium

Approach

State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Have you ever developed a plugin or integration for existing AI platforms?

Medium

Approach

Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Publish grades outward exactly once under retries and regrades

Medium · outbox · idempotency · ordering · retries

score_publication holds publication_id, tenant_id, attempt_id, target (lti_ags, webhook, sis_export), idempotency_key with UNIQUE (tenant_id, target, idempotency_key), score_given, score_maximum, score_timestamp, state (pending, in_flight, succeeded, failed_retryable, dead), attempt_count and next_attempt_at. The receiving endpoints are third-party: they rate-limit, they time out after having committed, and they can be down for days. Design the relay: how a row is claimed, what happens when the process dies after the HTTPS call but before the row is marked, how the idempotency key is derived, what stops a stale retry from overwriting a regrade, and which rows a claim is allowed to select. next_attempt_at is the only time column on the row; say what it means in each state.

Approach

Write the score row and the publication rows in one transaction. That is the entire point of an outbox: a queue publish after the commit can be lost, and a queue publish before the commit can announce a transaction that then rolls back. There must be no window in which a grade exists with no recorded intent to deliver it.
Claim with a single conditional statement rather than a read then a write, and let next_attempt_at carry two meanings: the backoff time for a row waiting to be sent, and the lease deadline for a row already out. UPDATE score_publication p SET state='in_flight', attempt_count = attempt_count + 1, next_attempt_at = now() + :lease FROM (SELECT publication_id FROM score_publication WHERE state IN ('pending','failed_retryable','in_flight') AND next_attempt_at <= now() ORDER BY next_attempt_at FOR UPDATE SKIP LOCKED LIMIT :n) c WHERE p.publication_id = c.publication_id RETURNING p.*. Both halves are load-bearing. Drop in_flight from the state list and that state becomes a one-way door: a row a process marked in_flight before dying matches no claim again, stalls permanently, and holds its tenant's backlog behind it. Drop the lease write and the row is re-claimable the instant it is claimed, because its next_attempt_at is already in the past, so every publication goes out once per polling worker. SKIP LOCKED is what lets N relay processes share one table without contending on the same head rows; the lease is what makes a crash recoverable. They are separate mechanisms and the design needs both. Index for the predicate with a partial index on next_attempt_at restricted to those three states, or the ORDER BY degrades into a scan over every succeeded row the table has ever accumulated.
Accept that the transport is at-least-once and make the receiver's side idempotent instead of chasing exactly-once on the sender. A process that dies after the call and before the mark leaves the row in_flight with a lease deadline now in the past, and the next claim picks it up and re-sends it. Size :lease above the HTTP timeout plus a margin: a lease shorter than the call reclaims a request that is still running and sends the same grade twice concurrently. That duplicate is harmless only if the key is derived from (attempt_id, score revision) and nothing else. A UUID minted per retry, or a key containing a clock reading, makes every retry a fresh write at the receiver and turns one grade into several. Increment attempt_count at claim time rather than at completion, so a publication that kills its worker on every attempt walks its own counter to the cap and reaches dead, rather than being reclaimed forever.
Treat ordering as a separate failure from duplication. A retry of an old score can land after a regrade's newer score, so score_timestamp must strictly increase per line item and learner; the AGS profile directs receivers to reject a non-increasing timestamp with a conflict, but a sender cannot assume every implementation honours it. Also suppress superseded rows at claim time, so a pending publication that is no longer the current revision for that attempt and target is retired rather than sent.
Classify responses into distinct actions instead of one retry rule. A timeout is an unknown outcome and must be retried with the same key. A conflict on a stale timestamp is success-equivalent and should be marked, not retried. A rate-limit response honours a retry hint when present and otherwise backs off exponentially with jitter under a per-tenant concurrency cap. An expired credential is retryable only after refresh. A permanent rejection, such as a line item that no longer exists, is terminal and moves to dead with last_error_code recorded, because a row retrying forever looks identical to a healthy system on every dashboard.

Worked solution (35 min)

Write the transaction that inserts the score and its publication rows together, and state precisely what is lost if the publish happens outside it.
Write the skip-locked claim, name every state its predicate selects, set the next lease deadline in that same statement, then trace a process that dies immediately after the HTTPS call through to the claim that picks the row up again.
Derive the idempotency key from named columns and show two separate retries producing an identical key.
Build the response classification table: timeout, stale-timestamp conflict, rate limit with and without a hint, credential expiry, line item gone, and server error, with the resulting state and next_attempt_at for each.
Trace a regrade racing a stale retry and show which score the receiver ends up holding.

Expected result: Score and outbox rows committed together; a skip-locked claim whose predicate includes in_flight rows whose lease has expired, writing the next lease deadline in the same statement and sizing it above the HTTP timeout; an idempotency key derived from attempt id and score revision; a strictly increasing ordering token plus suppression of superseded pending rows; and a classified response table where only unknown or transient outcomes retry while everything else reaches a terminal state.

Verify it yourself

Kill the relay at each of three points (after claim, after send, after mark) and confirm the receiver ends with the same grade in all three, and that the killed-after-send row is re-selected by your claim predicate instead of sitting in_flight forever.
Confirm two retries of one publication produce byte-identical idempotency keys.
Confirm a stale retry landing after a regrade cannot leave the receiver holding the older score.

Follow-up

A district's endpoint is down for four days and 200,000 publications accumulate. What happens when it returns, and what protects it from your entire backlog arriving at once?
An instructor regrades one attempt three times inside a minute. How many outbound calls does the receiver see, and which score does it hold at the end?
What does a support engineer see for a stuck tenant, and what exactly does a safe manual replay of a dead row do?

Launch latency spikes for twenty seconds at every period boundary

Easy · cache stampede · thundering herd · connection pool · pre-warming

Every period boundary, launch p99 rises from 180 ms to 4 s for about 20 seconds, then recovers with no deploy and no error spike. Roughly 2,000 sections start together; 50,000 launches arrive in 60 s, with 70% inside the first 15 s. Each launch reads a cached payload keyed (tenant_id, section_id) holding the section roster and its assignments, which costs 140 ms to build. The fleet connection pool is 200. Daily cache hit rate is 96%. Produce an ordered diagnostic checklist, the arithmetic that explains the spike, and the fix.

Approach

Order the checklist so the recovery-without-intervention clue is used early. First, plot cache misses per second against the period boundary, not hit rate over the day. Second, plot connection-pool checkout wait time on the same axis. Third, check whether the slow requests are the misses or the hits. Fourth, only then look at the database, which will show normal per-query time. A self-healing 20-second spike that coincides with pool wait is queueing, not a fault.
Distrust the 96% hit rate: it is a daily average over a workload that is mostly steady-state reads inside a period. It is fully compatible with a near-100% miss rate in the first two seconds of every period, because each period uses a different set of section keys and every one of them is cold.
Do the queueing arithmetic. 70% of 50,000 in 15 s is 2,333 roster builds per second, each holding a connection for 140 ms. By Little's law that needs 2,333 x 0.14 = 327 concurrent connections against a pool of 200, so the pool saturates, checkout queues, and the 400 ms launch budget is gone. Note that only about 2,000 of those 35,000 builds are distinct: 94% of the work is duplicate.
Fix the duplication first, because it is the largest term and costs nothing to run. Add per-key single flight: one loader per key, with the others waiting on the same result, coordinated across processes by a short-TTL lock (SET NX with a TTL safely above the 140 ms build) plus a bounded wait and re-read. That takes demand to 2,000 builds over 15 s = 133 per second, about 19 concurrent connections, under 10% of the pool.
Then remove the cold start entirely for the known case: the bell schedule is in the term calendar, so pre-warm the next period's section keys a few minutes ahead. Reactive autoscaling cannot help here because scale-up latency is tens of seconds to minutes against a step that completes in under 60.
Set the degradation rule in the domain's terms: roster and assignment reads may serve stale while revalidating, because a stale roster is a minor display error and a failed read is a reload, but attempt writes are never shed because a dropped write is lost student work.

Follow-up

Pre-warming 2,000 keys also costs 2,000 builds. How do you spread that so the pre-warm is not itself the spike?
The cache key carries a tenant dimension so entitlement cannot leak. What does that do to your hit rate, and what would you key differently?
A section roster changes mid-period from a roster sync. How does stale-while-revalidate interact with the enrollment-changed invalidation event?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

What is your experience with machine learning frameworks or concepts?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

How do you handle requirements that change as a project progresses?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

What is your experience with ChatGPT or similar AI tools?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?