Interview guides · Aleut Federal

Aleut Federal Software Engineer Interview Questions & Guide 2026

Software engineer prep for Aleut Federal: design exercises weighted toward the constraints of public sector and defense work.

The 60-second brief

A Software Engineer at Aleut Federal serves as a vital technical architect and educator within the defense and aerospace sectors. You are not merely writing code; you are building the digital foundations and instructional frameworks that empower the United States Space Force (USSF). Your work directly influences how guardians are trained to manage high-stakes environments, from Satellite Communications to Orbital Warfare.

If the seat owns a service boundary, scope your preparation toward failure behaviour rather than topology. Retrying over an at-least-once channel produces duplicates by construction, so a retry policy is only as safe as the idempotency key underneath it.

Focus areas

Make every determination replayable from a frozen snapshot
Enforce audit writes at the data-access boundary
Key idempotency on business facts, not requests

Overview

This role demands a unique intersection of deep technical expertise and a passion for mission-critical education. You will be responsible for translating complex systems into scalable curricula or robust architectural solutions that support national security. Whether you are developing training platforms or designing cloud-based infrastructure, your contributions ensure that the next generation of space operators is equipped with the precision, resilience, and advanced knowledge required for modern warfare.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Implementing eligibility rules as ordinary application code that ships on the release train, with thresholds as constants and reference tables edited in place

Rules change on statutory effective dates, are frequently backdated, and are sometimes corrected after decisions have already been issued under them. If the only record of a threshold is the value currently deployed, every historical determination becomes unreproducible the moment it changes, and an appeal cannot be adjudicated because nobody can demonstrate what the rule was. The fix is bitemporal: a rule_version keyed on the period it governs, plus the transaction time at which that text was recorded, so you can ask both 'what applied to this case' and 'what did we believe applied when we decided it'. Reference data the rules read has to be versioned the same way, because an UPDATE to a rate table is a silent rewrite of history.

Computing a statutory due date by adding an interval to a UTC timestamp

The deadline is a civil date in a jurisdiction that observes daylight saving, and a civil day is not always 86,400 seconds; the same arithmetic also ignores weekends and a holiday list that is amended annually and differs by jurisdiction. The result is a due date off by a day near a transition, which reports a case as on time when it is late by law - the direction of the error that produces findings rather than complaints. Store the event as an instant, convert to the case's civil time zone, and walk the business-day calendar as versioned data with its own effective dates, then test the transitions and the holidays explicitly rather than the happy path in July.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Finishing a solution without stating its complexity

Give time and space in the same breath as the code, and define n explicitly when there are two sizes, since n nodes and m edges are not interchangeable. Space is the half that gets skipped: count the auxiliary structures you allocate and the recursion stack at its deepest, not only the answer you hand back.

Questions & practice

Detect bulk-write bursts in a backdatable case event stream

Medium · sliding-window · two-pointer · event-time

One day of case_event holds 1.4 million rows, of which about 210,000 carry actor_type='staff' with a non-NULL actor_id. Each row has occurred_at (the asserted time of the act, which staff routinely backdate by days or weeks) and recorded_at (when the row was written). Flag every actor_id that records more than 200 transitions inside any rolling 10-minute window, so an unannounced bulk script can be told from caseworkers clicking. Choose which clock the window slides on, justify it, and state the time and memory bounds.

Approach

Slide on recorded_at, and separate the two independent reasons. Mechanically, the two-pointer invariant requires the stream key to be non-decreasing so the left pointer only ever advances; rows arrive in recorded_at order, so windowing on occurred_at while consuming them in arrival order breaks that invariant and you would first have to sort the day by occurred_at, an O(n log n) precondition the streaming form does not need. Semantically, even after that sort the occurred_at answer is the wrong one: a script that backdates its writes spreads its asserted times over weeks and fills no window, while the rows it wrote in one second are adjacent in recorded_at. The question is who wrote these rows in a short span, and that is a recorded_at question.
One pass with a deque of (recorded_at, actor_id) and a hash map from actor_id to in-window count. Advance the right pointer, increment; advance the left pointer past anything older than 10 minutes, decrement, and delete a key at zero so the map stays proportional to active actors. Each event is pushed and popped once: O(n) amortised.
Emit on threshold crossing, not on every event above it. A single actor holding 900 events in the window would otherwise produce hundreds of near-identical alerts; latch the actor on crossing and clear the latch only when its count falls back below the threshold.
Memory is the widest window, not the day. Filtering to actor_type='staff' first keeps the deque at the staff rate rather than the 1,800 rows per second the overnight batch reaches; storing 16 bytes per entry rather than the row keeps the worst realistic window inside tens of megabytes. State the bound as a function of peak staff rate times 600 seconds so it survives a change in traffic.
Refine the discriminator beyond count. Rows written by one statement share a recorded_at to the microsecond, while a caseworker's clicks land seconds apart, so the intra-window timestamp dispersion separates a bulk script from a fast human far more cleanly than volume alone, and it does not need a threshold tuned per unit.
Keep occurred_at for the separate report it actually answers: the gap between occurred_at and recorded_at is entry lag, and it spikes predictably at reporting boundaries when staff enter a backlog.

Follow-up

recorded_at defaults to now(), which in this database is transaction start time, so commit order and recorded_at order can differ. What does a reader tailing by recorded_at miss, and how do you bound it?
A legitimate nightly reassignment job runs under a staff actor_id because it was given a service account with a person's role. How does your detector avoid alerting on it, without an allowlist that grows forever?
Run this over an unbounded stream rather than a day. What is the new memory bound and what is the first thing that breaks?

Match a rail return file onto payment instructions in one pass

Easy · hashing · reconciliation · idempotency

You hold 400,000 disbursement_instruction rows for a 90-day window (instruction_id, determination_id, benefit_period_start, amount_minor, rail_reference, status) and a return file of 12,000 rows carrying rail_reference, amount_minor, return_code and returned_at. The rail guarantees rail_reference is unique only within 90 days and recycles values afterwards, and it may resend a return it already sent. Produce three buckets: returns matched to exactly one instruction, returns matching none, and returns matching more than one. One pass over each input. State your time and space bounds.

Approach

Build one hash map from rail_reference to a small list of instruction_ids in a single pass over the instruction rows. Lists, not scalars: recycling means a reference can legitimately collide across the window edge, and you need to observe that rather than let the last writer win. O(n) time, O(n) space for n=400k references.
Stream the return file and probe. Zero hits goes to unmatchable, one hit to matched, more than one to ambiguous. Do not disambiguate an ambiguous hit by amount_minor: a returned payment and a recycled reference can carry the same amount, and guessing turns a reportable exception into a wrong ledger entry.
Deduplicate resends on the business fact, not on the file row. A return is uniquely identified by (instruction_id, return_code, returned_at) once matched; a second identical row applies the same compensating state and must be a no-op, which is what makes replaying a redelivered file safe.
Do not filter the instruction side by rail_reference IS NOT NULL. A NULL reference means the acknowledgement has not been processed, not that nothing was sent, so those rows belong in the map's absence set and drive the separate unacknowledged-but-submitted report.
Memory: 400k entries of a ~20-byte reference key plus an 8-byte id is roughly 11 MB before hash overhead, comfortably in process. If the window ever widens past what fits, range-partition by the reference prefix and run k passes at k times the read cost rather than reaching for a database join per return row.

Worked solution (20 min)

Build an eight-row instruction fixture: six with distinct references, and two sharing reference R because the rail recycled it 91 days apart.
Build a five-row return file: three matching distinct references, one naming reference R, one naming a reference never assigned.
Run the two passes and record which bucket each return lands in.
Append a byte-identical copy of one already-matched return row and re-run, confirming the applied state is unchanged.

Expected result: Three returns matched, one ambiguous (reference R, two candidate instructions, no amount-based tiebreak attempted), one unmatchable. The duplicated row changes no instruction state and increments a duplicate-return counter.

Verify it yourself

The three bucket sizes sum to the return file row count, with no row counted twice.
Shuffling the return file changes nothing in the bucket membership.
Removing the instruction rows whose rail_reference is NULL does not change any bucket, which proves those rows were never being used as match candidates in the first place.

Follow-up

The rail sends a partial return for less than amount_minor. What does your matcher do, and what does the instruction row become?
A return arrives for an instruction that has already been reversed by a compensating row. How do you tell a duplicate return from a genuine second event?
You must run this continuously rather than per file. What bounds the map now, and what is the eviction rule?

Rebuild an award timeline as of a past decision date

Medium · intervals · bitemporal · sweep-line

One case has up to 400 determination rows across all reopen cycles: determination_id, reopen_cycle, supersedes_determination_id, decided_at TIMESTAMPTZ, effective_from DATE, effective_to DATE (NULL meaning open-ended, both endpoints inclusive), award_amount_minor. Given a set of queries (benefit_date D, as_of decision time T), return the award in force for D according to what had been decided by T, plus any benefit dates inside the covered span that no determination covers. Later-decided wins on overlap. Give the complexity for building the answer and for each query.

Approach

Convert every interval to half-open [effective_from, effective_to + 1 day) at the boundary of the system, once. Keeping inclusive ends in the algorithm produces an off-by-one at every adjacency, and the error is a single uncovered or doubly-covered day that a reviewer reads as a gap in benefit.
For a given T, filter to determinations with decided_at <= T. Do not use is_current: that column answers what is in force now, and the whole point of the as-of query is that a determination decided after T must be invisible even though it exists today.
Paint the surviving intervals onto a timeline in ascending decided_at order so later decisions overwrite earlier ones where they overlap. Equivalently, sweep the sorted endpoints with a max-heap keyed on decided_at and lazily discard heap entries whose interval has already ended. O(n log n) build.
Answer each query by binary search over the painted, non-overlapping interval list: O(log n). Gaps fall out of the same structure as the elementary intervals with no covering determination between the minimum effective_from and the maximum effective_to of the surviving set. Note what that definition implies: deleting one determination only produces a gap where nothing else still covers those dates, so an overlapping earlier award absorbs the deletion silently and correctly.
With n at most 400, a quadratic paint over elementary intervals is about 160k comparisons and is genuinely fine; say so rather than reaching for the heap by reflex. The heap earns its place when the same routine runs across a program-wide cohort rather than one case.
If queries carry many distinct T values, the painted timeline is per-T and rebuilding it per query is O(Qn log n). Sort queries by T and advance one sweep through decided_at order, materialising each version as you pass it, or accept the rebuild when Q is small.

Follow-up

A determination is decided with an occurred-earlier effective_from that reaches back behind an already-settled disbursement. What does the timeline say, and what does that imply for the payment side?
Two determinations in different reopen cycles share a decided_at to the microsecond. What is your tiebreak and why must it be total?
How do you answer the same question for 200,000 cases in a nightly report without rebuilding 200,000 heaps?

Add a uniqueness rule to a live payments table with duplicates

Hard · online migration · concurrent index build · lock queue · compensating rows

disbursement_instruction(instruction_id BIGSERIAL PK, determination_id BIGINT, benefit_period_start DATE, amount_minor BIGINT, status TEXT, reversal_of_instruction_id BIGINT NULL) has 40M rows and takes writes continuously. It was built without UNIQUE (determination_id, benefit_period_start); a retried batch created 11,400 duplicate pairs, and 3,900 of those duplicates reached settled. Money that moved is never deleted and never voided — it is unwound with a compensating reversal row carrying reversal_of_instruction_id, which repeats the same business key. status values include voided. Produce the migration: the query that finds the duplicates, what you do with the ones that settled, the DDL that adds the constraint without blocking writers, and the deploy ordering. Name the lock each statement takes and what happens when it cannot be acquired.

Approach

Find the duplicates with a window rather than a self-join: SELECT instruction_id, status, ROW_NUMBER() OVER (PARTITION BY determination_id, benefit_period_start ORDER BY instruction_id) AS n FROM disbursement_instruction and inspect every row with n > 1 together with its status, because the remediation differs entirely by status. The n = 1 row is the one that stays live; the rest are what you unwind.
Do not make the data fit the constraint by deleting rows. A duplicate that reached settled moved real money and the rail's own ledger will not forget it; deleting the row makes your records disagree with the counterparty. The 7,500 unsent duplicates may be set to voided. The 3,900 settled ones get compensating reversal rows referencing them via reversal_of_instruction_id — and stay.
That remediation is what rules out the obvious predicate. After it, each settled duplicate key still holds three non-voided rows: the original, the settled duplicate and its reversal. CREATE UNIQUE INDEX CONCURRENTLY ... WHERE status <> 'voided' therefore aborts with 23505 on the first of the 3,900. The fact the predicate needs — 'this row has been unwound' — cannot be a subquery either: a partial index predicate may reference only columns of the row being indexed and must be IMMUTABLE. So the fact has to be written onto the row.
Add it as a nullable column with no default, which is a catalog change and not a table rewrite (PostgreSQL 11 extended that to non-volatile defaults as well): ALTER TABLE disbursement_instruction ADD COLUMN reversed_by_instruction_id BIGINT. It still takes ACCESS EXCLUSIVE for the catalog write, so it carries the same lock_timeout guard as any other ACCESS EXCLUSIVE step; add the self-referencing foreign key separately as NOT VALID and then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE.
The constraint is then a partial unique index over the rows that are live originals: CREATE UNIQUE INDEX CONCURRENTLY uq_instr_business_key ON disbursement_instruction (determination_id, benefit_period_start) WHERE status <> 'voided' AND reversal_of_instruction_id IS NULL AND reversed_by_instruction_id IS NULL. Each conjunct does one job — never sent, is itself an unwind, has been unwound. Accept the consequences: a partial index cannot back an ALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING INDEX, so there is no named constraint and no foreign key can reference this key, and INSERT ... ON CONFLICT must repeat all three conjuncts in its inference clause.
Prove the data fits before you build. Run SELECT determination_id, benefit_period_start FROM disbursement_instruction WHERE <the index predicate> GROUP BY 1,2 HAVING count(*) > 1 and require zero rows. A unique build that hits a violation is not free: CREATE INDEX CONCURRENTLY detects it in the second pass, aborts after scanning 40M rows twice, and leaves an index with indisvalid = false that must be dropped with DROP INDEX CONCURRENTLY before retrying.
Name the locks. CREATE INDEX CONCURRENTLY takes SHARE UPDATE EXCLUSIVE, so readers and writers continue; it makes two passes plus waits for concurrent transactions and cannot run inside a transaction block. A plain CREATE INDEX or an ALTER TABLE variant takes ACCESS EXCLUSIVE, which queues behind any running query and blocks every reader that arrives behind it in the queue — a five-millisecond statement becomes an outage while one long report finishes. Guard every ACCESS EXCLUSIVE step with SET lock_timeout = '2s' and retry in a loop, so a failed acquisition ends as a failed migration step instead of a stalled table.
Order the deploy so the window is never open: add the column; ship the application code that derives the idempotency key from (determination_id, benefit_period_start), treats 23505 as a converged retry, and stamps reversed_by_instruction_id whenever it writes a reversal; remediate the existing duplicates in chunks; re-run the residual-duplicate check; build the index concurrently; verify indisvalid in pg_index. Building the index before the code can handle a violation converts every legitimate retry into a user-visible error.

Follow-up

The concurrent build fails at 90% because of a deadlock with a nightly batch. What is on disk now, and what exactly do you run before retrying?
A duplicate settled and then the original was returned by the rail. What does the compensating row look like, and what is the account's net position?
Once the partial index exists, how does the batch writer's ON CONFLICT inference clause have to be written against a three-conjunct predicate, and what happens if the predicate is omitted?

Diagnose why the overdue-case queue scans four million rows

Medium · index selectivity · query planning · partial index · sargability

case holds 4M rows, 2.6M of them closed, and idx_case_status ON "case"(status) is the only relevant index. The queue query is SELECT case_id, sla_due_on FROM "case" WHERE program_code = $1 AND status IN ('submitted','evidence_pending','in_review') AND sla_due_on <= CURRENT_DATE + 3 ORDER BY sla_due_on LIMIT 50. EXPLAIN (ANALYZE, BUFFERS) shows a sequential scan followed by a top-N sort, 900 ms. Explain why the existing index is not chosen, propose the index you would build, and state what you check in the new plan.

Approach

Start with selectivity, not with the index definition. status has seven values and the three open ones cover roughly 1.4M of 4M rows; an index whose leading column selects 35% of the table costs more than a sequential scan once random heap fetches are priced in, so the planner is making the correct decision given what it has.
Read the numbers, not the shape: compare each node's estimated rows to actual rows in EXPLAIN (ANALYZE, BUFFERS). A tenfold or worse gap on the conjunction of program_code and status points at correlation the planner assumes away, which CREATE STATISTICS case_prog_status ON program_code, status FROM "case" plus ANALYZE can correct.
Build the index the query actually wants: CREATE INDEX CONCURRENTLY idx_case_open_due ON "case" (program_code, sla_due_on) WHERE status IN ('submitted','evidence_pending','in_review'). Equality column first so it fixes a prefix, range and sort column second so the index scan delivers rows already ordered by sla_due_on and the LIMIT 50 stops after 50 heap fetches instead of sorting a large intermediate. The partial predicate drops the 2.6M closed rows out of the index entirely, which is most of its size.
Spell the query's predicate exactly as the index predicate is spelled. The planner will only use a partial index when it can prove the query implies the index predicate, and relying on it to derive that from a differently phrased condition is how a supposedly indexed query quietly regresses after a refactor.
Fold the row-level security qual into the index. Every query against this table carries the jurisdiction policy predicate; if jurisdiction_code is not in the index, the policy is a filter applied to rows already fetched from the heap, so the LIMIT reads more rows than it returns. Either add it as a leading column or make it part of the partial predicate, depending on whether one jurisdiction dominates.
Verify with the plan rather than the wall clock: you want an Index Scan (not Bitmap Heap Scan, which loses the ordering and forces the sort back), no Sort node above it, and buffer counts in the tens rather than the tens of thousands.

Worked solution (30 min)

Generate 4M rows with a realistic status skew and run the query under EXPLAIN (ANALYZE, BUFFERS); record the node types, the estimated-versus-actual row counts and the shared-block reads.
Add the single-column index on sla_due_on first and re-plan, to see the planner either ignore it or choose a bitmap scan that still sorts.
Build the partial composite index concurrently and re-plan; confirm the Sort node disappears and the scan stops early under the LIMIT.
Rephrase the query's status predicate (for example as three ORed equalities) and re-plan to see whether the partial index is still matched.
Enable the row-level security policy and re-plan to see the policy qual appear, then fold jurisdiction_code into the index and compare buffer counts.

Expected result: The partial composite index turns a sequential scan plus top-N sort over 4M rows into an index scan returning 50 rows with no sort, with shared-block reads dropping from tens of thousands to roughly the index depth plus 50 heap pages, and latency from ~900 ms to low single-digit milliseconds.

Verify it yourself

EXPLAIN shows Index Scan with no Sort node above it — a Bitmap Heap Scan means the ordering was lost and the LIMIT is no longer cheap.
pg_relation_size on the partial index is materially smaller than on the equivalent full index, confirming the predicate excluded closed rows.
Estimated rows at the scan node are within an order of magnitude of actual after CREATE STATISTICS and ANALYZE.
Re-running with the status predicate spelled differently either still matches the index or does not — and you can state which, rather than assuming.

Follow-up

A supervisor adds ORDER BY sla_due_on, case_id for stable pagination. Does your index still avoid the sort, and if not, what changes?
The same index is now the largest on the table and the write path does 1,500 transitions per second. What does maintaining it cost on UPDATE, and does a HOT update avoid it?
How does the plan change when $1 is a program holding 90% of open cases, and what does that do to a generic plan cached by the prepared statement?

Can you explain how you would architect a scalable cloud solution for a high-availability environment?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How do you break down complex technical topics for an audience with varying levels of expertise?

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How do you ensure your technical solutions remain robust under strict operational constraints?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Size and partition an append-only access log

Medium · partitioning · append-only · write amplification · retention

audit_access_log takes one row per access to a record carrying personal data, from web UI, internal API, batch jobs, report extracts, and support tooling: roughly 35 million rows a day, peaking at 3,000 writes a second, retained six years because audit retention outlives the records it describes. Rows are never updated. Investigators query by subject_id over a date range and by actor_id over a date range; nobody queries by request_id at scale. Choose the partitioning key, the indexes, the write path, and the destruction mechanism, and state what each costs.

Approach

Do the arithmetic first, because it eliminates most designs. At roughly 200 bytes a row including the heap tuple header, 35 million rows a day is about 7 GB a day before indexes, 12.8 billion rows a year, and near 77 billion rows and 15 TB of heap over six years. No single table with a btree on subject_id survives that, so tiering is part of the original design rather than a later migration.
Partition by range on occurred_at, daily or weekly. It is the only column that turns retention into DETACH plus DROP instead of a DELETE that leaves 35 million dead tuples a day for vacuum to chase. Note the constraint this imposes: in Postgres declarative partitioning a unique index on the partitioned table must include every partition key column, so audit_id alone cannot be the global primary key. Use (occurred_at, audit_id), or accept per-partition uniqueness and say so.
Index only the two queries that exist: (subject_type, subject_id, occurred_at) and (actor_id, occurred_at), and keep them on hot partitions. On an append-only table whose physical order correlates with occurred_at, a BRIN index on that column costs kilobytes where a btree costs gigabytes, but it prunes ranges only and is no substitute for the subject lookup, which is a point query.
Specify the write path as a bounded queue, not a fire-and-forget call. Batch 100 to 500 rows per statement, bound the queue, and decide explicitly what happens when it fills: blocking the read is fail-closed and couples read availability to the audit store, while an outbox keeps reads available and needs its own liveness alarm, because a stalled publisher looks exactly like a quiet afternoon.
Age detached partitions into an immutable, content-hashed columnar archive with the same investigator queries available, and keep retained_until on the partition. Audit retention runs on its own schedule and routinely outlives the case it describes, so it cannot inherit the business record's clock.

Worked solution (30 min)

Compute daily and six-year row counts and raw heap bytes from 35 million rows a day, and name the point at which a single unpartitioned table stops being viable.
Write CREATE TABLE ... PARTITION BY RANGE (occurred_at) with the two investigator indexes, and state why the primary key must include occurred_at.
Write the retention step as DETACH PARTITION, archive, then DROP, and compare its cost against DELETE in dead tuples and vacuum work.
Specify the write path: batch size, queue bound, the behaviour when the queue fills, and the alarm that distinguishes a stalled publisher from low traffic.

Expected result: About 12.8 billion rows a year and roughly 77 billion over six years at around 15 TB of heap; daily or weekly range partitions on occurred_at; a composite primary key including the partition key; (subject_type, subject_id, occurred_at) and (actor_id, occurred_at) on hot partitions with BRIN for range pruning; retention by DETACH and DROP; and a stated availability decision on the audit write.

Verify it yourself

A subject query over four years touches only the partitions in range, verifiable in the plan.
Dropping a partition frees space immediately with no vacuum backlog.
A batch job reading 2 million subject records produces 2 million audit rows, and you can state the investigator question that an aggregate row cannot answer.

Follow-up

An investigator asks for every access to one constituent across four years. What does that query touch, and what do you do to make it finish?
A report extract reads 2 million subject records in one job. How many audit rows, and why would an aggregate row not satisfy the obligation?
The outbox publisher stalls for an hour. How do you find out, and what exactly was lost?

Bulk replay is OOM-killed at hour six, intake never leaks

Hard · memory leak · gc · profiling

After a rule correction, a bulk replay re-evaluates 4 million historical determinations at about 8,000 per minute. Resident memory climbs monotonically, GC time goes from 2% to 35%, and the worker is OOM-killed near hour six; a restart resumes and dies further along. The same build serves intake at 3,000 evaluations per minute for weeks with flat memory. The evaluator memoises results in a map keyed by (rule_version_id, evidence_snapshot_hash), whose value holds the outcome and the clause-level trace. Deliver an ordered checklist that separates retention from allocator behaviour, and the fix.

Approach

Distinguish a leak from a high-water mark before profiling anything: compare the live set after a forced full collection at hour one, two and three, not resident memory. A live set that grows monotonically is retention; resident memory growing while the live set is flat is allocator or arena behaviour, and the two have entirely different fixes.
Take two heap snapshots an hour apart and diff by retained size, then walk the dominator tree to name the retainer. Ranking by allocation count is the common wrong move — the heaviest allocator is usually short-lived garbage, and the retainer is often a modest number of objects each holding a large graph.
Reason about workload shape before reading another line of code, because it is one counter: memoisation keyed on an evidence hash has a useful hit rate at intake, where a few thousand snapshots recur across a day, and a hit rate near zero in replay, where all 4 million hashes are distinct by construction. Measuring the hit rate converts this from an unexplained leak into an unbounded cache that is buying nothing.
Confirm the per-entry cost so the arithmetic matches the timeline: the key is a 32-byte hash plus an integer, but a value holding the clause-level trace and a reference to the evidence snapshot retains kilobytes. At 8,000 per minute, hour six is about 2.9 million entries; at roughly 2 kB retained each that is near 6 GB, which is where the ceiling is.
Fix so both modes behave the same: bound the cache with a size-capped eviction policy rather than switching it off by a mode flag, since a flag only moves the cliff to whoever forgets to set it. Then also disable memoisation on the replay path where the measured hit rate does not pay for the retention, and stream results to the sink instead of accumulating them in a result list.
Re-run and verify with the same instrument that diagnosed it: post-collection live set flat across two hours, GC time back near 2%, and throughput per minute unchanged or better because the allocator is no longer being pressured.

Follow-up

Replay must not create a second determination for a case. What uniqueness key makes a re-run converge onto the existing row?
Restarting resumes and dies further along. How would you make the replay checkpointed so hour six does not cost hours one through five?
Intake and replay share a build. What keeps a future optimisation tuned for intake from re-creating this?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

Describe your experience with modern software development lifecycles in a secure or classified environment.

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Why are you interested in supporting the mission?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

Own an outage where actions took effect without notices

Medium · incident response · statutory deadlines · event ordering · rollback

An overnight job moved 4,200 cases into a suspended status whose statutory notice period had never started: the render step failed, the print vendor's daily cutoff passed, and the effective-date flip ran on a timer instead of on the dispatch event. Describe an incident of this shape that you owned. State how you detected it, how you bounded it while the clock kept running, what you could and could not roll back, what the affected people were told, and the durable change that made the flip depend on dispatch.

Approach

Open with blast radius in the units the obligation is written in, not in technical ones: how many cases flipped, how many of those had an action that already took legal effect, how many had money stop. Then state the detection gap in minutes between the timer firing and anyone knowing, and say honestly if the only signal was a constituent phone call.
Split what is reversible from what is not before describing any fix. Restoring the prior status is reversible, but it is done by appending a corrective transition to case_event rather than by updating or deleting the erroneous one, because that log is append-only and the wrong transition is itself part of the record now. A suppressed payment is recoverable by issuing the instruction; an action that has already had legal effect on someone is not undone by a status column.
Name the ordering you violated and the ordering you restored: render, confirm dispatch, then flip, with the flip consuming the dispatch event. A timer fires whether or not the letter exists, so the timer is the defect and the render failure is only what exposed it.
Re-notice with a fresh statutory clock rather than back-dating the period to when the letter should have gone out. Back-dating is the tempting shortcut and is precisely what an appeal will find, since the dispatch record per channel contradicts it.
Close with a durable instrument, not a resolution: an alarm on the count of cases sitting in a gated state with no matching dispatch row beyond a threshold age, and the vendor cutoff tracked as its own objective, because missing it shifts a statutory date by a full business day.

Follow-up

The render queue is 40 minutes from the cutoff and behind. Do you cut a partial batch or hold everything to the next day, and who decides?
A constituent appeals citing the notice date. What must the data model produce to reconstruct what actually happened, and in what order?
How would this have been caught without a complaint from someone affected?