Interview guides · KGS
KGS Software Engineer Interview Questions & Guide 2026
System design practice for KGS software engineer candidates: reach a data model and an API contract, not a box diagram.
The 60-second brief
As a Software Engineer within the Amazon Leo for Government (ALG) team, you are at the forefront of space-based technology. Your work directly contributes to the development of a low Earth orbit (LEO) satellite constellation designed to provide secure, high-speed, and low-latency connectivity to government, defense, and international partners. You are not just writing code; you are building the software infrastructure that enables mission-critical communication, satellite coordination, and space sustainability.
The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.
Focus areas
- Version rules by statutory effective date, not release
- Treat accessibility conformance as a functional requirement
- Key idempotency on business facts, not requests
Overview
This role is unique because you are operating with minimal legacy constraints, allowing for the design and deployment of modern, high-assurance, and fault-tolerant systems from the ground up. You will frequently work at the intersection of embedded systems, networking, and high-performance computing, collaborating with hardware, systems, and product teams to ensure seamless integration between software and the physical satellite or ground terminal architecture. If you thrive in environments where you can influence system architecture and solve problems that impact global connectivity, this is a critical position within.
The KGS interview process
KGS candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.
Initial Screening
The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.
What to demonstrate
- Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
- Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
- Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year
How to prepare
- Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
- Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
- Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter
Technical Deep-Dive Interviews
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Coding Assessments
Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.
What to demonstrate
- Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
- Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
- Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
- Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply
How to prepare
- For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
- For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
- Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last
System Design Discussions
An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.
What to demonstrate
- Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
- Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
- Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating
How to prepare
- Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
- Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
- Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes
Behavioral Interviews
This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.
What to demonstrate
- Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
- Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
- Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage
How to prepare
- For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
- Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
- Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend
Final Onsite/Virtual Loop
A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.
What to demonstrate
- Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
- Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
- Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
- Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing
How to prepare
- Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
- Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
- Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one
Where candidates lose points
Running an EBCDIC-to-ASCII text conversion across a whole fixed-width record coming from a legacy system of record
Those records mix character fields with binary and packed-decimal (COMP-3) numeric fields, where two digits are packed per byte and the sign occupies the low-order nibble of the final byte. A character transcode rewrites those bytes, so an amount does not fail to parse - it parses into a different, entirely plausible number, and the corruption is discovered in reconciliation weeks later or not at all. Parse by field using the record layout: transcode only the character fields, unpack the numeric fields explicitly, and add a checksum or control-total record to the file contract so a mangled batch is rejected rather than posted.
Leaving accessibility conformance to a review at the end of the project
The obligation is usually statutory and expressed against WCAG success criteria at AA, and those criteria cut straight through component architecture rather than sitting on top of it. Custom select menus, modals and date pickers have to manage focus and expose correct roles and names; 2.4.11 Focus Not Obscured (AA) constrains sticky headers and cookie bars; 2.5.8 Target Size (Minimum) (AA) requires a 24 by 24 CSS pixel target unless a listed exception applies; 3.3.8 Accessible Authentication (Minimum) (AA) rules out a login step that depends on a cognitive function test with no alternative; and 2.2.1 Timing Adjustable (Level A) collides directly with the short session timeout a security review asked for, since the user must be able to extend it. Retrofitting these means rewriting the component layer and the session design, so they belong in the component contract and in the definition of done from the first sprint.
Sharing mutable state with no stated owner
Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.
Starting work without saying what you are about to spend time on
State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.
Questions & practice
Detect bulk-write bursts in a backdatable case event stream
Medium · sliding-window · two-pointer · event-time
One day of case_event holds 1.4 million rows, of which about 210,000 carry actor_type='staff' with a non-NULL actor_id. Each row has occurred_at (the asserted time of the act, which staff routinely backdate by days or weeks) and recorded_at (when the row was written). Flag every actor_id that records more than 200 transitions inside any rolling 10-minute window, so an unannounced bulk script can be told from caseworkers clicking. Choose which clock the window slides on, justify it, and state the time and memory bounds.
Approach
- Slide on recorded_at, and separate the two independent reasons. Mechanically, the two-pointer invariant requires the stream key to be non-decreasing so the left pointer only ever advances; rows arrive in recorded_at order, so windowing on occurred_at while consuming them in arrival order breaks that invariant and you would first have to sort the day by occurred_at, an O(n log n) precondition the streaming form does not need. Semantically, even after that sort the occurred_at answer is the wrong one: a script that backdates its writes spreads its asserted times over weeks and fills no window, while the rows it wrote in one second are adjacent in recorded_at. The question is who wrote these rows in a short span, and that is a recorded_at question.
- One pass with a deque of (recorded_at, actor_id) and a hash map from actor_id to in-window count. Advance the right pointer, increment; advance the left pointer past anything older than 10 minutes, decrement, and delete a key at zero so the map stays proportional to active actors. Each event is pushed and popped once: O(n) amortised.
- Emit on threshold crossing, not on every event above it. A single actor holding 900 events in the window would otherwise produce hundreds of near-identical alerts; latch the actor on crossing and clear the latch only when its count falls back below the threshold.
- Memory is the widest window, not the day. Filtering to actor_type='staff' first keeps the deque at the staff rate rather than the 1,800 rows per second the overnight batch reaches; storing 16 bytes per entry rather than the row keeps the worst realistic window inside tens of megabytes. State the bound as a function of peak staff rate times 600 seconds so it survives a change in traffic.
- Refine the discriminator beyond count. Rows written by one statement share a recorded_at to the microsecond, while a caseworker's clicks land seconds apart, so the intra-window timestamp dispersion separates a bulk script from a fast human far more cleanly than volume alone, and it does not need a threshold tuned per unit.
- Keep occurred_at for the separate report it actually answers: the gap between occurred_at and recorded_at is entry lag, and it spikes predictably at reporting boundaries when staff enter a backlog.
Follow-up
- recorded_at defaults to now(), which in this database is transaction start time, so commit order and recorded_at order can differ. What does a reader tailing by recorded_at miss, and how do you bound it?
- A legitimate nightly reassignment job runs under a staff actor_id because it was given a service account with a person's role. How does your detector avoid alerting on it, without an allowlist that grows forever?
- Run this over an unbounded stream rather than a day. What is the new memory bound and what is the first thing that breaks?
Surface the fifty most SLA-at-risk cases per unit
Medium · top-k · heap · index-selectivity
A queue view must return the 50 cases closest to breaching their service standard for one assigned_unit_id. Candidate rows come from case: case_id, jurisdiction_code, sla_due_on DATE, submitted_at, status, assigned_unit_id. Four million cases are open across 200 jurisdictions. Risk orders by business days remaining to sla_due_on ascending, then submitted_at ascending, then case_id ascending; already-breached cases sort first and must not clamp at zero. Business days come from a per-jurisdiction holiday calendar. Give the algorithm, the complexity, and where the work must happen.
Approach
- Keep a bounded max-heap of size k=50 ordered by the full comparison key and stream the candidates: push, and when the heap exceeds k pop the maximum. What survives is the k smallest. O(n log k) time and O(k) space, against O(n log n) time and O(n) space for sorting the unit's whole backlog.
- Narrow before you compute. Business days remaining is an expression over a calendar, so no index can order by it, and a naive ORDER BY on that expression sorts every candidate row. Within one jurisdiction the business-day remainder is monotone in sla_due_on, so an index range on (assigned_unit_id, sla_due_on) with a horizon bound gives a correct prefix cheaply; compute the calendar arithmetic only on that narrowed set.
- Do not carry the monotonicity across jurisdictions: two jurisdictions with different holiday lists order differently for the same sla_due_on. Run the k-selection per jurisdiction and merge the per-jurisdiction heaps, or pull the narrowed union and heap it in one pass, but do not assume one global sla_due_on order is the risk order.
- Make the key a total order. Ties on the first two components are common because overnight batch intake stamps thousands of submitted_at values inside the same second, and without case_id as the final component two replicas computing the same list return different pages. Determinism here is a correctness property of the page, not a nicety.
- Pin the as-of instant for the whole computation and return it with the list. Business days remaining changes at a civil midnight per jurisdiction, so a list computed across that boundary would otherwise mix two answers and be unreproducible when someone asks why a case ranked where it did.
- The heap ranks; it does not assign. Handing a case to a worker is a conditional UPDATE guarded on row_version, and a zero-row result means another worker took it, so the caller drops to the next candidate rather than retrying the same row.
Worked solution (25 min)
- Generate 100,000 synthetic open cases across three jurisdictions with different holiday lists, including 40 already past sla_due_on and 300 sharing one submitted_at value to the second.
- Implement the size-50 max-heap pass and record comparison count and peak heap size.
- Implement the full sort as a control and diff the top 50 against the heap's output.
- Drop case_id from the comparison key, run the heap pass twice with the input shuffled between runs, and diff the two outputs.
Expected result: Heap and full sort return an identical ordered 50. The heap performs on the order of n log k comparisons against n log n for the sort, roughly a fourfold reduction at these sizes, and its resident set stays at 50 entries. Without case_id in the key the two shuffled runs disagree on the membership of the last few positions.
Verify it yourself
- All 40 breached cases appear before any unbreached case, with their remaining-days values negative rather than floored at zero.
- Restricting the input to a single jurisdiction makes an sla_due_on ordering agree exactly with the business-day ordering; mixing jurisdictions breaks that agreement, which is the evidence for the per-jurisdiction narrowing.
- The reported as-of instant, replayed against the same fixture, reproduces the list byte for byte.
Follow-up
- Two workers open the queue in the same second and both see case X first. What happens, and whose problem is it?
- A jurisdiction's legislature adds a holiday with retroactive effect. Which cached sla_due_on values are now wrong, and how do you find them?
- The product asks for page two of this list. What does that cost with a size-k heap, and what would you build instead?
Parse a fixed-width record with packed-decimal amount fields
Easy · parsing · encoding · legacy-integration
A nightly file from a system of record arrives as fixed-length 180-byte records in EBCDIC code page 037. Per the copybook, bytes 1-40 are a character name field, bytes 41-48 a character date, bytes 49-52 an amount declared PIC S9(5)V99 COMP-3, and the rest character. The trailer record carries a record count and the sum of every amount field. Write the parser and state, in minor currency units, what a whole-record EBCDIC-to-ASCII text conversion does to a packed amount before it ever reaches your validation.
Approach
- Parse by field against the copybook offsets, never by transcoding the record. Slice bytes 1-40 and 41-48 and decode those with cp037; slice bytes 49-52 and hand them to a packed-decimal unpacker that never sees a codec.
- Unpack COMP-3 explicitly: two BCD digits per byte, and in the final byte the high nibble is the last digit while the low nibble is the sign. Nibble C and F are positive and D is negative under the common convention, so assert the sign nibble rather than defaulting an unrecognised one to positive. S9(5)V99 is seven digits plus a sign nibble, which is eight nibbles, which is exactly the four declared bytes.
- Return the value as an integer count of minor units. V99 is an implied decimal position with nothing stored for it, so the seven unpacked digits already are the minor-unit amount; converting to a float to reinsert a decimal point reintroduces the representation error the packed format exists to avoid.
- Reject the batch on the trailer, not per record. Accumulate the record count and the integer sum of every amount, compare to the trailer, and fail the whole file rather than post a partial batch. A corrupted amount that still parses is invisible to per-record validation and visible only to the control total.
- Complexity is O(total bytes) with O(1) working memory beyond one record buffer and two accumulators; the file is read as a byte stream, so nothing here needs the file resident.
Follow-up
- The copybook gains an OCCURS DEPENDING ON clause, so record length now varies with a count field. How does your offset arithmetic change?
- A record contains a sign nibble of 0xA. Is that valid, and what does your parser do with it?
- The sender starts padding numeric character fields with EBCDIC zeros in a field you decode as text. What breaks and where do you catch it?
Add a uniqueness rule to a live payments table with duplicates
Hard · online migration · concurrent index build · lock queue · compensating rows
disbursement_instruction(instruction_id BIGSERIAL PK, determination_id BIGINT, benefit_period_start DATE, amount_minor BIGINT, status TEXT, reversal_of_instruction_id BIGINT NULL) has 40M rows and takes writes continuously. It was built without UNIQUE (determination_id, benefit_period_start); a retried batch created 11,400 duplicate pairs, and 3,900 of those duplicates reached settled. Money that moved is never deleted and never voided — it is unwound with a compensating reversal row carrying reversal_of_instruction_id, which repeats the same business key. status values include voided. Produce the migration: the query that finds the duplicates, what you do with the ones that settled, the DDL that adds the constraint without blocking writers, and the deploy ordering. Name the lock each statement takes and what happens when it cannot be acquired.
Approach
- Find the duplicates with a window rather than a self-join:
SELECT instruction_id, status, ROW_NUMBER() OVER (PARTITION BY determination_id, benefit_period_start ORDER BY instruction_id) AS n FROM disbursement_instructionand inspect every row withn > 1together with its status, because the remediation differs entirely by status. Then = 1row is the one that stays live; the rest are what you unwind. - Do not make the data fit the constraint by deleting rows. A duplicate that reached
settledmoved real money and the rail's own ledger will not forget it; deleting the row makes your records disagree with the counterparty. The 7,500 unsent duplicates may be set tovoided. The 3,900 settled ones get compensating reversal rows referencing them viareversal_of_instruction_id— and stay. - That remediation is what rules out the obvious predicate. After it, each settled duplicate key still holds three non-voided rows: the original, the settled duplicate and its reversal.
CREATE UNIQUE INDEX CONCURRENTLY ... WHERE status <> 'voided'therefore aborts with 23505 on the first of the 3,900. The fact the predicate needs — 'this row has been unwound' — cannot be a subquery either: a partial index predicate may reference only columns of the row being indexed and must be IMMUTABLE. So the fact has to be written onto the row. - Add it as a nullable column with no default, which is a catalog change and not a table rewrite (PostgreSQL 11 extended that to non-volatile defaults as well):
ALTER TABLE disbursement_instruction ADD COLUMN reversed_by_instruction_id BIGINT. It still takes ACCESS EXCLUSIVE for the catalog write, so it carries the samelock_timeoutguard as any other ACCESS EXCLUSIVE step; add the self-referencing foreign key separately asNOT VALIDand thenVALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE. - The constraint is then a partial unique index over the rows that are live originals:
CREATE UNIQUE INDEX CONCURRENTLY uq_instr_business_key ON disbursement_instruction (determination_id, benefit_period_start) WHERE status <> 'voided' AND reversal_of_instruction_id IS NULL AND reversed_by_instruction_id IS NULL. Each conjunct does one job — never sent, is itself an unwind, has been unwound. Accept the consequences: a partial index cannot back anALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING INDEX, so there is no named constraint and no foreign key can reference this key, andINSERT ... ON CONFLICTmust repeat all three conjuncts in its inference clause. - Prove the data fits before you build. Run
SELECT determination_id, benefit_period_start FROM disbursement_instruction WHERE <the index predicate> GROUP BY 1,2 HAVING count(*) > 1and require zero rows. A unique build that hits a violation is not free:CREATE INDEX CONCURRENTLYdetects it in the second pass, aborts after scanning 40M rows twice, and leaves an index withindisvalid = falsethat must be dropped withDROP INDEX CONCURRENTLYbefore retrying. - Name the locks.
CREATE INDEX CONCURRENTLYtakes SHARE UPDATE EXCLUSIVE, so readers and writers continue; it makes two passes plus waits for concurrent transactions and cannot run inside a transaction block. A plainCREATE INDEXor anALTER TABLEvariant takes ACCESS EXCLUSIVE, which queues behind any running query and blocks every reader that arrives behind it in the queue — a five-millisecond statement becomes an outage while one long report finishes. Guard every ACCESS EXCLUSIVE step withSET lock_timeout = '2s'and retry in a loop, so a failed acquisition ends as a failed migration step instead of a stalled table. - Order the deploy so the window is never open: add the column; ship the application code that derives the idempotency key from
(determination_id, benefit_period_start), treats 23505 as a converged retry, and stampsreversed_by_instruction_idwhenever it writes a reversal; remediate the existing duplicates in chunks; re-run the residual-duplicate check; build the index concurrently; verifyindisvalidinpg_index. Building the index before the code can handle a violation converts every legitimate retry into a user-visible error.
Worked solution (45 min)
- Reproduce at reduced scale: load a few million rows with an injected duplicate rate and a mix of statuses, and keep a writer running throughout the migration.
- Run the ROW_NUMBER query, bucket the duplicates by status, and write the two remediation statements —
voidedfor unsent, a compensating reversal plusreversed_by_instruction_idon the duplicate for settled. - Attempt
CREATE UNIQUE INDEX CONCURRENTLY ... WHERE status <> 'voided'against the remediated data and watch it abort on 23505; inspectpg_index.indisvalidand drop the leftover index concurrently. - Add
reversed_by_instruction_id, backfill it for the remediated duplicates, and re-run the residual-duplicate check under the full three-conjunct predicate until it returns zero rows. - Attempt a plain
CREATE UNIQUE INDEXwith a long-runningSELECTopen in another session and observe the writer behind it stall; roll it back. - Run
CREATE UNIQUE INDEX CONCURRENTLYwith the full predicate while the writer is still running, and time it. - Replay the original duplicating batch and confirm the second attempt raises 23505 and the application converges on the existing instruction.
Expected result: The residual-duplicate check under the index predicate returns zero rows before the build starts, and the concurrent build then completes with the writer uninterrupted and indisvalid = true. Settled duplicates survive as rows: the n = 1 original stays inside the predicate, the duplicate carries reversed_by_instruction_id and its reversal carries reversal_of_instruction_id, so both fall outside it. Unsent duplicates carry status = 'voided' and are outside it too. A replayed batch pays once, and the duplicate attempt is visible as a counted 23505 rather than a second payment.
Verify it yourself
SELECT indisvalid FROM pg_index WHERE indexrelid = 'uq_instr_business_key'::regclassreturns true — a failed concurrent build leaves an index that exists and enforces nothing.- The
GROUP BY ... HAVING count(*) > 1check under the index predicate returns no rows both before and after the build: the build enforces the invariant, it is not how you discover whether the data satisfies it. - Leaving one settled duplicate unremediated reproduces the abort — 23505 during the build and
indisvalid = falseafterwards — which is the direct proof that the predicate, not the status column alone, is what makes the data fit. - Net settled amount per
(determination_id, benefit_period_start)after remediation equals the intended single payment, with the reversal accounted for. - During the concurrent build, the background writer's error count stays zero and its latency percentile does not step change; and
ALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING INDEXon the partial index fails, confirming you understood what you traded away.
Follow-up
- The concurrent build fails at 90% because of a deadlock with a nightly batch. What is on disk now, and what exactly do you run before retrying?
- A duplicate settled and then the original was returned by the rail. What does the compensating row look like, and what is the account's net position?
- Once the partial index exists, how does the batch writer's
ON CONFLICTinference clause have to be written against a three-conjunct predicate, and what happens if the predicate is omitted?
Measure time in status from an append-only case event log
Medium · window functions · event log · frame semantics · backdating
case_event(case_event_id, case_id, reopen_cycle, seq BIGINT, event_type TEXT, occurred_at TIMESTAMPTZ, recorded_at TIMESTAMPTZ) is append-only with UNIQUE (case_id, seq). Staff backdate occurred_at; recorded_at is when the row landed. A case can be reopened, so submitted and determined repeat. Write a query returning one row per decision episode with the interval from its submitted to its determined, plus a flag for episodes still open. State which column you order the window by and why the other one gives wrong answers, and say what your query returns for a cycle whose determined carries an occurred_at earlier than its submitted.
Approach
- Partition on the real grain,
(case_id, reopen_cycle). Withcase_idalone, an open cycle'ssubmittedtakes the next cycle'ssubmittedas its LEAD, so a cycle that never closed is reported closed with an interval spanning a cycle boundary. KeepLEAD(event_type)in the output so that is detectable rather than assumed. - Order by
seq, notoccurred_at.seqis unique per case by constraint, so the ordering is total and the result is deterministic.occurred_atis asserted by staff: adeterminedbackdated below its ownsubmittedswaps the two rows, thesubmittedrow becomes last in its partition, LEAD returns NULL, and a cycle that was decided is reported still open — it leaves the completed-cycle statistics entirely rather than showing up as a bad number. - Use LEAD over the filtered event set:
SELECT case_id, reopen_cycle, occurred_at AS submitted_at, LEAD(occurred_at) OVER w AS determined_at, LEAD(event_type) OVER w AS next_type FROM case_event WHERE event_type IN ('submitted','determined') WINDOW w AS (PARTITION BY case_id, reopen_cycle ORDER BY seq), then keep the rows whereevent_type = 'submitted'. WHERE is evaluated before window functions, so filtering to two event types first is both correct and cheap. - Separate pairing from measurement, because
seqonly fixes the first. Ordering byseqguarantees you paired the right two rows; it cannot repair a timestamp a human asserted, so a backdated cycle still yieldsdetermined_at - submitted_atbelow zero. Emit it as computed alongside(determined_at < submitted_at) AS asserted_interval_negative. Do not clamp withGREATEST(..., interval '0')and do not swap the endpoints: the negative value is the only evidence the entry is wrong, and the flag is what keeps it out of the published average without hiding it from the people who can correct it. - Carry the system-time measure beside the asserted one:
recorded_at - LEAD(recorded_at)inverted, i.e.LEAD(recorded_at) OVER w - recorded_at. Precondition for calling it non-negative:recorded_atis assigned by the database at insert and appends for one case are serialised — a per-caseseqissued under the case row lock gives you both properties at once. If a case can be appended to concurrently,recorded_atcan invert againstseqtoo and the flag must cover both measures. - Handle the open episode as data, not as a default: when LEAD returns NULL there is no
determinedyet, so emitdetermined_at IS NULLand astill_openflag. Coalescing tonow()silently converts a censored observation into a completed one and drags every average down. - Be explicit about frames. LAG and LEAD are not frame-sensitive — they address the partition directly — but the moment you add a running aggregate, the default frame is
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, where CURRENT ROW in RANGE mode means the last peer. Ordering on a tie-prone timestamp therefore makes peers of distinct events and sums them all at once; ordering on uniqueseqmakes RANGE and ROWS coincide.
Follow-up
- The published service standard is in business days in the case's jurisdiction. What does this query have to become, and where does the holiday calendar live?
- Two
determinedevents exist in one cycle because a correction was recorded. Which one does your query pair, and which one should it pair? - Rewrite the still-open flag so a cycle with
withdrawnbetweensubmittedanddeterminedis excluded rather than counted as open.
Explain how you would ensure system interoperability between ground terminals and space-based assets.
Medium · system design and architecture
Approach
- Choose a partition key and say what query it makes expensive.
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
Follow-up
- What would you drop to keep the system up under load?
- How does this behave when that dependency is down for an hour?
How would you architect a system to handle high data rates while maintaining low latency?
Medium · system design and architecture
Approach
- Name the read and write paths separately; they rarely have the same bottleneck.
- Name the failure you are designing for, then the recovery path.
- Choose a partition key and say what query it makes expensive.
Follow-up
- What would you drop to keep the system up under load?
- How does this behave when that dependency is down for an hour?
Design a software service for coordinating satellite communication tasks at scale.
Medium · system design and architecture
Approach
- State the consistency you need, and where you are willing to be stale.
- Choose a partition key and say what query it makes expensive.
- Fix the scope first: who calls this, how often, and what they do when it fails.
Follow-up
- What would you drop to keep the system up under load?
- What breaks first when traffic grows ten times?
How do you balance feature development with the need for high-assurance security in government-facing projects?
Medium · system design and architecture
Approach
- Fix the scope first: who calls this, how often, and what they do when it fails.
- Choose a partition key and say what query it makes expensive.
- Name the failure you are designing for, then the recovery path.
Follow-up
- What breaks first when traffic grows ten times?
- How does this behave when that dependency is down for an hour?
How do you approach debugging complex hardware-software integration issues?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
How do you ensure high performance and reliability in embedded Linux or RTOS environments?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
Replay millions of determinations after a retroactive rule fix
Medium · batch replay · idempotency · backpressure · partitioning
A rule version in force for fourteen months is corrected. Every determination decided under it must be re-evaluated against the corrected text and against the evidence snapshot frozen at the time; 3.2 million determinations are in scope. Eligibility performs no I/O and sustains 2,000 to 4,000 evaluations a minute, shared with live intake. A re-run must converge on the existing determination where the outcome is unchanged and supersede it where it is not. Deliverable: work partitioning, the convergence rule, rate control against intake, and the handling of a snapshot that no longer hashes.
Approach
- Compute the floor before choosing a topology. 3.2 million at 4,000 a minute is 800 minutes, about 13 hours, and that assumes the entire service. Capped at 1,000 a minute to leave headroom for intake it is 3,200 minutes, about 53 hours. Make the cap a runtime knob and pick it from the intake p95 you are willing to spend, not from a thread-pool constant.
- Partition into fixed chunks of case_id ranges within the rule version and claim them with SELECT ... FOR UPDATE SKIP LOCKED against a chunk table, so workers scale horizontally and a crashed worker's chunk becomes re-claimable after its lease expires. Do not partition by decision date: daily volume is deadline-shaped, so date chunks are wildly uneven and the tail workers idle while one chunk runs for hours.
- Write the convergence rule explicitly. Evaluate, then compare (outcome, reason_codes, award_amount_minor) against the current determination. Identical means write nothing but record a replay receipt. Different means insert a superseding determination with supersedes_determination_id set and flip is_current in one transaction, relying on the partial unique index UNIQUE (case_id, reopen_cycle) WHERE is_current to reject a concurrent writer instead of checking first under READ COMMITTED, where the check is not atomic.
- Drive worker concurrency from observed eligibility p95 rather than a fixed count. The replay has days of slack; a live submission has a closing window. A replay that pushes p95 past 200 ms is borrowing latency from people who cannot wait.
- Quarantine evidence drift instead of deciding through it. If the canonicalised artifact set no longer hashes to evidence_snapshot_hash, because a document was destroyed under retention or re-scanned, the replay is answering a different question. Route those cases to a manual list rather than re-deciding on today's evidence.
- Check the real bottleneck at the end. Every changed outcome produces a determination, a notice, and possibly a disbursement correction. Against a notice capacity of 1 to 2 million a month, the changed subset, not the evaluation rate, is what sets the wall-clock end date.
Worked solution (35 min)
- Compute the replay duration at 4,000 a minute and again at a capped 1,000 a minute, and state which you would run and what the cap is protecting.
- Write the chunk claim query with SELECT ... FOR UPDATE SKIP LOCKED and describe what happens to a chunk whose worker dies mid-flight.
- Write the convergence branch in full: the comparison tuple, the no-op path with its receipt, and the superseding insert plus is_current flip in one transaction.
- State the quarantine rule for a case whose current evidence set no longer matches evidence_snapshot_hash, and what the case worker sees.
- Estimate downstream notice volume from the changed subset and compare it against a capacity of 1 to 2 million notices a month.
Expected result: 800 minutes at full rate versus roughly 53 hours capped, with the cap justified by intake p95; re-claimable chunks keyed on case_id ranges; a convergence rule that writes nothing when nothing changed and supersedes atomically when it did; a quarantine list for drifted snapshots; and an explicit statement that notice fan-out gates the end date.
Verify it yourself
- Running the replay a second time over the same chunk produces zero additional superseding determinations.
- Two workers racing on one case produce exactly one constraint violation, handled as an ordinary outcome rather than a 500.
- A case whose snapshot hash changed appears on the quarantine list and has no new determination row.
Follow-up
- The replay finds 180,000 changed outcomes and all of them are adverse. What has to happen before any of them takes effect, and what does that do to your schedule?
- Half the workers die at hour nine. What work is lost, what is redone, and what proves nothing was decided twice?
- How do you demonstrate to an auditor that a determination the replay reported as unchanged was genuinely not altered?
Two current determinations survive a partial unique index
Hard · concurrency · partitioning · unique constraint
About twice a week a case has two determination rows with is_current true for one (case_id, reopen_cycle), which UNIQUE (case_id, reopen_cycle) WHERE is_current should make impossible. It has never reproduced in staging, and a build with extra logging ran clean for a week, so the team is calling it a concurrency heisenbug and proposing an advisory lock. The determination table was range-partitioned on decided_at six months ago. Deliver an ordered checklist, state the single query you run first, and explain precisely what its result rules out.
Approach
- Run the discriminating query before adopting anyone's theory: for each offending pair, select both rows' decided_at, decided_by, rule_version_id, supersedes_determination_id and the partition each row physically lives in. The spread in decided_at settles it — milliseconds apart is a genuine race, months apart cannot be one, and no amount of locking would have prevented it.
- Verify the constraint is real and enforcing. List indexes on the parent and on every partition, and check pg_index.indisvalid and indisready: a CREATE INDEX CONCURRENTLY that failed leaves an index present in the catalog, visible in tooling, and enforcing nothing. This costs one query and eliminates a whole class.
- Check the partitioning interaction, which is the likely answer. A unique index on a partitioned table must include every partition key column, so a partial unique index on (case_id, reopen_cycle) alone cannot exist on the parent; it was almost certainly created per partition instead. Per-partition uniqueness permits exactly one current row per partition, so an original determination and a correction re-run months later fall into different partitions and both stand.
- Explain both pieces of the folklore with that mechanism. Staging never reproduced it because seeded data occupies one partition, where the constraint does hold. The logging build proved nothing because the mechanism is not timing-dependent, and a week without an occurrence is entirely consistent with a twice-weekly rate.
- Fix by moving the invariant somewhere it can be global: either partition on something functionally determined by case_id so every row for a case shares one partition, or hold the invariant in a small unpartitioned current_determination table with UNIQUE (case_id, reopen_cycle) and a foreign key to the determination row, written in the same transaction as the supersession.
- Then backfill deliberately: resolve each existing duplicate pair by following supersedes_determination_id, not by ordering on decided_at, because a corrected determination can legitimately carry an earlier effective date than the one it supersedes. Record what was changed, since some of those cases have already been noticed and appealed.
Follow-up
- You switch to hash partitioning on case_id. What does that cost the archival job that currently drops old partitions by date?
- During the backfill, what does the read API return for a case that currently has two current rows?
- Some of these cases were noticed under the wrong current determination. What does correcting them require beyond a database change?
Suggested practice checklist
Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.
Day 1 — Coding, one pass at shallow depth
- Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
- For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
- Fix nothing today. The value of the pass is the unfixed record.
Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.
Day 2 — Design, one pass at shallow depth
- Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
- After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
- Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.
Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.
Day 3 — Fundamentals and the practical rounds
- Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
- Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
- Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.
Deliverable: Eight scored short answers and one written reading of unfamiliar code.
Day 4 — The rounds that are about you, and the map
- Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
- Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
- Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.
Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.
Day 5 — First chosen area, to the depth you skipped
- Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
- After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
- Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.
Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.
Day 6 — Second chosen area, where the gap is coverage rather than speed
- Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
- Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
- Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.
Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.
Day 7 — Reassemble the loop
- Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
- Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
- Reduce the week to one page holding only the rules you can state without reading them.
Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.
Bounding a vague ticket before writing code
When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.
Explain your experience with network layer packet processing and Software Defined Networking (SDN).
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Close with what you would do differently, concretely.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What did you decide not to do, and why?
- What would you do differently if you ran that again?
Describe a time you had to refactor a system to improve its scalability or reliability.
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
- Close with what you would do differently, concretely.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Unblock an engineer whose replay creates duplicate determinations
Easy · mentoring · idempotency · replay · constraints
An engineer's bulk replay job, re-running determinations after a rule correction, has left some cases with two rows flagged current and has re-issued notices for cases whose outcome did not change at all. They have been on it for two days and are now proposing a cleanup script to delete the extra rows. Describe unblocking someone in this shape: how you found where their model was wrong, what you had them do first, and how you left them able to handle the same class of bug without you.
Approach
- Get the invariant said out loud before anyone touches code. 'A replay that changes nothing writes nothing' is usually the sentence they have not yet said, and saying it reframes the problem from a concurrency puzzle into a missing no-op check, which is a different and much smaller fix.
- Separate the two failures they have merged into one bug. Duplicate current rows are a concurrency defect the database should have rejected; re-noticing unchanged cases is a logic defect in the job. A cleanup script addresses neither, and running it before the constraint exists guarantees running it again next week.
- Have them reproduce the smaller failure themselves — two concurrent inserts against a table with no partial unique index — rather than describing it. A reproduction they built transfers to the next bug; a diagnosis handed to them does not.
- Be explicit about what you deliberately did not do: you did not take the keyboard, and you did not let the cleanup script go first. Say how you decided that, because with an incident running the wrong answer is to teach instead of to stop the bleeding, and the judgement call is the interesting part.
- Close the loop on repeatability for them: a no-op predicate compared before writing, a dry-run mode whose output is counts by reason code, and agreement on who reviews the next replay before it runs.
Follow-up
- Notices already went to constituents whose determination did not change. What are they told, and who writes it?
- How do you tell whether the duplicate rows came from concurrency or simply from the job being run twice?
- What changes in your approach if an auditor found this rather than the engineer?