Interview guides · Abile Group

Abile Group Software Engineer Interview Questions & Guide 2026

Abile Group software engineer prep on databases: indexes, query plans, and the anomalies an isolation level still permits.

The 60-second brief

A Software Engineer at Abile Group occupies a critical position within a company dedicated to providing high-end technical solutions, particularly within defense, intelligence, and federal sectors. This role is not merely about writing code; it is about engineering robust, secure, and mission-critical systems that support complex organizational needs. You will be tasked with solving high-stakes challenges that require a blend of technical precision and strategic foresight.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

Model legal holds before modelling record destruction
Treat accessibility conformance as a functional requirement
Version rules by statutory effective date, not release

Overview

The impact of your work at Abile Group is tangible. Whether you are working on network engineering, insider threat programs, or specialized data guards, your contributions directly influence the security and operational efficiency of the projects you support. You will be working in an environment that values deep technical expertise, requiring you to remain adaptable as you navigate the intersection of software development and secure infrastructure.

The Abile Group interview process

Abile Group candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Deep-Dive

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Program Manager Interview

Expect the estimate to be probed harder than the outcome. A manager asking how long something will take is checking whether your number covers the parts that reliably go wrong: migrating the rows that already exist, and the second service that has to change at the same time. A single confident number is the weak answer. The strong one states what the estimate assumes, names the unknown that could double it, and says what you would cut first if the date were fixed. Scope is the variable you control; hours are not.

What to demonstrate

Whether an estimate arrives with its assumptions attached, including what it excludes and how much of it is waiting on review, environments or another team
Whether you can name work you cut and the person you told, since a cut nobody heard about surfaces later as a surprise rather than a decision
How you order two tasks that both look urgent, and whether that order survives being told the deadline moved in by a week
Whether you separate what must be done before launch from what must be done before the thing can be left running unattended

How to prepare

Take the last three things you shipped and write down, from memory, the estimate you gave and the time it actually took. Bring the largest miss and why it missed; a candidate who has never missed has never committed to a date.
Against your current queue, write who is blocked until each item lands. Anything with nobody behind it becomes your worked example of what you would drop and how you would say so.
Rehearse the fixed-date answer aloud: which slice ships, which slice is deferred and written down somewhere it will be found, and the one thing you would refuse to cut because removing it makes the release unsafe rather than smaller.

Where candidates lose points

Using soft delete (deleted_at IS NOT NULL) as the answer to a records-retention requirement

It fails both directions at once. The obligation to destroy is unmet because the bytes remain in the primary table, the backups, the search index, the analytics extract and any vendor queue that received a copy, and a competent disclosure request or audit will find them. Meanwhile the record is invisible to staff who legitimately need it, so the organisation loses the ability to answer questions it is still required to answer. Retention needs a destruction pipeline that enumerates derived copies, checks legal hold at execution time under lock, emits a receipt, and fails closed when a copy is unreachable - and the tombstone row that proves destruction happened has to survive the destruction.

Running an EBCDIC-to-ASCII text conversion across a whole fixed-width record coming from a legacy system of record

Those records mix character fields with binary and packed-decimal (COMP-3) numeric fields, where two digits are packed per byte and the sign occupies the low-order nibble of the final byte. A character transcode rewrites those bytes, so an amount does not fail to parse - it parses into a different, entirely plausible number, and the corruption is discovered in reconciliation weeks later or not at all. Parse by field using the record layout: transcode only the character fields, unpack the numeric fields explicitly, and add a checksum or control-total record to the file contract so a mangled batch is rejected rather than posted.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Questions & practice

Parse a fixed-width record with packed-decimal amount fields

Easy · parsing · encoding · legacy-integration

A nightly file from a system of record arrives as fixed-length 180-byte records in EBCDIC code page 037. Per the copybook, bytes 1-40 are a character name field, bytes 41-48 a character date, bytes 49-52 an amount declared PIC S9(5)V99 COMP-3, and the rest character. The trailer record carries a record count and the sum of every amount field. Write the parser and state, in minor currency units, what a whole-record EBCDIC-to-ASCII text conversion does to a packed amount before it ever reaches your validation.

Approach

Parse by field against the copybook offsets, never by transcoding the record. Slice bytes 1-40 and 41-48 and decode those with cp037; slice bytes 49-52 and hand them to a packed-decimal unpacker that never sees a codec.
Unpack COMP-3 explicitly: two BCD digits per byte, and in the final byte the high nibble is the last digit while the low nibble is the sign. Nibble C and F are positive and D is negative under the common convention, so assert the sign nibble rather than defaulting an unrecognised one to positive. S9(5)V99 is seven digits plus a sign nibble, which is eight nibbles, which is exactly the four declared bytes.
Return the value as an integer count of minor units. V99 is an implied decimal position with nothing stored for it, so the seven unpacked digits already are the minor-unit amount; converting to a float to reinsert a decimal point reintroduces the representation error the packed format exists to avoid.
Reject the batch on the trailer, not per record. Accumulate the record count and the integer sum of every amount, compare to the trailer, and fail the whole file rather than post a partial batch. A corrupted amount that still parses is invisible to per-record validation and visible only to the control total.
Complexity is O(total bytes) with O(1) working memory beyond one record buffer and two accumulators; the file is read as a byte stream, so nothing here needs the file resident.

Follow-up

The copybook gains an OCCURS DEPENDING ON clause, so record length now varies with a count field. How does your offset arithmetic change?
A record contains a sign nibble of 0xA. Is that valid, and what does your parser do with it?
The sender starts padding numeric character fields with EBCDIC zeros in a field you decode as text. What breaks and where do you catch it?

Rebuild an award timeline as of a past decision date

Medium · intervals · bitemporal · sweep-line

One case has up to 400 determination rows across all reopen cycles: determination_id, reopen_cycle, supersedes_determination_id, decided_at TIMESTAMPTZ, effective_from DATE, effective_to DATE (NULL meaning open-ended, both endpoints inclusive), award_amount_minor. Given a set of queries (benefit_date D, as_of decision time T), return the award in force for D according to what had been decided by T, plus any benefit dates inside the covered span that no determination covers. Later-decided wins on overlap. Give the complexity for building the answer and for each query.

Approach

Convert every interval to half-open [effective_from, effective_to + 1 day) at the boundary of the system, once. Keeping inclusive ends in the algorithm produces an off-by-one at every adjacency, and the error is a single uncovered or doubly-covered day that a reviewer reads as a gap in benefit.
For a given T, filter to determinations with decided_at <= T. Do not use is_current: that column answers what is in force now, and the whole point of the as-of query is that a determination decided after T must be invisible even though it exists today.
Paint the surviving intervals onto a timeline in ascending decided_at order so later decisions overwrite earlier ones where they overlap. Equivalently, sweep the sorted endpoints with a max-heap keyed on decided_at and lazily discard heap entries whose interval has already ended. O(n log n) build.
Answer each query by binary search over the painted, non-overlapping interval list: O(log n). Gaps fall out of the same structure as the elementary intervals with no covering determination between the minimum effective_from and the maximum effective_to of the surviving set. Note what that definition implies: deleting one determination only produces a gap where nothing else still covers those dates, so an overlapping earlier award absorbs the deletion silently and correctly.
With n at most 400, a quadratic paint over elementary intervals is about 160k comparisons and is genuinely fine; say so rather than reaching for the heap by reflex. The heap earns its place when the same routine runs across a program-wide cohort rather than one case.
If queries carry many distinct T values, the painted timeline is per-T and rebuilding it per query is O(Qn log n). Sort queries by T and advance one sweep through decided_at order, materialising each version as you pass it, or accept the rebuild when Q is small.

Worked solution (30 min)

Build five determinations for benefit year Y: d1, decided in January, pays 50000 minor units for [Jan 1, Dec 31]; d2, decided in June, pays 62000 for [Mar 1, Dec 31]; d3, decided in November on reopen cycle 1, pays 0 for [Jul 1, Sep 30]; d4, decided in December, restores 62000 for [Sep 1, Sep 30]; d5 covers [Jan 1, Jan 31] of year Y+1.
Paint the timeline with T set to the end of July and read the award for a benefit date in August.
Repeat with T set to the end of December and read the same August date, then read a September date under both values of T.
Delete d2 alone, re-run at T in December, and read both the August award and the gap report.
Delete d1 as well, leaving only d3, d4 and d5, and re-run the gap report.

Expected result: At T in July, August returns 62000 and September returns 62000. At T in December, August returns 0 and September returns 62000, because the November decision covers both and the December decision overwrites only September. Deleting d2 produces no gap at all: d1 still pays 50000 for the whole of year Y, so August returns 50000 and the gap report is empty — removing a superseding interval falls back to what it superseded rather than exposing a hole. Deleting d1 as well leaves a surviving span of [Jul 1 of Y, Jan 31 of Y+1] with nothing covering Oct 1 through Dec 31, and the gap report names exactly those dates.

Verify it yourself

Every interval in the painted output is half-open, and on the untrimmed fixture each shares an endpoint with its neighbour, so the union has no one-day seam.
Any query with T earlier than the first decided_at returns no coverage at all rather than the earliest award.
Re-running with the intervals shuffled produces the identical painted list, because the order is imposed by decided_at and not by input order.
The gap report and the award query never disagree: on the fixture with d2 deleted both say August is covered, at 50000, and on the fixture with d1 and d2 deleted both say October through December is uncovered.

Follow-up

A determination is decided with an occurred-earlier effective_from that reaches back behind an already-settled disbursement. What does the timeline say, and what does that imply for the payment side?
Two determinations in different reopen cycles share a decided_at to the microsecond. What is your tiebreak and why must it be total?
How do you answer the same question for 200,000 cases in a nightly report without rebuilding 200,000 heaps?

Add N business days in constant time over amendable calendars

Hard · rank-select · civil-time · precomputation

Case Service computes sla_due_on at 1 to 2 thousand transitions per second, with higher bursts when overnight intake lands. A holiday table holds (jurisdiction_code, holiday_date, observed_date, calendar_version_id) across 200 jurisdictions and a 1990-2040 civil date span; a jurisdiction may not observe a given holiday, and calendar versions are amended over time. The statutory convention is that the period begins the day after the triggering act. Design the structure that answers 'add N business days to civil date D in jurisdiction J' in constant time, and state its memory cost.

Approach

Work entirely in civil date ordinals, never in instants. A civil day is not always 86,400 seconds across a daylight-saving transition, so any arithmetic that adds an interval to a UTC timestamp is wrong twice a year in the direction that reports a late case as on time. Convert the triggering instant to a civil date in the case's zone once, at the boundary, and stay in ordinal space after that.
Precompute two arrays per (jurisdiction, calendar_version): rank[d], the count of business days in the span up to and including ordinal d, and select[i], the ordinal of the i-th business day. Build both in one linear pass over the span using observed_date, not holiday_date, so a holiday shifted off a weekend lands where the jurisdiction actually observes it.
The query is then two array reads: r = rank[D], answer = select[r + N]. This is correct for both cases without branching. If D is a business day, rank counts it and select[r + N] is N business days later; if D is a weekend or holiday, rank[D] equals rank of the last preceding business day, so select[r + N] still lands N business days after the period begins on the following business day. O(1) per query after an O(span) build.
Cost it: 18,628 days in the span, two int32 arrays of roughly 74 KB and 52 KB, so about 126 KB per jurisdiction-version. Two hundred jurisdictions at six live versions each is 1,200 array pairs, roughly 150 MB resident. That is a deliberate trade of memory for a constant-time query at 2,000 per second; a day-walking loop is O(N) per query and, at a 45 business-day standard, is 45 bounds-checked iterations per transition instead of two.
Select the calendar version bitemporally, because there are two different questions. A due date being set now uses the currently recorded version; recomputing a past due date for an audit uses the version recorded as in force when the original computation was made. Key the cache on calendar_version_id so an amendment invalidates exactly the affected arrays rather than the whole cache.
Guard both ends of the span. A query for D outside [1990, 2040], or an r + N beyond the last business day, must raise rather than clamp: a clamped due date is a plausible wrong date, and this is the arithmetic that decides whether a case is late by law.

Follow-up

A jurisdiction amends its calendar with retroactive effect, adding a holiday in a period whose sla_due_on values are already stored. Which stored dates are wrong and how do you enumerate them?
The statute for one program counts calendar days, not business days, but still rolls a deadline off a non-business day. Does your structure answer that, and what changes?
Two jurisdictions share a holiday list except for three dates. Is it worth sharing the arrays, and what does the sharing cost you when one of them amends?

Measure time in status from an append-only case event log

Medium · window functions · event log · frame semantics · backdating

case_event(case_event_id, case_id, reopen_cycle, seq BIGINT, event_type TEXT, occurred_at TIMESTAMPTZ, recorded_at TIMESTAMPTZ) is append-only with UNIQUE (case_id, seq). Staff backdate occurred_at; recorded_at is when the row landed. A case can be reopened, so submitted and determined repeat. Write a query returning one row per decision episode with the interval from its submitted to its determined, plus a flag for episodes still open. State which column you order the window by and why the other one gives wrong answers, and say what your query returns for a cycle whose determined carries an occurred_at earlier than its submitted.

Approach

Partition on the real grain, (case_id, reopen_cycle). With case_id alone, an open cycle's submitted takes the next cycle's submitted as its LEAD, so a cycle that never closed is reported closed with an interval spanning a cycle boundary. Keep LEAD(event_type) in the output so that is detectable rather than assumed.
Order by seq, not occurred_at. seq is unique per case by constraint, so the ordering is total and the result is deterministic. occurred_at is asserted by staff: a determined backdated below its own submitted swaps the two rows, the submitted row becomes last in its partition, LEAD returns NULL, and a cycle that was decided is reported still open — it leaves the completed-cycle statistics entirely rather than showing up as a bad number.
Use LEAD over the filtered event set: SELECT case_id, reopen_cycle, occurred_at AS submitted_at, LEAD(occurred_at) OVER w AS determined_at, LEAD(event_type) OVER w AS next_type FROM case_event WHERE event_type IN ('submitted','determined') WINDOW w AS (PARTITION BY case_id, reopen_cycle ORDER BY seq), then keep the rows where event_type = 'submitted'. WHERE is evaluated before window functions, so filtering to two event types first is both correct and cheap.
Separate pairing from measurement, because seq only fixes the first. Ordering by seq guarantees you paired the right two rows; it cannot repair a timestamp a human asserted, so a backdated cycle still yields determined_at - submitted_at below zero. Emit it as computed alongside (determined_at < submitted_at) AS asserted_interval_negative. Do not clamp with GREATEST(..., interval '0') and do not swap the endpoints: the negative value is the only evidence the entry is wrong, and the flag is what keeps it out of the published average without hiding it from the people who can correct it.
Carry the system-time measure beside the asserted one: recorded_at - LEAD(recorded_at) inverted, i.e. LEAD(recorded_at) OVER w - recorded_at. Precondition for calling it non-negative: recorded_at is assigned by the database at insert and appends for one case are serialised — a per-case seq issued under the case row lock gives you both properties at once. If a case can be appended to concurrently, recorded_at can invert against seq too and the flag must cover both measures.
Handle the open episode as data, not as a default: when LEAD returns NULL there is no determined yet, so emit determined_at IS NULL and a still_open flag. Coalescing to now() silently converts a censored observation into a completed one and drags every average down.
Be explicit about frames. LAG and LEAD are not frame-sensitive — they address the partition directly — but the moment you add a running aggregate, the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, where CURRENT ROW in RANGE mode means the last peer. Ordering on a tie-prone timestamp therefore makes peers of distinct events and sums them all at once; ordering on unique seq makes RANGE and ROWS coincide.

Worked solution (30 min)

Seed one case with five cycles: two clean closed cycles; one open cycle holding only a submitted; one cycle whose determined has an occurred_at eight days before its submitted but a recorded_at fifteen days after it; and one cycle whose two events share an occurred_at to the millisecond.
Write the LEAD query ordered by seq, keeping next_type, and record asserted_interval, the system-time interval and the negative flag for all five cycles.
Re-run the identical query ordered by occurred_at and diff the two result sets row by row, noting which cycle changes category rather than value.
Drop reopen_cycle from the PARTITION BY and confirm the open cycle acquires a pair with next_type = 'submitted' that spans a cycle boundary.
Add a running COUNT(*) with the default frame ordered by occurred_at, then by seq, and compare the two columns on the tied pair.

Expected result: Ordered by seq, all four closed cycles pair correctly and the open cycle yields determined_at IS NULL with still_open true. Exactly one row carries a negative asserted_interval — the backdated cycle, at minus eight days, with asserted_interval_negative true and a system-time interval of plus fifteen days. Ordered by occurred_at, that cycle instead drops out of the closed set and is reported still open, and the tied pair's running count jumps by two at once under the default RANGE frame.

Verify it yourself

Every output row's next_type is determined or NULL, and the output row count equals COUNT(DISTINCT (case_id, reopen_cycle)) over cycles containing a submitted — a row paired with another submitted means the partition crossed a cycle boundary.
The one negative asserted_interval is present in the output and flagged, not clamped to zero and not endpoint-swapped; the published average excludes it by the flag rather than by it being missing.
The open cycle appears in the output with a NULL end rather than being dropped by the join or filled with now().
Switching the ORDER BY changes which cycles count as closed, not merely their numbers — the direct evidence that the choice of column is load-bearing rather than cosmetic.
Switching the running aggregate's frame from the default RANGE to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW changes only the tied rows.

Follow-up

The published service standard is in business days in the case's jurisdiction. What does this query have to become, and where does the holiday calendar live?
Two determined events exist in one cycle because a correction was recorded. Which one does your query pair, and which one should it pair?
Rewrite the still-open flag so a cycle with withdrawn between submitted and determined is excluded rather than counted as open.

Decide what the caseworker queue is allowed to denormalise

Medium · denormalisation · read model · write amplification · identity merge

The queue screen shows 50 cases with constituent display name, program label, assigned unit, days remaining against sla_due_on, and current outcome — data spread over five tables. A separate supervisor view aggregates the same fields across 2M open cases. Product asks to denormalise all five onto case. Decide what you would and would not copy, justify each choice from its read pattern, and describe the write path that keeps any copy correct. Constituent merges rename people across millions of rows, and status is written only by the Case Service.

Approach

Price the read before redesigning for it. The queue returns 50 rows; resolving five foreign keys per row is 250 primary-key lookups against small tables that live in cache, which is microseconds. Denormalisation buys nothing here, and a candidate who copies columns for this read is solving a problem the numbers do not show.
Separate a pointer from a copy. case.current_determination_id is a pointer maintained by the same transaction that sets is_current, so it cannot drift by more than that transaction's atomicity; status and sla_due_on are case-owned facts, not copies of anything. Neither is denormalisation, and conflating them with copied labels is how the conversation goes wrong.
Refuse the copy that a merge invalidates. Constituent display name must stay behind constituent_id, because a merge or a corrected legal name would otherwise require rewriting millions of case rows, and any row missed is a person shown under the wrong name on a legal artifact. The cost is not the write volume, it is that the failure mode is silent and personal.
Identify the read that genuinely justifies a copy: the supervisor aggregate over 2M open cases, which cannot use 50-row nested loops. Serve it from a rollup keyed on (program_code, assigned_unit_id, due_bucket) refreshed from case events, not by widening case — a rollup is bounded in size, can be rebuilt from the event log, and a stale rollup shows a stale count rather than a wrong name on a case.
Define the write path and the staleness contract: the Case Service is the sole writer of case status and publishes case_event; the rollup consumes those events and carries an explicit as-of timestamp that the UI shows. A derived copy whose staleness is invisible is worse than a slow join, because a caseworker will act on it.
State the one column worth copying and why: determination.outcome onto the rollup, because the aggregate groups by it and the value is immutable once written — a copy of an immutable fact needs no invalidation path at all, which is the only cheap kind.

Follow-up

The rollup is 40 seconds behind at intake peak and a supervisor reassigns work from it. What does the reassignment write, and against which row version?
Full-text search over constituent names is now required across 2M cases. Does that change your answer about copying the name, and what does a merge do to the index?
How do you rebuild the rollup from scratch after a bug, and what in the event log makes that replay deterministic?

How do you ensure code security throughout the development lifecycle?

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

How do you approach debugging complex issues in a distributed system?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Deliver and verify case-event webhooks to partner agencies

Medium · webhooks · signature verification · ordering

A partner agency subscribes to case events for one program. The producer reads case_event (case_id, reopen_cycle, seq, event_type, occurred_at, recorded_at, payload, payload_version). The consumer is the partner's HTTP endpoint, which may be down for hours and whose operators will later replay deliveries from an archive. Design the delivery contract: the signed request, the verification the receiver performs, the delivery guarantee, the ordering guarantee, and what a receiver does with a duplicate. State explicitly what the receiver must not infer from a delivery. Deliver the request shape, the signature scheme and the receiver's pseudocode.

Approach

Sign bytes, not objects: HMAC-SHA256 over a canonical string of timestamp, a separator and the raw request body, compared in constant time, with the timestamp also in a header and rejected outside a tolerance window so a captured request is not replayable forever. A receiver that parses and re-serialises JSON before verifying will fail on key order, whitespace and number formatting, and will end up disabling verification.
Decide where archive replay lives before choosing the tolerance. Re-posting two-day-old signed bytes forces the window open to uselessness, so operators replay through an authenticated backfill endpoint that re-signs at send time instead.
Rotate keys without an outage: send one signature per active key with a key id and let the receiver accept any valid one. With a single key, rotation is downtime for every partner that cannot deploy in the same minute.
State the guarantee as at-least-once. Publish from a transactional outbox written in the same transaction as the case_event row, so nothing is lost when the publisher dies mid-flight, and duplicates are expected rather than a defect.
Give the receiver a dedup key it can hold with a unique constraint: (case_id, reopen_cycle, seq). A per-delivery id deduplicates retries of one delivery but not an archive replay, which is precisely the case the partner's operators will create.
Order per case only, by seq, never globally. A receiver drops a delivery whose seq is below the highest applied for that (case_id, reopen_cycle), and must not order on occurred_at — staff backdate it deliberately, so it is an assertion about the world and not a clock. And say what a delivery is not: a 2xx from a partner is not dispatch of a statutory notice and releases no effective date; only the Notice Service's own dispatch record does that.

Worked solution (30 min)

Define the headers — event id, event timestamp, and a signature of the form v1=<key_id>:<hex> — and write the exact canonical string that is signed.
Write the receiver: reject when the timestamp is outside 300 s, verify with a constant-time compare over the raw body, then INSERT the dedup key and return 200 on conflict.
Re-deliver the same event with a new delivery id and confirm the receiver deduplicates on (case_id, reopen_cycle, seq).
Deliver seq 7 before seq 6 and confirm the receiver does not move the case backwards.
Kill the publisher between the outbox write and the HTTP call, restart it, and confirm exactly one logical apply at the receiver.

Expected result: Each event is applied once at the receiver despite duplicate deliveries and an out-of-order pair; verification succeeds on raw bytes and fails after any re-serialisation; the archive replay path re-signs rather than widening the tolerance window.

Verify it yourself

A body with reordered JSON keys fails verification, proving the signature covers bytes.
Two key ids both verify during a rotation window.
The receiver's deduplication still holds when the delivery id is new but the case event is old.
No code in the notice effective-date path reads a partner's response status.

Follow-up

The partner is down for six hours. What is your retry schedule, when do you stop, and how do they catch up without you replaying from memory?
payload_version moves to 2. How do you ship that to a partner still parsing version 1?
The partner asks for evidence document bytes inline in the webhook. What do you send instead, and what changes about retention?

Case projection lag grows on one partition while the consumer looks healthy

Medium · poison message · head-of-line blocking · idempotency

A consumer builds a read model from case_event (case_id, seq, event_type, payload JSONB, payload_version), applying events per case in seq order. Overnight, lag on one partition grows steadily while the others stay current. CPU is low, the process never crashes and restarts on its own schedule, and the error log shows the same case_event_id every few seconds. A producer shipped payload_version 4 that day. Deliver an ordered checklist, state exactly what the consumer should do with that message, and explain why dead-lettering it and advancing the offset corrupts the read model.

Approach

Separate head-of-line blocking from slowness with two signals: committed offset flat on one partition while others advance, and a redelivery count climbing on a single message. A slow consumer advances; a poisoned one does not, and low CPU rules out the consumer simply being behind.
Read the actual failure rather than inferring it. Distinguish an unreadable envelope (deserializer rejects payload_version 4 outright) from a readable envelope with a shape the handler does not cover (a field added in v4 that a branch dereferences). The first is a contract defect, the second a handler defect, and the fixes differ.
Choose the parking unit, which is the real question. The projection depends on per-case ordering by seq, guarded by UNIQUE (case_id, seq). Dead-lettering one event and advancing the offset means event seq n+1 is applied to a state that never saw seq n, and the read model for that case is silently wrong from then on with nothing to flag it. Park the case key: stop that case's stream, record the gap, let the rest of the partition drain.
Make the handler idempotent on (case_id, seq) before enabling any retry, because at-least-once delivery will re-apply whatever the handler does. A projection upsert keyed on that pair is naturally idempotent; a side effect such as dispatching a notice is not, and must be moved behind its own business-fact key.
Harden the contract so this class cannot recur: ignore unknown fields, route an unknown payload_version to the park path instead of raising, bound retries with jittered backoff, and alarm on two things — redelivery count, and per-case seq gaps in the projection. The gap alarm is the one that catches a future silent skip.
Plan the drain: once a v4-aware handler ships, replay the parked cases from their lowest parked seq, with side effects suppressed or deduplicated, and verify each case's projection matches a full rebuild from case_event.

Follow-up

The producer cannot be rolled back and 40,000 v4 events are already written. What is the order of operations to recover?
Replaying a parked case re-emits notice_dispatched handling. What key stops the notice going out twice?
How do you prove, after recovery, that no case silently skipped a seq?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

'We rewrote it in a better framework' is not a story

A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.

Describe your experience with network architecture and security protocols.

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Describe a time you had to communicate a complex technical concept to a non-technical stakeholder.

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Walk me through a time you had to troubleshoot a critical system failure under a deadline.

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

Tell me about a time you identified a process improvement that increased team efficiency.

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?