cswork / libraryOpen practice workspace

Interview guides · City of San Antonio

City of San Antonio Software Engineer Interview Questions & Guide 2026

A prep plan for City of San Antonio software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

The Software Engineer role within the City of San Antonio is a pivotal position dedicated to building and maintaining the digital infrastructure that serves one of the nation’s largest and fastest-growing municipalities. You will be responsible for developing, deploying, and supporting mission-critical applications that directly impact the daily lives of San Antonio residents.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

  • Make every determination replayable from a frozen snapshot
  • Treat accessibility conformance as a functional requirement
  • Key idempotency on business facts, not requests

Overview

This role is unique because it combines technical rigor with a profound sense of public service. You are not just writing code; you are facilitating government efficiency, improving accessibility to city services, and ensuring that internal systems function reliably for thousands of city employees. Whether you are working on Application Support or new feature development, your work serves as the backbone for city operations.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

  • Breadth across SQL, experimentation and product reasoning
  • Ability to state assumptions before choosing a method

How to prepare

  • Drill the practice exercises below and time yourself
  • Prepare three quantified stories about decisions you drove

Where candidates lose points

Computing a statutory due date by adding an interval to a UTC timestamp

The deadline is a civil date in a jurisdiction that observes daylight saving, and a civil day is not always 86,400 seconds; the same arithmetic also ignores weekends and a holiday list that is amended annually and differs by jurisdiction. The result is a due date off by a day near a transition, which reports a case as on time when it is late by law - the direction of the error that produces findings rather than complaints. Store the event as an instant, convert to the case's civil time zone, and walk the business-day calendar as versioned data with its own effective dates, then test the transitions and the holidays explicitly rather than the happy path in July.

Treating an integration partner as if it were a synchronous API when the system of record is a nightly batch

The effective consistency model is 'as of last night', so a screen that shows a constituent their updated status immediately after a submit is telling them something the authoritative system will not agree with for hours, and a caseworker who acts on it creates a conflict that surfaces as a reconciliation exception nobody owns. Read-after-write expectations also encourage retry-until-consistent loops that hammer a mainframe window sized for one pass. Make staleness explicit in the model and in the interface - show the as-of time, keep the local record in a pending state until the batch confirms, and design the reconciliation job as the place where disagreement is resolved rather than as an exception report.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Questions & practice

Rebuild an award timeline as of a past decision date

Medium · intervals · bitemporal · sweep-line

One case has up to 400 determination rows across all reopen cycles: determination_id, reopen_cycle, supersedes_determination_id, decided_at TIMESTAMPTZ, effective_from DATE, effective_to DATE (NULL meaning open-ended, both endpoints inclusive), award_amount_minor. Given a set of queries (benefit_date D, as_of decision time T), return the award in force for D according to what had been decided by T, plus any benefit dates inside the covered span that no determination covers. Later-decided wins on overlap. Give the complexity for building the answer and for each query.

Approach

  1. Convert every interval to half-open [effective_from, effective_to + 1 day) at the boundary of the system, once. Keeping inclusive ends in the algorithm produces an off-by-one at every adjacency, and the error is a single uncovered or doubly-covered day that a reviewer reads as a gap in benefit.
  2. For a given T, filter to determinations with decided_at <= T. Do not use is_current: that column answers what is in force now, and the whole point of the as-of query is that a determination decided after T must be invisible even though it exists today.
  3. Paint the surviving intervals onto a timeline in ascending decided_at order so later decisions overwrite earlier ones where they overlap. Equivalently, sweep the sorted endpoints with a max-heap keyed on decided_at and lazily discard heap entries whose interval has already ended. O(n log n) build.
  4. Answer each query by binary search over the painted, non-overlapping interval list: O(log n). Gaps fall out of the same structure as the elementary intervals with no covering determination between the minimum effective_from and the maximum effective_to of the surviving set. Note what that definition implies: deleting one determination only produces a gap where nothing else still covers those dates, so an overlapping earlier award absorbs the deletion silently and correctly.
  5. With n at most 400, a quadratic paint over elementary intervals is about 160k comparisons and is genuinely fine; say so rather than reaching for the heap by reflex. The heap earns its place when the same routine runs across a program-wide cohort rather than one case.
  6. If queries carry many distinct T values, the painted timeline is per-T and rebuilding it per query is O(Qn log n). Sort queries by T and advance one sweep through decided_at order, materialising each version as you pass it, or accept the rebuild when Q is small.

Worked solution (30 min)

  1. Build five determinations for benefit year Y: d1, decided in January, pays 50000 minor units for [Jan 1, Dec 31]; d2, decided in June, pays 62000 for [Mar 1, Dec 31]; d3, decided in November on reopen cycle 1, pays 0 for [Jul 1, Sep 30]; d4, decided in December, restores 62000 for [Sep 1, Sep 30]; d5 covers [Jan 1, Jan 31] of year Y+1.
  2. Paint the timeline with T set to the end of July and read the award for a benefit date in August.
  3. Repeat with T set to the end of December and read the same August date, then read a September date under both values of T.
  4. Delete d2 alone, re-run at T in December, and read both the August award and the gap report.
  5. Delete d1 as well, leaving only d3, d4 and d5, and re-run the gap report.

Expected result: At T in July, August returns 62000 and September returns 62000. At T in December, August returns 0 and September returns 62000, because the November decision covers both and the December decision overwrites only September. Deleting d2 produces no gap at all: d1 still pays 50000 for the whole of year Y, so August returns 50000 and the gap report is empty — removing a superseding interval falls back to what it superseded rather than exposing a hole. Deleting d1 as well leaves a surviving span of [Jul 1 of Y, Jan 31 of Y+1] with nothing covering Oct 1 through Dec 31, and the gap report names exactly those dates.

Verify it yourself

  • Every interval in the painted output is half-open, and on the untrimmed fixture each shares an endpoint with its neighbour, so the union has no one-day seam.
  • Any query with T earlier than the first decided_at returns no coverage at all rather than the earliest award.
  • Re-running with the intervals shuffled produces the identical painted list, because the order is imposed by decided_at and not by input order.
  • The gap report and the award query never disagree: on the fixture with d2 deleted both say August is covered, at 50000, and on the fixture with d1 and d2 deleted both say October through December is uncovered.

Follow-up

  • A determination is decided with an occurred-earlier effective_from that reaches back behind an already-settled disbursement. What does the timeline say, and what does that imply for the payment side?
  • Two determinations in different reopen cycles share a decided_at to the microsecond. What is your tiebreak and why must it be total?
  • How do you answer the same question for 200,000 cases in a nightly report without rebuilding 200,000 heaps?

Replay sixty million determinations after a retroactive rule correction

Hard · bulk-replay · inverted-index · determinism

A regulator corrects three clauses of a ruleset with effect six years back. Sixty million determination rows fall under the affected rule family, each carrying rule_version_id, evidence_snapshot_hash and a trace of the clauses that fired. Evaluation is a pure function costing about 8 ms of CPU, but each replay must rehydrate the frozen evidence snapshot from a content-addressed store capped at 500 fetches per second shared with live traffic. Replaying everything is correct. Explain why it is not viable, and give the algorithm that is.

Approach

  1. Cost the naive run before optimising it. CPU is not the constraint: 60 million evaluations at 8 ms is about 133 CPU-hours and parallelises freely. The store is: 60 million fetches against a 500 per second ceiling is 120,000 seconds, roughly 33 hours of wall clock while consuming the entire budget live traffic also needs. The run is correct because replay is deterministic and unaffected determinations converge on the existing row through the partial unique index, so it is purely a cost problem.
  2. Build an inverted index from clause to determination. Either a separate determination_clause table indexed on clause_id, or a GIN index over the trace array. The three corrected clauses then yield a candidate set by index scan instead of a full scan, typically well under one percent of the 60 million rows.
  3. Recognise that the fired-clause filter alone is unsound. A correction that widens a guard makes a clause newly reachable, so determinations that never reached it before can change now, and those are precisely the denials the correction was issued to reverse. If the trace only records clauses that fired, the filter silently omits the population the regulator cares about.
  4. Close the gap with what the trace can actually support, and be explicit that the target is a sound superset, not an exact set. If the trace records evaluated-and-false clauses as well as fired ones, include both: every determination that can change is then selected, along with some that cannot — a determination that already passed the widened guard is selected and replays identical, because widening a guard cannot exclude an input it already admitted. That asymmetry is the correct trade. A false positive costs one rehydration; a false negative is an uncorrected determination. If the trace records only fired clauses, fall back to a wider superset: every determination under the affected rule_versions whose effective period intersects the corrected period, which is far larger than the clause filter and still orders of magnitude under 60 million.
  5. Deduplicate the rehydration, not the evaluation. Distinct evidence_snapshot_hash values are the unit of store work, and a single snapshot attached to several cases in a household or an appeal chain is fetched once and evaluated per determination. Order the candidate scan by hash so the cache hits are adjacent rather than scattered across the run.
  6. Make the run resumable and rate-limited by construction: a durable cursor over the candidate set, a token bucket sized to the share of the 500 per second ceiling you are permitted, and writes that rely on the uniqueness key rather than a read-then-insert, so a restart after a crash converges instead of producing a second current determination.

Follow-up

  • The correction changes an amount rather than an outcome. Does your candidate filter change, and does the notice obligation?
  • A replayed determination comes back identical. Do you write anything at all, and what does an auditor need to see that you checked it?
  • Halfway through the run the regulator issues a correction to the correction. What happens to the work already done?

Surface the fifty most SLA-at-risk cases per unit

Medium · top-k · heap · index-selectivity

A queue view must return the 50 cases closest to breaching their service standard for one assigned_unit_id. Candidate rows come from case: case_id, jurisdiction_code, sla_due_on DATE, submitted_at, status, assigned_unit_id. Four million cases are open across 200 jurisdictions. Risk orders by business days remaining to sla_due_on ascending, then submitted_at ascending, then case_id ascending; already-breached cases sort first and must not clamp at zero. Business days come from a per-jurisdiction holiday calendar. Give the algorithm, the complexity, and where the work must happen.

Approach

  1. Keep a bounded max-heap of size k=50 ordered by the full comparison key and stream the candidates: push, and when the heap exceeds k pop the maximum. What survives is the k smallest. O(n log k) time and O(k) space, against O(n log n) time and O(n) space for sorting the unit's whole backlog.
  2. Narrow before you compute. Business days remaining is an expression over a calendar, so no index can order by it, and a naive ORDER BY on that expression sorts every candidate row. Within one jurisdiction the business-day remainder is monotone in sla_due_on, so an index range on (assigned_unit_id, sla_due_on) with a horizon bound gives a correct prefix cheaply; compute the calendar arithmetic only on that narrowed set.
  3. Do not carry the monotonicity across jurisdictions: two jurisdictions with different holiday lists order differently for the same sla_due_on. Run the k-selection per jurisdiction and merge the per-jurisdiction heaps, or pull the narrowed union and heap it in one pass, but do not assume one global sla_due_on order is the risk order.
  4. Make the key a total order. Ties on the first two components are common because overnight batch intake stamps thousands of submitted_at values inside the same second, and without case_id as the final component two replicas computing the same list return different pages. Determinism here is a correctness property of the page, not a nicety.
  5. Pin the as-of instant for the whole computation and return it with the list. Business days remaining changes at a civil midnight per jurisdiction, so a list computed across that boundary would otherwise mix two answers and be unreproducible when someone asks why a case ranked where it did.
  6. The heap ranks; it does not assign. Handing a case to a worker is a conditional UPDATE guarded on row_version, and a zero-row result means another worker took it, so the caller drops to the next candidate rather than retrying the same row.

Follow-up

  • Two workers open the queue in the same second and both see case X first. What happens, and whose problem is it?
  • A jurisdiction's legislature adds a holiday with retroactive effect. Which cached sla_due_on values are now wrong, and how do you find them?
  • The product asks for page two of this list. What does that cost with a size-k heap, and what would you build instead?

Measure time in status from an append-only case event log

Medium · window functions · event log · frame semantics · backdating

case_event(case_event_id, case_id, reopen_cycle, seq BIGINT, event_type TEXT, occurred_at TIMESTAMPTZ, recorded_at TIMESTAMPTZ) is append-only with UNIQUE (case_id, seq). Staff backdate occurred_at; recorded_at is when the row landed. A case can be reopened, so submitted and determined repeat. Write a query returning one row per decision episode with the interval from its submitted to its determined, plus a flag for episodes still open. State which column you order the window by and why the other one gives wrong answers, and say what your query returns for a cycle whose determined carries an occurred_at earlier than its submitted.

Approach

  1. Partition on the real grain, (case_id, reopen_cycle). With case_id alone, an open cycle's submitted takes the next cycle's submitted as its LEAD, so a cycle that never closed is reported closed with an interval spanning a cycle boundary. Keep LEAD(event_type) in the output so that is detectable rather than assumed.
  2. Order by seq, not occurred_at. seq is unique per case by constraint, so the ordering is total and the result is deterministic. occurred_at is asserted by staff: a determined backdated below its own submitted swaps the two rows, the submitted row becomes last in its partition, LEAD returns NULL, and a cycle that was decided is reported still open — it leaves the completed-cycle statistics entirely rather than showing up as a bad number.
  3. Use LEAD over the filtered event set: SELECT case_id, reopen_cycle, occurred_at AS submitted_at, LEAD(occurred_at) OVER w AS determined_at, LEAD(event_type) OVER w AS next_type FROM case_event WHERE event_type IN ('submitted','determined') WINDOW w AS (PARTITION BY case_id, reopen_cycle ORDER BY seq), then keep the rows where event_type = 'submitted'. WHERE is evaluated before window functions, so filtering to two event types first is both correct and cheap.
  4. Separate pairing from measurement, because seq only fixes the first. Ordering by seq guarantees you paired the right two rows; it cannot repair a timestamp a human asserted, so a backdated cycle still yields determined_at - submitted_at below zero. Emit it as computed alongside (determined_at < submitted_at) AS asserted_interval_negative. Do not clamp with GREATEST(..., interval '0') and do not swap the endpoints: the negative value is the only evidence the entry is wrong, and the flag is what keeps it out of the published average without hiding it from the people who can correct it.
  5. Carry the system-time measure beside the asserted one: recorded_at - LEAD(recorded_at) inverted, i.e. LEAD(recorded_at) OVER w - recorded_at. Precondition for calling it non-negative: recorded_at is assigned by the database at insert and appends for one case are serialised — a per-case seq issued under the case row lock gives you both properties at once. If a case can be appended to concurrently, recorded_at can invert against seq too and the flag must cover both measures.
  6. Handle the open episode as data, not as a default: when LEAD returns NULL there is no determined yet, so emit determined_at IS NULL and a still_open flag. Coalescing to now() silently converts a censored observation into a completed one and drags every average down.
  7. Be explicit about frames. LAG and LEAD are not frame-sensitive — they address the partition directly — but the moment you add a running aggregate, the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, where CURRENT ROW in RANGE mode means the last peer. Ordering on a tie-prone timestamp therefore makes peers of distinct events and sums them all at once; ordering on unique seq makes RANGE and ROWS coincide.

Follow-up

  • The published service standard is in business days in the case's jurisdiction. What does this query have to become, and where does the holiday calendar live?
  • Two determined events exist in one cycle because a correction was recorded. Which one does your query pair, and which one should it pair?
  • Rewrite the still-open flag so a cycle with withdrawn between submitted and determined is excluded rather than counted as open.

A quarterly payout total that triples through two joins

Medium · join fan-out · aggregation grain · minor units · reversals

A quarterly report sums disbursed amounts per program. It joins case to determination (one current row per reopen cycle), determination to disbursement_instruction (one row per benefit period, plus compensating reversal rows carrying reversal_of_instruction_id), and case to case_event to count transitions. Totals come out roughly triple the payment rail's settled figure. Write the corrected query returning program_code, settled_minor and transition_count, and explain the arithmetic that broke. Amounts are integer minor units; do not reach for SUM(DISTINCT ...).

Approach

  1. Say the grain out loud for every table before writing a line: case is one row per case, determination is one row per (case, reopen_cycle) once filtered to is_current, disbursement_instruction is one row per (determination, benefit period), case_event is one row per transition. Joining a one-row-per-case anchor to two independent one-to-many children multiplies them: a case with 3 instructions and 12 events yields 36 rows, and every amount is counted twelve times.
  2. Note the second fan-out most candidates miss: is_current is unique per (case_id, reopen_cycle), not per case, so a case reopened twice contributes three current determinations and multiplies the event branch again.
  3. Reduce each branch to a common grain in its own CTE before joining. Aggregate instructions up to case_id through determination, aggregate case_event to case_id, then LEFT JOIN both onto case — now each side contributes at most one row per case and the products cannot form.
  4. Net the reversals arithmetically rather than by filtering: amount_minor carries CHECK (amount_minor > 0), so a reversal is a positive row whose sign comes from its role — SUM(CASE WHEN reversal_of_instruction_id IS NULL THEN amount_minor ELSE -amount_minor END). Filtering reversals out instead leaves the original settlement in the total and overstates it.
  5. Reject SUM(DISTINCT amount_minor) explicitly: two genuinely separate benefit periods that happen to pay the same amount collapse into one, so it fixes the symptom by deleting real money. COUNT(DISTINCT case_event_id) is legitimate because ids are unique; SUM(DISTINCT) over a non-unique measure is almost never correct.
  6. Keep every total in BIGINT minor units through the whole pipeline and divide only at render. A cast to float or a language-side NUMERIC rounded by the default rule reintroduces exactly the error the integer representation was chosen to avoid.

Worked solution (30 min)

  1. Build a fixture: one case with 3 settled instructions and 12 events, one case reopened twice, one case with a settled instruction and a later reversal, one case with events but no payment.
  2. Run the naive three-way join and record the inflated total, then compute by hand the multiplier for each fixture case to confirm the arithmetic matches the inflation.
  3. Rewrite with one CTE per child aggregated to case_id, LEFT JOIN both, and re-run.
  4. Add the reversal sign expression and confirm the reversed case nets to zero rather than to twice its amount or to nothing at all.
  5. Assert the result against a directly computed control total taken from disbursement_instruction alone with no joins.

Expected result: settled_minor per program equals the control total computed from disbursement_instruction alone, with reversals netted; transition_count equals a direct COUNT(*) on case_event for the same window. The reopened case contributes its payments once, not three times, and the payment-free case appears with zero rather than being dropped.

Verify it yourself

  • The query's settled_minor matches SUM over the instruction table computed with no joins at all — the only control total that cannot fan out.
  • Removing the reversal sign expression changes the total by exactly twice the reversed amount, confirming the expression is doing what you claim.
  • Row counts of each CTE equal the distinct case count they cover, which is the direct proof that the grain is one row per case.
  • The result type stays BIGINT end to end; no intermediate is a float or a rounded decimal.

Follow-up

  • A reversal arrives in the following quarter for a benefit period inside this one. Which quarter's total moves, and how do you present a figure that changes after the books were shown to someone?
  • Rewrite the instruction branch as a LATERAL subquery instead of a CTE. Does the plan change, and does the correctness argument?
  • The report must also show cases with no payments at all. Which join becomes wrong if you move a predicate from the CTE's WHERE into the outer WHERE?

How do you approach debugging an application issue in a production environment?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How do you ensure your code is accessible and compliant with standards?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Work from the requirement backwards to the design.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What is the use of the div tag?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Clarify what is being asked and what a complete answer contains.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Explain the difference between client-side and server-side processing.

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Redesign a case transition endpoint that double-advances on retry

Medium · idempotency · optimistic concurrency · state machines

POST /cases/{case_id}/transitions with body {"to":"determined"} sits behind a gateway that retries once on 504. The Case Service is the sole writer of case.status and appends to case_event under UNIQUE (case_id, seq). A caseworker's determine action timed out, the gateway retried, and the case ended with two events, two determinations and a wrong reopen_cycle. The caller is the staff UI, which on failure shows a retry button. Redesign the request and response so a retry converges on one effect. Deliver the new contract, the concurrency control, and what the UI does on each status code.

Approach

  1. Name the defect: a POST whose effect is identified by the server is not idempotent, and a gateway cannot distinguish a lost response from a lost request. Every fix either moves the identity of the intended effect into the request or makes the write conditional on the state the caller read.
  2. Make the transition conditional. The UI already read the case, so send If-Match with an ETag derived from row_version and execute UPDATE ... WHERE case_id = $1 AND row_version = $2. Zero rows affected is a conflict, not a success: return 412 with the current representation. A request with no If-Match gets 428 (RFC 6585) rather than an unconditional write.
  3. Give the append an effect identity: the client generates transition_id once per user action and reuses it across retries, and the server holds UNIQUE (case_id, transition_id). A replay raises SQLSTATE 23505, the handler re-reads and returns the original 200 with the same case_event_id. The business fact here is the click, so the UI must not mint a fresh id on the retry path.
  4. Close the determination side in the database: the partial unique index UNIQUE (case_id, reopen_cycle) WHERE is_current makes the second writer fail instead of producing two current determinations. Under READ COMMITTED the check-then-insert both writers pass is write skew, and Postgres REPEATABLE READ is snapshot isolation and still permits it — so it is the index, or SERIALIZABLE with a retry on SQLSTATE 40001.
  5. Make the response decidable: return the full case representation with status, row_version, reopen_cycle and the resulting determination_id, so the UI can distinguish already-applied-by-me from applied-by-someone-else without a second call.
  6. Specify the UI's branches: on 412 re-read and re-render rather than offering retry, because the transition the worker intended may no longer be legal from the current status; on a 5xx, re-send the same body with the same transition_id.

Worked solution (30 min)

  1. Add UNIQUE (case_id, transition_id) on case_event and the partial unique index on determination.
  2. Implement the handler as a conditional UPDATE plus the event and determination inserts in one transaction, mapping 23505 to a re-read and a replay response.
  3. Send the same request twice concurrently with one transition_id and assert one created response, one replay, one case_event row and one determination.
  4. Change row_version between the two sends and assert 412 carrying the current status.
  5. Simulate the gateway: send, drop the response, re-send after the state has advanced, and assert no third event appears.

Expected result: One transition_id always yields exactly one case_event and one determination under every interleaving; a stale If-Match produces 412 with the current status and row_version; no code path performs an unconditional status UPDATE.

Verify it yourself

  • Dropping the partial unique index makes the concurrency test produce two is_current rows, proving the invariant is enforced by the database.
  • A replayed request returns the original case_event_id rather than a new one.
  • The UI's failure branch re-reads on 412 and only re-sends unchanged bodies on 5xx.

Follow-up

  • The gateway retries thirty seconds later, after the worker has already reopened the case. What does your contract return, and is that the same answer as an immediate retry?
  • Where does seq come from, and what happens when two writers compute the same value?
  • Would PUT /cases/{id}/status be a better shape? What does it break for reopen cycles and for the event log?

Bulk replay is OOM-killed at hour six, intake never leaks

Hard · memory leak · gc · profiling

After a rule correction, a bulk replay re-evaluates 4 million historical determinations at about 8,000 per minute. Resident memory climbs monotonically, GC time goes from 2% to 35%, and the worker is OOM-killed near hour six; a restart resumes and dies further along. The same build serves intake at 3,000 evaluations per minute for weeks with flat memory. The evaluator memoises results in a map keyed by (rule_version_id, evidence_snapshot_hash), whose value holds the outcome and the clause-level trace. Deliver an ordered checklist that separates retention from allocator behaviour, and the fix.

Approach

  1. Distinguish a leak from a high-water mark before profiling anything: compare the live set after a forced full collection at hour one, two and three, not resident memory. A live set that grows monotonically is retention; resident memory growing while the live set is flat is allocator or arena behaviour, and the two have entirely different fixes.
  2. Take two heap snapshots an hour apart and diff by retained size, then walk the dominator tree to name the retainer. Ranking by allocation count is the common wrong move — the heaviest allocator is usually short-lived garbage, and the retainer is often a modest number of objects each holding a large graph.
  3. Reason about workload shape before reading another line of code, because it is one counter: memoisation keyed on an evidence hash has a useful hit rate at intake, where a few thousand snapshots recur across a day, and a hit rate near zero in replay, where all 4 million hashes are distinct by construction. Measuring the hit rate converts this from an unexplained leak into an unbounded cache that is buying nothing.
  4. Confirm the per-entry cost so the arithmetic matches the timeline: the key is a 32-byte hash plus an integer, but a value holding the clause-level trace and a reference to the evidence snapshot retains kilobytes. At 8,000 per minute, hour six is about 2.9 million entries; at roughly 2 kB retained each that is near 6 GB, which is where the ceiling is.
  5. Fix so both modes behave the same: bound the cache with a size-capped eviction policy rather than switching it off by a mode flag, since a flag only moves the cliff to whoever forgets to set it. Then also disable memoisation on the replay path where the measured hit rate does not pay for the retention, and stream results to the sink instead of accumulating them in a result list.
  6. Re-run and verify with the same instrument that diagnosed it: post-collection live set flat across two hours, GC time back near 2%, and throughput per minute unchanged or better because the allocator is no longer being pressured.

Follow-up

  • Replay must not create a second determination for a case. What uniqueness key makes a re-run converge onto the existing row?
  • Restarting resumes and dies further along. How would you make the replay checkpointed so hour six does not cost hours one through five?
  • Intake and replay share a build. What keeps a future optimisation tuned for intake from re-creating this?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

What the behavioural round is actually measuring

Every story you tell gets read for blast radius and judgement: what could have broken, who else it touched, what you knew at the moment you decided. Nobody can audit your code in an hour, so they audit your reasoning instead. Pick work where the call was genuinely yours and the consequences were real enough to remember.

Reverse a decision to write audit rows asynchronously

Hard · auditability · transactional outbox · reversal · data-access boundary

You chose to write audit_access_log rows through a fire-and-forget asynchronous call so a read path would stay inside its latency budget. Months later the rows are missing in exactly the windows an investigator cares about. The table is audit_access_log(audit_id, occurred_at, actor_id, actor_role, delegation_id, subject_type, subject_id, action, purpose_code, source, client_ip, request_id, retained_until). Describe a reversal of this shape that you owned: what made the original choice defensible, the evidence that changed your mind, what replaced it, and what the migration cost.

Approach

  1. State the original reasoning without apologising for it. The read path had a budget, the audit write is not on the user's critical path, and asynchronous is the textbook answer. A reversal narrative that opens with 'it was obviously wrong' tells the interviewer you never understood why anyone chose it, which makes the reversal luck rather than judgement.
  2. Give the evidence in the form that actually moved people: not 'some rows are missing' but a measured drop ratio against a known denominator, counting audited reads at the data-access boundary versus rows landed, bucketed by hour, showing loss concentrated in the highest-load hours. Those are the hours an investigation will later want to reconstruct, which is what makes the loss non-random rather than merely unfortunate.
  3. List the bypasses separately from the drops, because they need different fixes: a read replica, a report extract, a cache hit that never touches the table, and a search that returned the record, since a search hit is an access even if nobody opened the result. Some of those can only be closed below the application, in a policy or a view, rather than in a controller.
  4. Choose between same-transaction and an outbox deliberately and price both. Writing the audit row inside the read's transaction turns every audited read into a write transaction, which rules out serving it from a read replica at all and couples read availability to the audit table. The outbox keeps the read path fast but is only trustworthy with a liveness alarm on the age of the oldest unpublished row, not on whether the publisher process is up.
  5. Cost the migration honestly. The gap in the historical record cannot be backfilled from anything, so part of the reversal is documenting that period as incomplete and saying so out loud, rather than quietly starting to be correct from a Tuesday.

Follow-up

  • The outbox publisher is wedged for six hours. What does the read path do in the meantime, and who is told, and when?
  • How do you prove to an auditor that capture cannot be bypassed, instead of asserting that it is not?
  • purpose_code is an enum precisely so it is auditable. What stops staff from always selecting whichever value sorts first?

Explain a denied determination to an auditor and a caseworker

Easy · communication · provenance · appeals · evidence

A determination denied an application and the constituent has appealed fourteen months later. An auditor wants to know which rules were applied and why; a caseworker wants to know what to tell the constituent this afternoon. You have the determination row (rule_version_id, evidence_snapshot_hash, ordered reason_codes, clause trace, decided_by), the case_event history and the per-channel notice dispatch record. Write both explanations. State what you will not assert, and what you would check before asserting it.

Approach

  1. Answer the two audiences separately, because they need different objects. The caseworker needs the ordered reason codes turned into what the constituent can do next; the auditor needs provenance: which rule_version governed, which clauses fired, which evidence snapshot fed them, and who or what decided.
  2. Lead the auditor's version with the fact that decided_by being NULL means the determination was fully automated, which is a materially different fact from a human decision and is the first thing they will ask about. Papering over it costs you the rest of the conversation.
  3. Separate what the record proves from what you are inferring. The record proves which clauses fired against which evidence hash. It does not prove the constituent received the notice, because dispatch is recorded per channel and dispatch is not receipt.
  4. Distinguish procedural reason codes from substantive ones, since they lead to opposite advice: evidence not returned is curable by sending a document, while a determination over a threshold is not, and the caseworker will act on whichever you imply.
  5. Say what you would verify before asserting any of it: replay the determination against its own rule_version_id and evidence_snapshot_hash and confirm it reproduces byte for byte, and report a mismatch as a different question rather than as an answer.

Follow-up

  • The replay does not reproduce the original outcome. What do you do first, and what do you tell the auditor in the meantime?
  • The caseworker asks you to describe it as a system error so the conversation is easier. How do you handle that?
  • The rendered notice is itself evidence. What does that imply about template versions and fonts?

Unblock an engineer whose replay creates duplicate determinations

Easy · mentoring · idempotency · replay · constraints

An engineer's bulk replay job, re-running determinations after a rule correction, has left some cases with two rows flagged current and has re-issued notices for cases whose outcome did not change at all. They have been on it for two days and are now proposing a cleanup script to delete the extra rows. Describe unblocking someone in this shape: how you found where their model was wrong, what you had them do first, and how you left them able to handle the same class of bug without you.

Approach

  1. Get the invariant said out loud before anyone touches code. 'A replay that changes nothing writes nothing' is usually the sentence they have not yet said, and saying it reframes the problem from a concurrency puzzle into a missing no-op check, which is a different and much smaller fix.
  2. Separate the two failures they have merged into one bug. Duplicate current rows are a concurrency defect the database should have rejected; re-noticing unchanged cases is a logic defect in the job. A cleanup script addresses neither, and running it before the constraint exists guarantees running it again next week.
  3. Have them reproduce the smaller failure themselves — two concurrent inserts against a table with no partial unique index — rather than describing it. A reproduction they built transfers to the next bug; a diagnosis handed to them does not.
  4. Be explicit about what you deliberately did not do: you did not take the keyboard, and you did not let the cleanup script go first. Say how you decided that, because with an incident running the wrong answer is to teach instead of to stop the bleeding, and the judgement call is the interesting part.
  5. Close the loop on repeatability for them: a no-op predicate compared before writing, a dry-run mode whose output is counts by reason code, and agreement on who reviews the next replay before it runs.

Follow-up

  • Notices already went to constituents whose determination did not change. What are they told, and who writes it?
  • How do you tell whether the duplicate rows came from concurrency or simply from the job being run twice?
  • What changes in your approach if an auditor found this rather than the engineer?