Interview guides · Banner Health

Banner Health Software Engineer Interview Questions & Guide 2026

A prep plan for Banner Health software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

As a Software Engineer at Banner Health, you play a pivotal role in maintaining and advancing the digital infrastructure that supports one of the largest non-profit health systems in the United States. Your work directly impacts the efficiency of clinical operations and, ultimately, the quality of care provided to patients. By building, maintaining, and optimizing the systems that clinicians and administrative staff rely on every day, you contribute to a mission-driven environment where technology is a critical enabler of health outcomes.

If the seat owns a service boundary, scope your preparation toward failure behaviour rather than topology. Retrying over an at-least-once channel produces duplicates by construction, so a retry policy is only as safe as the idempotency key underneath it.

Focus areas

Design person merges that remain reversible afterwards
Hold accumulator limits under concurrent claim adjudication
Make HL7 and X12 ingestion idempotent under replay

Overview

This role is both challenging and rewarding due to the scale and complexity of the healthcare ecosystem. You will likely collaborate with cross-functional teams, including product owners, system architects, and clinical stakeholders, to solve real-world problems. Whether you are developing internal tools, integrating complex data sets, or ensuring system reliability, you are expected to be a problem-solver who can balance technical excellence with the practical needs of a fast-paced medical environment.

The Banner Health interview process

Banner Health candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Interviews with Hiring Managers

The design portion here is shorter and lower-stakes than a dedicated design round, which changes what it tests. There is rarely time to reach a full component diagram, so what gets read is your first two minutes: whether you pin down constraints, meaning request rate, data size, what must not be lost and how stale a read may be, before naming any technology. Opening with a stack list invites being steered back. Once the numbers are on the table, say what breaks first if they grow tenfold, and defend the plain option where the load does not justify more.

What to demonstrate

Whether constraints come before components: peak request rate, data volume, what must survive a process dying, and the staleness the product can tolerate
Whether you can name what saturates first when traffic grows by an order of magnitude, and whether that matches the design you just sketched
Whether a cache is reasoned about on both paths, since a cold or recently flushed cache sends the full request rate to the origin, so capacity has to cover the miss case and not only the steady state
Whether you distinguish what you have operated from what you have only read about, which usually shows up in the answer to why a particular component is there

How to prepare

Take one system you worked on and write down the numbers you would need to defend it: requests per second at peak, rows in the largest table, the latency you were actually held to. Not having them is the common stall in this part.
Practise the tenfold question on that system out loud, naming the first bottleneck you would hit, whether that is a single writer, connection limits, disk, or a queue that grows faster than it drains, and the smallest change that buys headroom.
Prepare one decision where the plain option was correct: the cache you did not add or the queue you did not introduce, with the load figure that made that the right call. Being able to argue for less is rarer than being able to argue for more.

Development Team Interview

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Writing the access audit record inside the read transaction, or firing it off after the response with no durability.

Inside the transaction, an audit-store outage blocks clinical reads and turns a logging dependency into a care outage. Fire-and-forget afterwards means the audit trail is incomplete during precisely the incidents it exists to reconstruct, and the gaps are invisible until someone asks for the log. The usual resolution is committing the access decision and its audit row together to a local outbox and shipping asynchronously, which keeps the read path available while preserving durability.

Upserting on the order or result identifier, so a correction overwrites the original row.

It makes the question 'what did the clinician see at 14:02' unanswerable, which is exactly what an incident review or a legal hold asks. It also leaves downstream consumers that already acted on the preliminary value with no correction event to react to, because the state transition was collapsed into a single mutated row and never emitted.

Quoting amortised or average cost as if it were a worst-case guarantee

Appending to a dynamic array is amortised O(1), but the append that triggers a resize copies every element, and hash lookup is constant only while the hash spreads the actual keys. Say which guarantee you are offering when the caller cares about the latency of one call rather than the total over many.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Questions & practice

Merge twelve resource streams into one patient summary page

Medium · k-way merge · heap · pagination

A patient summary fans out to between 8 and 12 resource types. Each returns a network-backed, paged iterator of resource versions sorted by issued_ts descending, up to 200,000 versions per type for one person. Return the 50 most recent current versions across all types, where current means no later version supersedes it within the same logical resource, and a logical resource whose current version is entered_in_error is omitted entirely. You may not materialise the iterators. Give time and space in terms of k types, the result size and the page size.

Approach

k-way merge with a max-heap holding one head per iterator, keyed on issued_ts. Seeding is O(k), each pop is O(log k), so reaching R emitted rows costs O(k + P log k) for P pops, with O(k) heap space plus one page buffered per iterator. Fetching everything and sorting is O(V log V) over V up to 2.4 million versions and drags every page across the network to produce 50 rows.
Suppress with a hash set of logical resource ids already seen, recorded on first sight whether or not that version is emitted. A logical resource has exactly one resource type, so all of its versions arrive on one iterator, and that iterator is descending in issued_ts: the first version you see for a logical resource is its newest. Deciding on first sight and suppressing every later pop for that id is therefore correct in one pass with no lookahead.
Count emits, not pops. A correction-heavy chart can burn many pops per emitted row, so a loop that stops at 50 pops returns a short page. Put a bound on total pops as well, and when it trips, return what you have with a continuation token rather than spinning.
Break issued_ts ties deterministically on (resource type, resource id). Without it two identical requests return two different orderings and the next page silently skips or repeats rows.
Handle entered_in_error at first sight: the erroneous version still supersedes its predecessor, so record the id in the seen set and emit nothing, dropping the whole logical resource instead of falling back to the value it replaced. Recording it is the load-bearing half. Skip it and the next pop re-displays the value a clinician already retracted.
Summarise the budget honestly: the composite p99 is what the user feels, and it is bounded below by the slowest of the k iterators, so the merge fixes the ordering cost but not the fan-out tail.

Follow-up

One of the twelve iterators has a p99 of 400ms while the rest return in 20ms. What is your composite p99 and what would you change first?
The user pages to rows 51 through 100. How do you resume without re-reading from the top, and what breaks if issued_ts is not unique?
One resource type is accidentally returning ascending order. How would your code detect that rather than quietly emitting the oldest rows?

Sum surviving claim versions in one pass over unordered lines

Easy · hash map · versioning · streaming aggregation

You are streamed up to 50 million claim_line records in arbitrary order: claim_id, claim_version, line_number, frequency_code (original, replacement, void), enterprise_person_id, allowed_amount_cents. Every version of a claim shares its claim_id, and a replacement arrives as a higher claim_version. Return total allowed_amount_cents per enterprise_person_id, counting each claim once at its highest version and contributing zero when that version is a void. Amounts are non-negative int64 minor units. Target O(N) time and O(C) space for C distinct claims, single pass, no sort.

Approach

Say the two grains out loud before writing anything: the version lives at claim grain, the money lives at line grain. Every wrong answer here comes from applying a claim-level rule with a line-level filter.
Keep one hash map from claim_id to a small record (best_version, sum_cents, person_id, is_void). Per line: if claim_version is greater than best_version, reset sum_cents to this line's amount and overwrite the person and void flag; if equal, add to it; if lower, discard the line. That reset is what makes the pass order-independent, so a replacement arriving before its original still wins.
Resolve the void at the end, not at ingest. A void only nullifies the claim if it is the surviving version, and zeroing on sight would let an earlier replacement that arrives later resurrect the money.
Fold the C claim records into a person-keyed map in a second phase, O(C). Do not accumulate into the person total during the stream: you cannot subtract a superseded version you have already forgotten.
Cost is O(N) time, O(C) space. The sort-based alternative, group by (claim_id, claim_version) and keep the max, is O(N log N) and needs the set resident; prefer it only when C approaches N and the map will not fit.
Keep cents in int64 throughout. Fifty million lines at a realistic per-line ceiling stay four orders of magnitude below 2^63, so the integer type costs nothing and removes the float drift class entirely.

Follow-up

You shard the stream across eight workers. Sharding by hash of claim_id works; what exactly breaks if you shard by enterprise_person_id instead?
A replacement arrives for a claim_id whose original never appears in the batch. What does your map produce, and is that the financially correct answer or a reconciliation break?
The same claim_id is reused by two different submitting organisations. How does the key have to change, and when would you have noticed?

Flatten overlapping coverage spans into a primary-payer timeline

Medium · sweep line · intervals · bitemporal

For one enterprise_person_id you hold up to 10,000 coverage_span rows: coverage_id, payer_id, plan_id, effective_date, termination_date (null means open-ended), coverage_order (1 is primary), status, valid_from and valid_to (system time). Given a system instant T and a date range D0 to D1, return disjoint business-date intervals covering that range, each labelled with the active coverage_id of lowest coverage_order, and with uncovered stretches returned explicitly. termination_date is the last covered day. Only status active counts. Target O(N log N) time, O(N) space.

Approach

Apply the system-time slice before any interval logic: keep rows where valid_from is at or before T and valid_to is after T. That single predicate is what makes the answer 'what we believed at T' rather than 'what we believe now', and a retroactive termination loaded after T leaks straight into the output if you skip it.
Convert each surviving row to a half-open day interval from effective_date up to termination_date plus one day, using positive infinity for a null termination. Half-open removes every off-by-one at the join between a termination and the next plan's effective date, which is where same-day switches get turned into false gaps.
Sweep: emit 2N boundary events, sort by date in O(N log N), and hold the active set in a min-heap keyed by (coverage_order, coverage_id) with lazy deletion, popping the top while it has already expired at the current boundary. Each coverage is pushed once and popped once, so the sweep is O(N log N) total and O(N) space.
Emit an output interval whenever the heap top changes between consecutive boundaries, and emit an explicit uncovered interval when the heap empties. A gap reported as a gap is a different answer from a gap omitted, and downstream the difference is a denial versus a silent assumption of coverage.
Fix the tie-break and state it: lowest coverage_order, then lowest coverage_id. Two rows at order 1 is a data error, and without a deterministic rule the coordination-of-benefits answer changes between two runs over identical data.
Clip to D0 and D1 last, so a coverage that starts before D0 still contributes its correct order inside the window.

Worked solution (30 min)

Filter to rows live at T, then map each to a half-open interval and a (coverage_order, coverage_id) priority.
Build the boundary list, sort it, and sweep with a lazily-deleted min-heap.
At each boundary, drop expired heap tops and compare the new top to the previous one, emitting an interval on change.
Emit an uncovered interval whenever the heap is empty between two boundaries.
Clip the emitted list to D0 through D1 and assert the intervals are disjoint and contiguous.

Expected result: Window 2026-01-01 to 2026-12-31 with three live rows: coverage 1 at order 1 effective 01-01 terminating 06-30, coverage 2 at order 2 effective 03-01 open-ended, coverage 3 at order 1 effective 09-01 open-ended. Output is three intervals: 01-01 to 06-30 labelled coverage 1, 07-01 to 08-31 labelled coverage 2, 09-01 to 12-31 labelled coverage 3, with no gaps. Delete coverage 2 and the middle interval becomes an explicit uncovered stretch of 07-01 to 08-31.

Verify it yourself

The emitted intervals are sorted, pairwise disjoint, and their union is exactly D0 through D1 once uncovered stretches are included.
Every boundary date in the output is D0, the day after D1, some effective_date, or some termination_date plus one day. Any other date means a boundary was synthesised.
Removing the lowest-order coverage promotes exactly the next order and leaves every boundary it did not touch unchanged.
Choosing T before the load time of a retroactive termination returns the pre-correction timeline, and T after it returns the corrected one, from the same table.

Follow-up

Run the same query at two system instants and diff the timelines. What did the correction change, and what does each extra instant cost you?
A termination arrives with an effective date earlier than the service date of a claim that already adjudicated and paid. What does the timeline now say, and what has to happen to that claim?
Two enrolment files disagree on coverage_order for a dependent. What does your tie-break do, and what should the system do instead of tie-breaking?

Namespace source identifiers so a cross-facility join cannot collide

Easy · identity · composite keys · constraints

person_identity_link holds one row per (assigning_authority, source_person_id) per link version: link_id, enterprise_person_id, assigning_authority, source_person_id, match_score, link_status in ('auto_linked','manual_linked','potential_duplicate','unlinked','rejected'), version, superseded_by_link_id, decided_by, decided_at. Two facilities in one network both issued medical record number 004821, to different people. An existing extract joins on source_person_id alone and has been merging their charts. Give the constraint set that makes the bare join impossible to write by accident, then write the query that resolves one facility's (authority, MRN) to its current enterprise_person_id.

Approach

Name the root cause precisely: an MRN is unique only inside the authority that issued it, and a member ID only inside payer plus plan. The identifier is half a key; the namespace is the other half.
Make the composite the only addressable key. UNIQUE (assigning_authority, source_person_id, version) gives version history; a partial unique index on (assigning_authority, source_person_id) WHERE superseded_by_link_id IS NULL enforces one live decision per source record.
Remove the bare column as a join target from every consuming view: expose a view or function that takes both arguments, so a one-argument lookup is a compile-time error rather than a silent cross-patient merge.
Write the resolution query against the live version only, and filter link_status separately — 'unlinked' and 'rejected' are live rows that mean 'no enterprise identity', so they must not be confused with 'no row'.
State the detection query you would run before trusting any existing extract: group source_person_id across authorities and count distinct authorities, which surfaces every collision already in the data.

Worked solution (20 min)

Write the two constraints and say which invariant each one holds: the three-column UNIQUE holds version history, the partial unique index holds single-live-decision.
Write the resolution query filtering on superseded_by_link_id IS NULL plus the two linked statuses.
Write the collision detector over the existing table and run it mentally against the 004821 case to confirm it returns that value with two authorities.
Decide the contract for zero rows versus a live 'unlinked' row: zero rows means the source record was never seen, 'unlinked' means it was seen and deliberately not attached, and the caller must handle them differently.
Write the one-line fix for the broken extract: add the authority column to the extract and to the join, then re-run and diff the row count against the old output.

Expected result: Constraints: UNIQUE (assigning_authority, source_person_id, version) and CREATE UNIQUE INDEX ON person_identity_link (assigning_authority, source_person_id) WHERE superseded_by_link_id IS NULL. Resolution: SELECT l.enterprise_person_id FROM person_identity_link l WHERE l.assigning_authority = :aa AND l.source_person_id = :mrn AND l.superseded_by_link_id IS NULL AND l.link_status IN ('auto_linked','manual_linked'); zero or one row by construction. Detector: SELECT source_person_id, count(DISTINCT assigning_authority) AS authorities, array_agg(DISTINCT assigning_authority) FROM person_identity_link GROUP BY 1 HAVING count(DISTINCT assigning_authority) > 1;

Verify it yourself

The detector returns 004821 with two authorities, confirming the collision is in the data and not a theory.
Inserting a second live row for one (authority, MRN) raises a unique violation, while inserting a superseded row for the same pair succeeds.
The resolution query returns at most one row for every (authority, MRN) pair in the table — run it as a LATERAL over the distinct pairs and assert no count exceeds one.
Re-running the repaired extract yields a strictly different enterprise_person_id for at least one of the two 004821 patients.

Follow-up

The spine stores enterprise_person_id directly on encounter and claim_line. What does a wrong merge cost you with that design, and what would you store instead to keep unmerge a supported operation?
A link is downgraded from auto_linked to potential_duplicate. Should the resolution query return the old enterprise identity, nothing, or an error, and who decides?
Your partial unique index blocks a second live row. How does the merge path insert the new version and supersede the old one without violating it mid-transaction?

Return the displayable version of each analyte with a window function

Medium · window functions · versioned rows · indexing

observation_result is append-only: observation_id, enterprise_person_id, encounter_id, accession_id, placer_order_id, filler_order_id, loinc_code, value_numeric, unit_ucum, result_status in ('registered','preliminary','final','amended','corrected','entered_in_error'), collected_ts, issued_ts, version, supersedes_observation_id. A chart view needs the currently displayable value for each (accession_id, loinc_code) for one person over the last 90 days, excluding analytes whose newest version is entered_in_error but keeping analytes whose older versions were. Write the query, give the index that serves it, and say honestly which part of the plan the index cannot remove.

Approach

Recognise the shape: this is a top-1-per-group over a version chain, not a filter. Pick the newest version first, then apply the status predicate to the winner — the order is the whole exercise.
Use DISTINCT ON (accession_id, loinc_code) with ORDER BY accession_id, loinc_code, version DESC, observation_id DESC on PostgreSQL; the ORDER BY must start with the DISTINCT ON expressions or it is a syntax error. Use ROW_NUMBER() OVER (PARTITION BY ... ORDER BY version DESC, observation_id DESC) with an outer WHERE rn = 1 if you need portability, since a window function cannot be referenced in the WHERE of its own query level.
Push only person and time into the inner WHERE. Pushing result_status <> 'entered_in_error' inside promotes a retracted result's predecessor back onto the chart, which is the bug the prompt is testing for.
Index (enterprise_person_id, collected_ts DESC) so the 90-day slice is an index range scan rather than a scan of the person's whole history.
Be straight about the limit: because collected_ts carries a range predicate, no b-tree index can also deliver rows pre-ordered by (accession_id, loinc_code, version DESC), so the sort stays in the plan. That is acceptable because one person's 90-day slice is hundreds to low thousands of rows, and it is a quicksort in work_mem rather than an external merge — verify that rather than assume it.

Follow-up

A correction arrives carrying the same filler_order_id and the same version number as the row it corrects. What breaks, and what do you add to the sort key?
The same query for a cohort of 50,000 people instead of one person: does DISTINCT ON still hold up, and at what point do you materialise a latest-version table with a partial index?
How would you show the clinician that the displayed value was amended, given the chart query returns only the winner?

What is your prior networking experience?

Medium

Approach

Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How much coding experience do you have, and are you willing to be trained in multi-specialty coding?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

What type of background skills do you have that fit the job opening?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Model an order placement so a lost response stays recoverable

Hard · idempotency · partial failure · reconciliation

An ordering system POSTs a medication or lab order to /orders. It assigns a placer identifier; the performing system later assigns a filler identifier, and neither side controls both namespaces. The caller times out after 3s and cannot distinguish an order that never arrived, one that was accepted with the acknowledgement lost, and one still in flight. Duplicating an order is a patient-safety event and silently abandoning one is worse. Design the contract so the caller can always determine the true outcome without guessing, and specify its behaviour at the timeout, on retry, and after repeated failure.

Approach

Remove the ambiguity at its source by letting the caller name the resource before it exists, so the identity of the attempt never depends on our response arriving. The placer identifier is already caller-assigned and unique within the caller's namespace, so the server key is (placer_namespace, placer_order_id) under a unique constraint. A timeout stops being an unknown and becomes a question with a stable key.
Prefer the shape that makes the retry trivially safe. PUT /orders/{placer_namespace}/{placer_order_id} is idempotent by HTTP semantics and needs no key header; POST /orders needs an Idempotency-Key to reach the same place. Under either, the write is one INSERT ... ON CONFLICT DO NOTHING, because a check-then-insert lets two retries through under READ COMMITTED. A repeat with an identical canonical body returns the existing resource; a repeat with a different body is 409 rather than a silent replace, since an order is not a document to overwrite.
Give the caller a read that resolves a timeout without writing: GET on the same key returns accepted, routed, filled with its filler_order_id, or rejected with a reason, and 404 means we genuinely never saw it. Without that read, the caller's only instrument is another write, which is precisely the behaviour being designed out.
Be honest about what acceptance means. Accept is durably persisted and queued for the performing system, not performed, so return 202 with the order resource and a status the caller polls or subscribes to. Returning 201 for an order that is not yet routed makes the caller believe a stronger fact than is true, and the filler identifier arrives later on the performing system's own timeline.
Write the caller's behaviour explicitly, because the contract is only half the design: at timeout, GET the key; on 404, retry the write with the same key; on 5xx, exponential backoff with full jitter up to a bounded attempt count; after five failures, stop and raise a human-visible alert carrying the placer identifier. A queued-for-human state is a better outcome than either a duplicate order or a silent drop, and this domain cannot absorb the drop.

Worked solution (40 min)

Draw the three timelines a 3s timeout can hide - request lost, work done and response lost, work still in flight - and mark what the caller can distinguish in each, with and without a caller-assigned key.
Write the resource path, the unique constraint, and the single atomic statement that performs the write.
Write the state machine the GET exposes, state what 404 means, and name the one state that must never be inferred from a timeout.
Write the caller's pseudocode for timeout, retry, backoff and give-up, with the attempt count and the alert payload.
Run two concurrent retries of the same placer identifier against a real database and confirm one order exists and both callers observe the same state.

Expected result: PUT /orders/{placer_namespace}/{placer_order_id} under UNIQUE (placer_namespace, placer_order_id), written by one INSERT ... ON CONFLICT DO NOTHING; a repeat with the same canonical body returns the existing resource, a repeat with a different body returns 409. GET on the same key returns {status in accepted | routed | filled | rejected, filler_order_id when known, reason when rejected}, with 404 meaning never seen. Acceptance is 202 - durably queued, not performed. Caller: on timeout GET the key; on 404 retry the same key; on 5xx back off with full jitter for five attempts, then stop and alert with the placer identifier rather than resubmitting.

Verify it yourself

No outcome requires the caller to guess; every timeout resolves into a read.
Two concurrent retries of one placer identifier leave exactly one order.
A 202 is never returned for an order that is not durably persisted.
The give-up path terminates in a human-visible state rather than a dropped order.

Follow-up

The performing system returns a filler identifier for an order we have no record of. What do you do with it?
The caller reissues an old placer identifier for a genuinely different order after a counter rollover. What breaks, and how would you detect it?
Where does the record of the failed attempts live, and what must it carry for an incident review?

Gateway instances OOM every five days with flat traffic

Medium · memory leak · heap profiling · idempotency

Interoperability gateway instances are OOM-killed with exit 137 every five to six days. Resident memory grows about 400MB a day per instance and never falls; restarting resets it. Message throughput, connection count and partner mix are unchanged over that period. The gateway keeps an in-process set of recently seen idempotency keys — (sending facility, sending application, message control ID, event timestamp) — in front of the durable ledger. Give the ordered investigation and say what evidence would rule the idempotency cache out.

Approach

Establish that this is a retained-object leak before profiling. Compare resident memory with live heap measured immediately after a forced collection, sampled daily. Resident memory rising while post-collection live heap stays flat points at allocator fragmentation, off-heap or native buffers, or thread stacks, and a heap dump will implicate the wrong object. Rising post-collection live heap is a genuine leak.
Find what the growth is linear in. Plot daily growth against messages processed, connections accepted and wall-clock hours across instances carrying different traffic. Proportional to messages means something is retained per message; proportional to connections means per-connection state is not released on abnormal close, such as a half-open socket whose FIN never arrives.
Diff two heap dumps taken six hours apart on the same instance, sorted by retained size, and identify the dominator. An unbounded set or map shows up as one root holding millions of small entries. This is a five-minute answer once you have the dumps, which is precisely why the first two steps exist: they tell you whether the dump can answer the question at all.
Check the idempotency cache directly against arithmetic rather than suspicion. At single-digit millions of messages a day and a key of roughly 80 bytes plus per-entry overhead, an unevicted set grows by hundreds of megabytes a day, which matches the observed 400MB. A live gauge of its entry count that stays flat is the evidence that rules it out; a gauge tracking messages processed convicts it.
Fix and restore the invariant: bound the cache by size or by a time window, and keep the durable unique constraint as the authority. An in-process set is a latency optimisation and cannot be the correctness mechanism, because it does not survive a restart and is not shared across instances — so a replay right after a deploy would bypass it entirely.
Verify over 48 hours that post-collection live heap is flat, and alert on the cache's entry-count gauge rather than on resident memory, which only tells you after the growth has already happened.

Follow-up

How long must the time window be? Tie the number to the sender's replay behaviour after an interface restart and to how far back the broker redelivers.
If growth had tracked connections instead of messages, what would you look at first, and how would a half-open MLLP connection show up in the socket table?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

Tell me about a time where you had to gather requirements for a project.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Name a time you had to deal with a difficult situation and how did you handle it?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Tell me about a time that you and another employee had a disagreement on a project and how did you resolve it to get the job done?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?