Interview guides · Eversource Energy

Eversource Energy Software Engineer Interview Questions & Guide 2026

System design practice for Eversource Energy software engineer candidates: reach a data model and an API contract, not a box diagram.

The 60-second brief

As a Software Engineer at Eversource Energy, you play a vital role in building, maintaining, and modernizing the digital infrastructure that powers New England. This position directly impacts critical systems planning, grid modernization, asset management, and corporate IT services, supporting millions of customers who rely on safe and reliable energy daily. You will work at the intersection of traditional engineering principles and modern software solutions, driving projects that optimize energy distribution and operational efficiency.

The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.

Focus areas

Keep a site operating during a network partition
Fold late, out-of-order scan events by event time
Reconcile a movement ledger against derived positions

Overview

The work environment at Eversource Energy combines the stability of a major utility provider with complex technical challenges found in large-scale data processing, systems integration, and infrastructure automation. You might contribute to teams focused on transmission systems planning, smart grid rollouts, or enterprise-wide corporate applications. These problem spaces require you to handle high-reliability constraints, legacy system integrations, and modern cloud or hybrid architectures.

Expect an environment where collaboration, methodical problem-solving, and a focus on reliability are paramount. While the pace is generally steady and structured around large utility lifecycles, the technical scope is broad and deeply rewarding. You will collaborate closely with cross-functional teams, including operations, electrical engineers, and product stakeholders, to deliver robust software solutions that stand the test of time.

The Eversource Energy interview process

Eversource Energy candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Phone Screen

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Interview

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Interviews

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Final Discussions

Coding rounds mostly set a floor. They decide whether you clear the bar, not where you land on the ladder. Level tends to come out of the design discussion and the ownership stories, so the question worth auditing beforehand is whether the scope you describe matches the scope of the job. Work that stops at your own service, or a story whose hard part was writing the code rather than getting several people to agree on an interface, reads a level below where you think you are interviewing, and that gap is usually resolved downwards.

What to demonstrate

Whether the largest thing you describe owning ran end to end — the decision, the migration path, the rollout, and what you did when it went wrong — or stopped at the change you merged
Whether design answers include what you would not build, what you would defer, and what you would measure before committing, rather than only what the boxes are
Whether a disagreement in a story was settled with something checkable — a benchmark, a prototype, a written proposal — instead of by seniority or by waiting it out
Whether you can say which calls you made alone and which you escalated, and why the line sat where it did

How to prepare

Write your largest piece of owned work as a timeline of decisions — who decided what, when, and what you did when the plan broke — then delete every sentence whose subject is "we" and see how much survives
Take one system you know well and drill the migration answer: how old and new paths run side by side under live traffic, how you compare their outputs, what the rollback is once writes are going to both, and which step you would not automate
Map each line of the ladder in the job posting to a specific thing you have done, find the line you cannot support, and prepare the closest evidence you have plus an honest account of the gap

Where candidates lose points

Judging delivery windows and cut-offs in UTC

A delivery appointment, a receiving cut-off and a same-day promise are all statements about local wall-clock time at a specific stop, so evaluating them in UTC misclassifies on-time performance for any leg that crosses a zone, and misses cut-offs by an hour on the two days a year a region changes offset. The subtlety that catches experienced engineers is that converting a future local time to a UTC instant at write time bakes in the offset rules as they were known then: when a jurisdiction changes its rules, an appointment stored that way silently moves, which is why a future window is better stored as local wall time plus an IANA zone identifier and resolved at read time, while a past event is correctly stored as an instant. Duration arithmetic needs the same care, because a transit that spans a DST boundary is genuinely 23 or 25 hours long in local terms even though the elapsed instant count is unchanged. Storing only a date with no zone - common on partner feeds - is the version of this that cannot be fixed downstream, so the zone has to be captured at ingestion.

Treating the database as the truth about physical stock

Every system in this domain drifts from reality, through miscounts, unrecorded damage, mis-scans and theft, so a design with no adjustment path forces the drift somewhere worse. The usual shape is a CHECK constraint or application guard that clamps a position at zero: the negative that would have exposed a missing receipt is silently swallowed, and the first visible symptom is a picker standing at an empty location for an order the system promised. The correct posture is to let the invariant fail loudly - record the cycle-count adjustment as a first-class movement with its own reason code, surface the variance, and let planning see that this location is unreliable - because an adjustment that is indistinguishable from a normal pick destroys the only signal that would have found the root cause. A related version of the same mistake is deleting or updating a movement row to fix a mistake, which leaves the position correct and the history a lie.

Issuing one query per row of a result set

Fetch related rows in a single batched query keyed by the ids you already hold, or join them into the original query. A per-row round trip multiplies network latency by the row count, and it looks perfectly fine against the ten rows in your development database.

Reading the constraints as preamble rather than as part of the problem

The bounds are usually there to eliminate the obvious approach: n up to 10^5 makes an O(n^2) scan roughly 10^10 operations, far outside any per-test time budget, and an input larger than memory rules out loading it at all. When a bound is not given, ask for it, then say out loud which approach it kills.

Questions & practice

Trace a suspect lot forward to every shipped order line

Medium · graph traversal · genealogy · indexing

lot_edge(parent_lot_id, child_lot_id) has 400 million rows with btree indexes on both (parent_lot_id, child_lot_id) and (child_lot_id, parent_lot_id). lot_shipment(lot_id, order_line_id) maps lots to shipped lines. Given one suspect received lot, return every downstream order_line_id. The reachable subgraph can reach 2 million lots and 12 levels; the full edge table does not fit in memory. Give the traversal, its complexity in terms of the reachable subgraph, and what you do if the data contains a cycle.

Approach

Genealogy is a directed acyclic graph, not a tree: a production run consumes several lots and one lot splits across many shipments, so a node is reached by several paths and a tree traversal revisits it combinatorially. Breadth-first from the suspect lot over the (parent_lot_id, ...) index, with a hash set of visited lot ids, is O(V' + E') where V' and E' count only the reachable subgraph — 2 million 8-byte ids is 16 MB of payload and a few tens of MB with open-addressing overhead, which is why the visited set is affordable and the edge table's 400 million rows never enter memory.
Batch each level into one indexed read. Issue WHERE parent_lot_id = ANY($1) with a few thousand parents per call instead of a query per node: at 12 levels and 2 million nodes, per-node round trips dominate the wall clock entirely, and the recall's deadline is a real-world response time, not a query-plan aesthetic. The batch size is the tuning knob between round-trip count and per-statement planning cost.
Keep the visited set even though the schema says DAG. Rework and repack loops — material returned into an earlier lot — do occur, and an unguarded traversal on a cyclic edge set does not terminate; a set membership test is O(1) and turns a hang into a correct answer plus a data-quality alert. Also cap the traversal by level and by node count, and fail loudly on the cap rather than returning a partial set silently.
Join to lot_shipment once at the end over every visited lot id, not over the leaves and not per node. A lot is routinely both shipped and consumed: half a lot goes out on an order line while the other half is repacked into a child lot, so it has a lot_shipment row and outgoing edges at the same time and is not a leaf. Restricting the join to nodes with no outgoing edges drops exactly those order lines, which is under-recall — the one failure mode this trace is not allowed to have. One bulk join against an index on lot_id over the whole visited set is a single pass; per-node lookups multiply the round-trip problem by the number of visited lots.
State the semantics honestly: reachability answers which shipped lines could contain material from the suspect lot, and without propagating edge quantities it is a superset of physical containment. For a recall that is the correct direction of error — over-recall, never under-recall — and saying so is the point, because a trace that returns quickly and incompletely will be believed. The upstream question, which received lots are in a suspect shipped unit, is the same traversal over the reverse index, which is why both indexes exist.

Worked solution (30 min)

Draw a six-lot fixture containing a merge (two parents into one child), a split (one parent into three children), and one interior lot that is partly shipped on an order line and partly consumed into a child lot, then write down the expected reachable set and the expected order lines by hand.
Implement level-batched BFS with an explicit frontier list, a visited hash set, and a per-level batch read.
Add a cycle to the fixture and confirm the traversal terminates and flags it rather than hanging.
Bulk-join lot_shipment once over every visited lot id — the whole visited set, not the leaves — and count the queries issued, confirming it is one per level plus one.
Re-run with the frontier order reversed to confirm the output set is unchanged.

Expected result: Every lot downstream of the suspect appears exactly once in the visited set; the merge node is visited once, not twice; the partly-shipped interior lot contributes its order line even though it has children; the cyclic fixture terminates with the same reachable set plus a flagged edge; the query count is levels + 1, not proportional to node count.

Verify it yourself

The reachable set is a superset of the set you computed by hand, and equal to it on the acyclic fixture.
Restricting the final join to lots with no outgoing edges drops the partly-shipped interior lot's order line — the failing test that pins the over-recall rule to code.
Running the same traversal over the reverse index from a leaf returns the suspect lot.
Removing the visited set makes the diamond fixture return duplicates — a failing test that documents why the set is there.

Follow-up

A recursive CTE would express this in ten lines. What is its cost here if the index on the traversal column is missing, and how would you notice?
You need the quantity of suspect material in each downstream line, not just reachability. What has to be on the edge, and what does that do to the traversal?
The traversal must run while a production run is writing new edges. What does a consistent answer mean here, and what do you cut it at?

Fold late out-of-order status events into a monotonic leg state

Hard · semilattice fold · event time · idempotency

Leg statuses form the chain planned < tendered < accepted < picked_up < in_transit < delivered, with exception and cancelled outside it. You ingest 300 million observation_event rows a day carrying subject_id (leg), observed_status, occurred_at, received_at, clock_offset_ms and dedupe_key; five percent arrive over an hour late and a tail arrives a day late. Write the merge that produces each leg's current status, prove the result depends only on the set of events and not their order, and state the per-event cost.

Approach

Define the per-leg state as a tuple ordered totally: (status_rank, occurred_at, source_priority, event_id), and define merge(a, b) as the maximum under that order. Rank leads; occurred_at and the rest only break ties. The proof is then one line: maximum over a total order is idempotent, commutative and associative, so the state space with merge is a join-semilattice and the fold over any permutation of the event set yields the same value. That is also why no global sort is needed — the O(n log n) disappears and the fold is O(1) per event, O(distinct legs) state.
Rank before time, deliberately. occurred_at is device-reported and sometimes wrong; if it led the comparison, the leg's status would be handed to whichever device has the most badly skewed clock. Ranking first means a bad clock can only reorder events within one rank. Correct occurred_at by clock_offset_ms where it was measured, and where the offset exceeds a threshold keep the observation but bar it from winning a tie — the event is still evidence, it just stops being authority.
Make the lattice total, because exception and cancelled are not points on the chain and max over a partial order is undefined for incomparable pairs. Model cancelled as an absorbing top element, and carry exception as a boolean flag merged by OR alongside the rank rather than as a rank of its own — otherwise an exception at rank 4 either blocks delivered at rank 6 or is erased by it, and both are wrong. A product of semilattices is a semilattice, so the order-insensitivity argument still holds over the tuple.
Deduplicate on dedupe_key even though the merge is idempotent. Idempotence covers a byte-identical redelivery; a partner replaying a window with a fresh event_id is not identical, and the tie-break would let the replay displace the original for no reason. Dedupe on the stable key, then fold — both halves are needed, and conflating them is the usual defect.
Partition the stream by subject_id so all events for one leg land on one partition: broker ordering guarantees are per-partition only, and while a commutative merge makes interleaving harmless, a concurrent read-modify-write on the same leg row is still a lost update. Where that cannot be guaranteed, make the write a compare-and-set on a version column, or a conditional update that only raises the state. Because the state is mergeable, shard maps combine cleanly and replaying the whole day converges to the same answer — which is what makes reprocessing safe.
Say where genuine regressions live: a delivered shipment that is refused and returns is a new leg, not a rank decrease. Monotonicity is only sound if real reversals are modelled as new objects, and keeping every superseded observation is what lets you answer the dispute later, so the fold discards nothing — it marks losers superseded.

Follow-up

An event arrives a week late for a leg already closed and invoiced. Does it change the status, the on-time metric, both, or neither?
Two events for one leg have the same rank and the same occurred_at from different sources. What decides, and is that decision stable across a replay?
How would you test order-insensitivity rather than assert it?

Size dock doors from overlapping appointment windows

Medium · sweep line · intervals · time zones

For one node and one local calendar day you have up to 200,000 planned dock appointments from shipment_leg: leg_id, planned_arrive_at, planned_depart_at (both TIMESTAMPTZ), plus the node's IANA zone. A door is occupied over [arrive, depart). Return the minimum number of doors that lets every appointment start on time, and the maximal interval over which that peak is sustained. Target O(n log n). Say how you derive the day's boundaries and how you break ties between an arrival and a departure at the same instant.

Approach

Name the result before computing it: with one interchangeable door type, the minimum door count equals the maximum number of simultaneously occupied intervals. That equality is not a heuristic — interval graphs are perfect, so their chromatic number equals their clique number, and greedy left-to-right assignment achieves it. Say this, because it is what licenses solving a scheduling question with a counter.
Emit 2n endpoints, (t, +1) at each arrival and (t, -1) at each departure, sort by t, and at equal t order the -1 before the +1. Half-open occupancy means a trailer leaving at 10:00 frees the door for one arriving at 10:00; ordering arrivals first inflates the answer by exactly the number of back-to-back handoffs, which on a well-packed schedule is most of them.
Scan once with a running counter, tracking the maximum and the endpoint index where it was first reached. The peak is attained on a half-open interval between two consecutive endpoints, [t_i, t_{i+1}), not at an instant — report it that way or the operations team cannot act on it. Extend the interval while the counter stays at the maximum.
Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by planned_arrive_at from an index, use the min-heap variant instead — push each departure, pop all departures at or before the current arrival, and the heap size is the current occupancy. Same time bound, O(peak) space rather than O(n), which matters when peak occupancy is 40 and n is 200,000.
Derive the day's boundaries from the zone, not from arithmetic on instants. Convert local midnight and the next local midnight to instants in the node's IANA zone; on a transition day that span is 23 or 25 hours, so adding 86,400 seconds silently drops or duplicates an hour of appointments. Decide explicitly what a zero-length dwell means — under half-open semantics it occupies nothing — and state it.

Follow-up

Doors are typed: some break pallets, some are parcel-only. Does max-overlap still equal the door count?
Appointments have a tolerance — a trailer may start up to 20 minutes late without penalty. How does that change the objective, and is it still solvable by a sweep?
Half the appointments are actuals and half are plans. Which do you sweep for tomorrow's staffing, and which for last week's utilisation report?

Model the movement ledger so a retried write is a no-op

Easy · append-only ledger · idempotency · audit history

inventory_movement is append-only: movement_id, item_id, node_id, lot_id, state, delta_qty (signed BIGINT), uom_code, reason_code, ref_type, ref_id, idempotency_key, occurred_at, recorded_at, reversed_by. inventory_position is keyed (item_id, node_id, lot_id, state) with qty, version, last_movement_id. Give the DDL constraints and the exact statements a pick confirmation runs so that a client which times out after the commit and retries changes nothing. Then say how you correct a pick recorded against the wrong lot two days ago, and why the position carries CHECK (qty >= 0) rather than clamping at zero.

Approach

Put the guarantee in the schema, not the handler: UNIQUE (idempotency_key) on inventory_movement. A SELECT-then-INSERT cannot work, because two concurrent retries both read nothing before either commits and both then insert.
Claim with INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING movement_id, and branch on rowcount. Zero rows means the first attempt already committed, so the handler returns success without touching the position. This branch is the whole point: applying the position update on the conflict path is what makes the ledger and the shelf disagree by one pick forever.
Keep the insert and the position update in one transaction: UPDATE inventory_position SET qty = qty + :delta, version = version + 1, last_movement_id = :id WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = :s AND version = :v, then test the rowcount and retry the transaction on zero.
Use lot_id BIGINT NOT NULL DEFAULT 0 rather than NULL. In a default unique index NULLs are distinct from each other, so a NULL lot would let two position rows exist for the same untracked item and split the running sum in half.
Correct the mis-keyed pick with two compensating movements, never an UPDATE or a deleted_at flag: one reversal restoring the wrong lot, one fresh pick against the right lot, both with reason_code = 'reversal' or 'pick' and ref pointing at the original movement_id. The original row keeps its values; reversed_by is a back-pointer written in the same transaction so a reader does not need a reverse scan to see that the row was undone.
CHECK (qty >= 0) aborts the transaction and surfaces a real defect. GREATEST(qty + :delta, 0) writes a plausible number instead, so the missing receipt that caused the negative is never investigated and the first symptom is a picker at an empty location.

Follow-up

Two retries of the same pick arrive concurrently on different application instances. Walk through both transactions statement by statement and say which one commits a position change.
occurred_at from a handheld is five minutes ahead of recorded_at because the device clock drifted. Which of your queries break, and which column should each of them actually be using?
How long do you keep idempotency_key unique for, given the table grows by tens of millions of rows a month?

Two scan-event queries that stopped using their indexes

Medium · indexing · partition pruning · brin correlation

observation_event is range-partitioned monthly on occurred_at, with a btree on (subject_type, subject_id, occurred_at), a BRIN on occurred_at and a GiST on geo. Rows are inserted in received_at order and events are routinely hours late. Two queries regressed: (A) WHERE occurred_at::date = current_date AND observed_status = 'delivered'; (B) a backfill scanning WHERE received_at >= now() - interval '1 day'. For each, say why no index is used and no partition is pruned, and give the fix.

Approach

Query A wraps the column in a cast, so the predicate is over occurred_at::date and not over occurred_at. A btree or BRIN on the bare column cannot be probed, and the planner has no range to prune partitions with, so it plans every monthly partition. Fix it by rewriting to a half-open range: occurred_at >= :start AND occurred_at < :start + interval '1 day', with :start computed by the caller from the relevant IANA zone.
Do not reach for an expression index as the first fix here. The timestamptz-to-date cast is STABLE rather than IMMUTABLE because it depends on the session TimeZone, so PostgreSQL refuses to index it; indexing ((occurred_at AT TIME ZONE 'UTC')::date) is accepted but silently pins the day boundary to UTC, which is the wrong day for any stop outside it.
Query B filters a column that is neither indexed nor the partition key, so it is a full scan of every partition by construction. The subtler half is why the BRIN cannot rescue it: BRIN stores a min and max per block range and only pays when physical order correlates with the column. Rows arrive in received_at order while the indexed column is occurred_at, and late events interleave hours-old timestamps with current ones, so each range's min and max span days and almost every range qualifies.
Measure the correlation rather than assuming it: SELECT attname, correlation FROM pg_stats WHERE tablename = 'observation_event'. Near 1 means BRIN earns its keep, near 0 means it is a full scan with extra steps. Expect received_at near 1 and occurred_at well below it on this insert pattern.
Fix B with a btree on received_at, which stays compact because it is correlated with insert order, or better, drive the backfill from a monotonic cursor it already has such as event_id, which needs no new index on a table taking tens of thousands of inserts per second.
Confirm with EXPLAIN (ANALYZE, BUFFERS) and read the plan, not the clock: check how many partitions appear, whether rows removed by filter dwarfs rows returned, and whether shared buffers read matches the size of one partition or of the table.

Worked solution (25 min)

Load a few million rows across three monthly partitions with occurred_at deliberately shuffled against insert order.
EXPLAIN (ANALYZE, BUFFERS) query A as written and record the partitions scanned and the buffers read.
Rewrite A as a half-open range and re-run, comparing partitions scanned and buffers.
Query pg_stats for the correlation of occurred_at and received_at, then add a btree on received_at and re-plan query B.

Expected result: Query A plans one partition and an index scan after the rewrite, versus every partition before. Query B moves from a full scan of all partitions to an index scan on received_at, and the measured correlation explains why the existing BRIN never helped.

Verify it yourself

The rewritten A returns byte-identical rows to the cast version for a day fully inside the chosen zone.
Buffers read for A drops by roughly the ratio of one partition to the whole table.
pg_stats shows occurred_at correlation well below received_at, matching the insert pattern.
Forcing enable_seqscan = off does not make the original A use the index, confirming the cast and not the planner is the cause.

Follow-up

Where would you put the partition boundary given events arrive up to a week late, and what happens to a row whose occurred_at falls in a partition that has already been detached?
The delivered-today query is asked per node in local time. Does that change your partitioning, your index, or only the caller?
Adding a btree to this table costs write amplification. How would you decide whether the backfill is worth it?

If a system failure occurs or a spare unit is unavailable in our system, how do you diagnose and resolve the issue?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Walk me through your resume and highlight your most relevant past work experiences.

Medium

Approach

State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Fan shipment status out to five thousand customer webhooks

Medium · fan-out · queues · backpressure · at-least-once

Customers subscribe to shipment status by webhook: 5,000 endpoints, some with an 8 second p99, some down for a day. The upstream is folded leg status, not raw observations. Requirements are per-shipment ordering at each subscriber, at-least-once delivery with a stable dedupe id, and no subscriber able to slow another. Assume 3 million legs a day, about six status transitions each, and a mean of three subscribers per shipment. Design the fan-out and quantify what folding first saves you.

Approach

Fan out transitions, not observations, and do the arithmetic before designing delivery. Raw ingest averages 9,000 observations a second, about 780 million a day. Three million legs at six transitions each is 18 million notification sources a day, 208 per second, and three subscribers apiece is 54 million deliveries a day, 625 per second. Folding first is worth a factor of about 43 - 208 sources per second instead of 9,000, and 625 deliveries per second instead of roughly 27,000 - and that factor dominates every subsequent optimisation, so settle it before tuning delivery.
Keep delivery state as a durable per-subscriber cursor over a per-subscriber queue rather than a row per attempt. At 54 million deliveries a day a row per attempt is roughly 19.7 billion rows a year before a single retry is counted, and retries only add to it; a cursor plus a bounded in-flight window gives retry and progress without any of them. Per-event rows earn their cost only where you must later prove a specific delivery in a dispute, so scope them to the subscribers who contract for that rather than to all 5,000.
Preserve ordering deliberately: a bounded in-flight window greater than one can reorder. Give each subscriber's queue a small number of lanes keyed by hash(shipment_id), so order is strict within a lane and parallelism comes from lanes. Ordering is then a property of the key, not of a hopeful single-threaded consumer.
Isolate subscribers with a per-subscriber concurrency cap drawn against a shared worker pool. By Little's law an 8 second endpoint receiving 20 notifications per second wants 160 workers in flight; without a cap it takes them from the pool and starves the other 4,999. The cap converts a slow subscriber into its own growing backlog instead of a shared outage.
Handle the dead endpoint as a bounded liability: circuit-break after consecutive failures, stop attempting, let the cursor lag, and retain the queue to a fixed horizon such as 72 hours. At the horizon, advance the cursor, record a gap, and tell the subscriber to re-sync from the pull API. Unbounded retention for one dead customer is how the fan-out store becomes the next incident.
Make at-least-once an explicit contract rather than an accident: every payload carries a stable event id and the leg's status_rank, so a customer can discard a duplicate or a stale retry on their side, and that requirement is documented rather than discovered.

Worked solution (30 min)

Derive the notification rate from legs per day, transitions per leg and subscribers per shipment, set it beside the raw observation rate, and state the reduction factor that justifies folding first.
Choose between per-event delivery rows and a per-subscriber cursor, and show how the chosen shape still preserves per-shipment ordering under retry.
Set a per-subscriber concurrency cap and compute an 8 second endpoint's in-flight occupancy at its share of traffic.
Write the circuit-breaker states, the retention horizon and the re-sync contract for a subscriber that returns after a day offline.

Expected result: Notifications sourced from folded transitions - about 208 per second and 625 deliveries per second, against 9,000 observations per second raw, a factor of about 43 - carried on per-subscriber queues with lanes keyed by shipment so ordering holds while lanes run in parallel; a per-subscriber concurrency cap bounding a slow endpoint's occupancy; a circuit breaker with a bounded retention horizon and a re-sync path; and an explicit at-least-once contract with a stable dedupe id.

Verify it yourself

A subscriber that stops responding for 24 hours changes nothing for the other 4,999.
Two notifications for one shipment cannot arrive at a subscriber out of order, including after a retry.
Retention is bounded and there is a defined, documented behaviour when the horizon is exceeded.
The volume reduction from folding first is quantified and the ratio matches the stated rates - about 43 times, 9,000 observations per second down to 208 transitions - rather than asserted.

Follow-up

A customer demands exactly-once. What can you actually offer, and what must they implement to get the effect they want?
One subscriber covers 40 percent of all shipments. What breaks first in a per-subscriber queue design, and what do you shard on instead?
How do you replay a day of notifications for one subscriber after a bug on your side, without re-sending to the other 4,999?

Orchestrator memory climbs a gigabyte a day at flat throughput

Medium · memory leak · timers · heap analysis

The fulfilment orchestrator's resident memory rises about 900 MB a day and old-generation occupancy after each full collection rises monotonically; a restart resets both. Transition throughput and latency are unchanged, and saga state itself lives in the database. The process keeps an in-heap timer wheel with one timeout entry per forward step of each order line, and lines legitimately sit in one state for days. Diagnose in order, prove the hypothesis with arithmetic before changing code, and give the fix.

Approach

Distinguish a leak from a large working set first, because the remedies are opposite. Plot occupancy immediately after each full collection, not resident memory: a monotone rise across days is retention, while a rise that plateaus is a cache doing its job. Live sagas legitimately number in the millions here, so the working set is genuinely large and the question is only whether it is bounded.
Diff two heap dumps taken two hours apart by retained size per dominator rather than guessing at the suspect. The growth will show as timer entries that transitively pin whole saga objects; the reference path from the GC root through the wheel's bucket array to the entry to the saga is what names the owner, and without that path you are guessing.
Prove the hypothesis numerically before touching code. 900 MB a day divided by roughly 1.5 KB retained per entry is about 600,000 entries a day, near seven a second. Compare that to entries created per second minus entries removed per second. If the rates do not match within an order of magnitude, the dominator is something else and the timer wheel is a bystander.
Name the defect precisely: the entry is registered when a step starts and is only released when its timeout fires, so a step that completes early leaves an orphan that survives until its deadline, and a step whose line is cancelled or re-sourced leaves one that may never be cleaned at all. Age cannot be used to identify orphans, because a line sitting in one state for days is normal.
Fix by cancelling on transition, keyed by (order_line_id, step), so the entry is removed by the same code path that advances the saga. Then move the timer out of the heap entirely: persist due_at on the saga row and let a sweeper poll WHERE due_at < now() AND status IN (open states) behind an index on due_at partial to those states. That bounds memory by the poll batch instead of by open sagas, and fixes the separate bug that an in-heap wheel silently loses every pending timeout on restart or deploy.
Add a bound so the next version of this is loud. Cap the in-flight timer structure, emit a metric on entries created versus removed, and alert on the difference rather than on memory, which only moves after the damage is done.

Follow-up

The sweeper now polls millions of open sagas. What does the index look like and how do you keep the poll from scanning the whole table?
Two orchestrator instances poll the same due sagas. How do you stop both from firing the same timeout?
How would you have detected the lost-timeouts-on-restart bug, given that it produced no error and no memory symptom?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

Run the same sub-skill decomposition on the second-largest gap in half the time.
Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

Tell me about a time when you faced adversity and how you handled it.

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?

How does your current role or past experience compare to the Software Engineer role you applied for here?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Tell me about a time when you provided strong customer service to internal or external stakeholders.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Tell me about a time when you demonstrated trust or worked effectively within a team.

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?