Interview guides · Epicor Software
Epicor Software Software Engineer Interview Questions & Guide 2026
Code reading practice for Epicor Software software engineer prep: work out what unfamiliar code does before judging it.
The 60-second brief
A Software Engineer at Epicor Software serves as a vital architect of industry-specific solutions that power global supply chains, manufacturing, and retail operations. You are not just writing code; you are building the digital backbone for businesses that rely on Epicor Software to manage complex, mission-critical data. Whether working on Cloud R, Fullstack development, or Product development, your contributions directly impact the efficiency and scalability of the platforms that keep the global economy moving.
Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.
Focus areas
- Make every write idempotent under client retries
- Bound blast radius with per-tenant concurrency limits
- Build at-least-once pipelines with explicit deduplication horizons
Overview
The role demands a blend of technical precision and product-centric thinking. You will be expected to navigate complex systems, troubleshoot intricate logic, and collaborate across global teams to deliver high-quality software. This position is ideal for engineers who thrive on solving real-world problems and who are motivated by the challenge of evolving legacy systems while integrating modern, cloud-native technologies.
The Epicor Software interview process
Epicor Software candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Initial Assessments
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Technical Screening
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Team-Based Interviews
You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.
What to demonstrate
- Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
- Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
- Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
- Whether your part in a team effort is stated accurately, including what other people did
How to prepare
- Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
- Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
- Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.
Where candidates lose points
Checking a quota with a select and then writing
Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.
Treating a timed-out write as a failed write
A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.
Assuming the input fits in memory
Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.
Assuming the bug is in the framework
Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.
Questions & practice
Find peak concurrent sandbox usage from run intervals
Medium · sweep-line · intervals · concurrency-caps · null-semantics
Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.
Approach
- Turn each run into two sweep events,
(started_at, +1)and(end, -1), then sort the 2n events by timestamp with-1ordered before+1at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap. - Decide each null out loud before sweeping, because each choice moves the answer. A null
started_atmeans queued and contributes nothing. A nullfinished_atwith statusrunningorleasedis clipped to the window end. Statuslosthas no observed end at all, so clip it atstarted_at + wall_clock_limit_secondson the grounds that the supervisor owns the timeout, and record that you did. The table'scheck (finished_at is null or started_at is not null)guarantees you never see an end without a start. - Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update
peak_atonly on a strict increase, or you will report the last such instant instead of the earliest). Capture the firstrun_idwhose+1takes the counter to C+1 during the same sweep rather than in a second pass. - Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by
started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on(tenant_id, started_at). - If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.
Follow-up
- Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
- The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
- How would you answer 'peak concurrency within any 5-minute window' without re-sorting?
Order a job dependency graph and find its critical path
Medium · topological-sort · dag-longest-path · cycle-detection · critical-path
A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.
Approach
- Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
- Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
- Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order:
earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E). - Second pass in reverse topological order for
latest_finish, thenslack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain.slack[v] = 0is exactly the statement that some longest path runs through v; equivalently, the longest path through v has lengthT - slack[v]. - The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by
min(d, T - L_avoid(v)), whereL_avoid(v)is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, whilemin(10, 15 - 14)predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan ismax(T - d, L_avoid(v)). - Compute
L_avoid(v)the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, sinceL_avoid(v)only ever matters through that max: setduration[v] := 0, recompute the makespan asT0(v) = max(T - duration[v], L_avoid(v)), and the gain ismin(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.
Worked solution (30 min)
- Build four fixtures. A: 12 jobs, two branches of 100 s and 95 s that share no job. B: fixture A plus one back edge. C: two disjoint paths tied at 100 s. D: the shared-prefix case, one job of 10 s feeding a 5 s job and a 4 s job, so the longest path is 15 s and the runner-up is 14 s.
- Run Kahn; on fixture B confirm it emits fewer than V nodes, then run the residual-subgraph DFS and print the actual cycle.
- Compute
earliest_finishforward andlatest_finishbackward, and list the zero-slack set for each fixture. - For each zero-slack job v, recompute the makespan with
duration[v] := 0to getT0(v), and record both the correct boundT - T0(v)and the wrong one,T - second_longest_path, side by side. - Apply the shortening for real (20 s off the critical branch of A, 10 s off the shared prefix of D) and diff the recomputed makespan against each prediction.
Expected result: Fixture A: makespan 100 s, and shortening by 20 s leaves 95 s, a gain of 5 s. Both formulas agree here, because the 95 s branch avoids the shortened job. Fixture D: makespan 15 s, and shortening by 10 s leaves 5 s, a gain of the full 10 s, which T - T0(v) = 15 - 5 = 10 predicts and T - second_longest = 1 does not. Fixture C: the zero-slack set covers both tied paths, and shortening a job on one of them alone gains nothing, since the other path still runs 100 s.
Verify it yourself
- The cycle printed by the DFS is a real cycle: every consecutive pair is an edge present in the input.
- Summing durations along any zero-slack path equals the makespan exactly.
- For every candidate, the predicted gain equals the makespan recomputed from scratch with the shortened duration. Assert the equality; do not eyeball it.
- Fixture D is the regression test for the bound, and a fixture whose branches are disjoint is not: it passes under either formula and proves nothing.
Follow-up
- Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
- Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
- Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?
Schedule ordered webhook retries with a heap of subscription queues
Medium · heap · scheduling · backoff · head-of-line-blocking
Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.
Approach
- Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's
next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, incrementattempt_countand setnext_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.- Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
- Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
- All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on
(subscription_id, next_attempt_at) where status in ('pending','failed_retryable')claimed withFOR UPDATE SKIP LOCKED, and the write-back must be fenced onlease_tokenso a worker that stalled and resumed cannot overwrite a newer attempt.
Follow-up
- One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
- A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
- The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?
Find the join that inflates every invoice total
Medium · join fan-out · aggregation grain · sql correctness
invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.
Approach
- Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row:
sum(l.amount_minor)is the true total multiplied by the attempt count, andcount(p.attempt_id)is attempts times lines. Three lines and two attempts report double the money and six attempts. - Reject the reflex repair.
count(distinct p.attempt_id)does fix the count, because attempt_id is unique.sum(distinct l.amount_minor)does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match. - Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
- Keep invoices with no attempts by making the attempt branch a LEFT JOIN with
coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about. - Push each filter to its own grain:
where l.voided_at is nullbelongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants. - Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.
Follow-up
- Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with
filter (where ...)be better? - Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
- The same report is needed per tenant per month. What index makes the line CTE cheap?
Decide which facts an invoice line copies instead of joining
Medium · denormalisation · immutable records · foreign keys · rounding
invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.
Approach
- Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
- Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
- Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
- Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason
source_rollup_watermarkis stored per line: without it, nobody can reconstruct what the customer was told. - Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
- Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.
Worked solution (25 min)
- Write the DDL: unique (invoice_id, tenant_id) on invoice, the composite FK from the line, and a comment on each denormalised column saying whether it is a snapshot or a cache.
- Attempt to insert a line whose tenant_id differs from its invoice's and confirm the foreign key rejects it.
- Rename a tenant, re-render a sealed invoice, and confirm the rendered name is the one stored on the header.
- Recompute amount_minor from quantity times unit_price_micros for a thousand synthetic lines rounding half-up, sum both ways, and record the divergence from the stored half-even values.
Expected result: tenant_id kept and enforced by a composite foreign key to (invoice_id, tenant_id); amount_minor kept because rounding happens once with a stated mode; legal name and billing address snapshotted onto the invoice header and never updated; mutable operational attributes left normalised. Each decision paired with its read pattern, its anomaly, and the mechanism that prevents the anomaly.
Verify it yourself
- A line cannot be attached to an invoice belonging to another tenant, and the rejection comes from the database rather than the application.
- Re-rendering a sealed invoice after editing the tenant record produces identical output.
- Rounding per line and rounding once at the end differ measurably over a thousand lines, which is the argument for storing the value.
- No column on the line is recomputed by any reader.
Follow-up
- Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
- A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
- Where does currency live, and what breaks if a tenant's billing currency changes between two periods?
If a system is experiencing latency issues, what steps do you take to identify the bottleneck?
Medium · system design and architecture
Approach
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
- Name the read and write paths separately; they rarely have the same bottleneck.
Follow-up
- How does this behave when that dependency is down for an hour?
- What breaks first when traffic grows ten times?
How would you design a microservice to handle high-frequency data updates?
Medium · system design and architecture
Approach
- State the consistency you need, and where you are willing to be stale.
- Name the read and write paths separately; they rarely have the same bottleneck.
- Fix the scope first: who calls this, how often, and what they do when it fails.
Follow-up
- What breaks first when traffic grows ten times?
- What would you drop to keep the system up under load?
How do you approach debugging a critical production issue under time pressure?
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- State your assumptions explicitly before working the problem.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
How do you manage state in a complex Fullstack application?
Medium
Approach
- Work from the requirement backwards to the design.
- Say what you would check first and why it is the highest-information step.
- State your assumptions explicitly before working the problem.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Design the batch ingest endpoint metering agents retry into
Hard · batch api · partial failure · deduplication
A customer-run agent posts usage events in batches of up to 1,000 to metering-ingest with a 30-second timeout and at-least-once retry of the whole batch. Each event carries event_id, idempotency_key, sku, quantity and occurred_at; the server adds ingested_at, and usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). Design the endpoint: the request shape, what the response says when 900 events are new, 90 are duplicates and 10 are malformed, the status code, and the agent's algorithm on timeout. Then state the deduplication horizon and justify it against that unique constraint.
Approach
- Fix the per-item outcome taxonomy first, because the status code follows from it: accepted, duplicate, and rejected with a permanent code. A duplicate is a success; reporting it as an error makes the agent either re-send revenue it already delivered or drop it.
- Allow only permanent failures per item. A transient per-item failure inside a 200 invites the agent to discard that event, so anything transient escalates to a 5xx for the entire batch. A 200 is then a promise that every event not marked rejected is committed and durable.
- Return 200 with a results array aligned by index and carrying the event id, so the agent can retry precisely the subset that needs it and quarantine the ten malformed events instead of hard-looping a poison batch forever. Cap the batch at 1,000 items and a byte size, with 413 beyond it and 429 with Retry-After for backpressure.
- Deduplicate per event, never per batch: the agent may split, merge or reorder a retried batch, so a batch-level key matches nothing on the second attempt. The key is (tenant_id, idempotency_key), and the tenant comes from the resolved credential; a tenant id present in the body is compared against it, never trusted.
- Size the horizon as a correctness parameter. The unique index includes the partition key, so it deduplicates only within one day: a retry that crosses midnight, or a replay run a week later, passes straight through it. A separate dedup store keyed (tenant_id, idempotency_key) with a TTL exceeding the agent's maximum retry window plus the longest replay you intend to support is what actually enforces the invariant, which makes its retention a correctness setting rather than a cost knob.
- Order the commit against the response and the acknowledgement: commit then respond at the endpoint, and downstream commit the fold then acknowledge the message. Acknowledging first turns a crash into silently lost revenue with no error raised anywhere.
Worked solution (40 min)
- Write the request body schema with the batch envelope and one event, and state which fields the server assigns rather than accepts.
- Write the 200 response for the 900/90/10 case, showing three result entries, one of each outcome, with the rejected one carrying a permanent code.
- Write the rule separating per-item rejection from whole-batch failure, and list which conditions fall on each side.
- Compute the dedup horizon from the agent's retry window plus the replay window you support, and say where the dedup state lives and how it ages out.
- Write the agent's pseudocode for timeout, 5xx, 429 and 200-with-rejections, four branches, and mark which branch may drop an event.
- Trace the crash between commit and response, and between fold and acknowledgement, and say what each produces.
Expected result: A 200 with an index-aligned results array where duplicates count as success and rejections are permanent-only, transient problems escalated to a whole-batch 5xx, per-event deduplication keyed on (tenant_id, idempotency_key), and a published horizon larger than retry plus replay because the partitioned unique index only covers one day.
Verify it yourself
- The agent can distinguish 'already counted' from 'never counted' for every event in the batch.
- Exactly one branch of the agent's algorithm ever discards an event, and it is the permanent-rejection branch.
- A retry crossing midnight is still deduplicated, and the mechanism named is not the unique index.
- No crash point in the pipeline both commits work and loses it without an error.
Follow-up
- The agent times out at 30 seconds having received nothing. What exactly does it do next, and what in your design makes that safe?
- Ten events are rejected every hour for a week and nobody notices. What does the endpoint owe the customer beyond a per-item 4xx code?
- A replay pushes 40 million events through this endpoint in an hour. Which part of your design degrades first?
Metering partition crash-loops and the sealing watermark freezes
Medium · poison message · at-least-once · dead-letter queue
One metering-ingest partition has stopped advancing. Lag grows linearly, the consumer restarts about every 40 seconds, and the same offset appears in every startup log while other partitions stay healthy. Events are committed in batches of a few thousand and the acknowledgement follows the commit. Sealing is six hours away and source_max_ingested_at for that partition's tenants is frozen. Give an ordered checklist, a containment action available within minutes, and the durable fix, saying what each does to exactly-once accounting.
Approach
- Distinguish a poison record from a capacity problem in one measurement: compare the offset and the exception across restarts. An identical pair every time is deterministic failure on one record, whereas a throughput problem still advances the offset between crashes.
- Read the record from a separate consumer group so the bytes can be inspected without perturbing the stuck consumer, then classify the defect: schema violation, a quantity failing the non-negative check, a null workspace, an unmappable SKU enum, or a payload past a size limit. That classification decides whether this is a producer bug or a missing consumer guard.
- Account for batch granularity before acting. With commits of a few thousand, one bad record fails thousands of good ones, so the blast radius is the batch. Halve the batch around the offset to isolate the record, or move to per-record error isolation so the radius becomes the record.
- Contain by diverting that record to a dead-letter store with its raw bytes and offset, then resume. This is safe here precisely because the acknowledgement follows the commit: the good records from the failed batch are re-consumed and absorbed by the uniqueness check on (tenant_id, idempotency_key) rather than counted twice.
- Make the fix durable with per-record error isolation, a bounded poison counter, and an alert on dead-letter rate rather than on lag alone, since lag only reveals this after the sealing margin has already been eaten.
- Check the horizon before replaying anything. The unique index lives on a daily-partitioned table and therefore includes the partition key, so it deduplicates within a day only; a replay landing on a later ingest day needs the separate dedup store or it double-counts into a tenant's bill.
Follow-up
- Move the acknowledgement before the commit and describe exactly what is lost and what is duplicated in each of the two crash windows.
- Sealing is in six hours and the partition will not drain in time. What do you seal on, and what does the invoice have to record so the difference is explainable later?
- A producer replays two weeks of events next month. Which part of your fix stops holding, and what is the dedup horizon you would actually configure?
Suggested practice checklist
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Day 1 — Measure before reasoning
- Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
- Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
- Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.
Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.
Day 2 — References, copies, and the bugs they produce
- Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
- Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
- Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.
Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.
Day 3 — Types, once, in a language that checks them
- Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
- Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
- Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.
Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.
Day 4 — Concurrency, starting with what actually runs at the same time
- Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
- Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
- Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.
Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.
Day 5 — Debugging as a procedure rather than an instinct
- Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
- Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
- Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.
Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.
Day 6 — Tests that catch the bug you are about to write
- Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
- Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
- Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.
Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.
Day 7 — Debug something broken, out loud
- Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
- Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
- Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.
Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.
Review and mentoring: what you let through
Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.
Tell me about a time you had a conflict with a stakeholder regarding technical requirements.
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Give the blast radius: what could have broken, and what you measured.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Describe a project where you had to learn a new technology stack on the fly.
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What did you decide not to do, and why?
- What would you do differently if you ran that again?
How do you ensure your work remains aligned with the broader product goals?
Medium · behavioural and engineering judgement
Approach
- Give the blast radius: what could have broken, and what you measured.
- Close with what you would do differently, concretely.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?