cswork / libraryOpen practice workspace

Interview guides · Spritle Software

Spritle Software Software Engineer Interview Questions & Guide 2026

Spritle Software software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

A Software Engineer at Spritle Software is a versatile technical contributor tasked with building scalable, high-quality digital solutions. You will be working in a dynamic environment where the focus is not just on writing code, but on solving complex business problems through efficient architecture and clean implementation. Your work directly impacts the products delivered to clients, requiring a blend of technical precision and a strong understanding of the full software development lifecycle.

Prepare in one language you know well enough to debug in rather than the one you think reads best. Under a clock an unfamiliar language costs you standard-library lookups and iteration mechanics, and that time comes out of your thinking budget, not your typing budget.

Focus areas

  • Make every write idempotent under client retries
  • Scope every query and cache key by tenant
  • Keep money in integer minor units

Overview

This role is critical to the company’s mission of delivering robust enterprise and consumer-grade software. You will contribute to diverse projects that span web, mobile, and backend technologies, often working with modern stacks such as Ruby on Rails, React.js, Node.js, and Python. The environment is fast-paced and collaborative, making it an excellent space for engineers who are eager to learn, adapt to new frameworks, and take ownership of their contributions from design to deployment.

Spritle Software values practical, hands-on experience. Be prepared to discuss not just the "how" of your code, but the "why" behind your architectural decisions and tool selections.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

  • Breadth across SQL, experimentation and product reasoning
  • Ability to state assumptions before choosing a method

How to prepare

  • Drill the practice exercises below and time yourself
  • Prepare three quantified stories about decisions you drove

Where candidates lose points

Holding money in a floating-point type, or rounding it more than once

Binary floating point cannot represent 0.01 or 0.1 exactly, so sums drift and two code paths that should agree disagree by cents nobody can trace back. The fix is integer minor units or an exact decimal type end to end, with sub-cent rates expressed as scaled integers such as micro-units, because a per-request price genuinely is smaller than a cent. The second half of the trap is rounding position: rounding each line and then summing gives a different total from summing and rounding once, and half-up and half-even diverge systematically across many lines, so rounding must happen at one named place and every downstream reader must carry the rounded value rather than recompute it from quantity and rate.

Paginating a growing table with limit and offset

Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.

Listing technologies instead of trade-offs

Name the property the design needs first, such as ordered range scans, multi-entity transactions, cheap appends, or a predictable p99, then pick something that provides it and say what it gives up in exchange. Almost any component is defensible once you state the requirement it satisfies and the one it sacrifices.

Writing code before the input contract is pinned down

Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.

Questions & practice

Order a job dependency graph and find its critical path

Medium · topological-sort · dag-longest-path · cycle-detection · critical-path

A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.

Approach

  1. Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
  2. Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
  3. Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order: earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E).
  4. Second pass in reverse topological order for latest_finish, then slack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain. slack[v] = 0 is exactly the statement that some longest path runs through v; equivalently, the longest path through v has length T - slack[v].
  5. The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by min(d, T - L_avoid(v)), where L_avoid(v) is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, while min(10, 15 - 14) predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan is max(T - d, L_avoid(v)).
  6. Compute L_avoid(v) the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, since L_avoid(v) only ever matters through that max: set duration[v] := 0, recompute the makespan as T0(v) = max(T - duration[v], L_avoid(v)), and the gain is min(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.

Follow-up

  • Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
  • Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
  • Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

  1. Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
  2. The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
  3. Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
  4. Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
  5. Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
  6. State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Worked solution (25 min)

  1. Write both key tuples down before any code: dedup key (tenant_id, idempotency_key), cell key (tenant_id, workspace_id, sku, hour_start), with hour_start derived from occurred_at in UTC.
  2. Build a 10,000-row fixture containing one event duplicated three times under the same idempotency_key, two events sharing an idempotency_key across different tenant_id values, one event whose occurred_at is two hours before its ingested_at, and one staging event inside an otherwise production cell.
  3. Fold it and assert each of those four expectations separately rather than eyeballing a grand total.
  4. Re-run with the input shuffled and diff the output files.
  5. Size the dedup set for 250M keys using the load-factor arithmetic and write the number down next to the fixture.

Expected result: The triplicate contributes one event and its quantity once. The two same-key, different-tenant events both count, because the dedup key is the pair. The late event lands in the hour of its occurred_at while that cell's source_max_ingested_at advances to the later timestamp. The staging event is included or excluded per the stated filter and never silently.

Verify it yourself

  • Shuffled input produces byte-identical cells, including event_count and source_max_ingested_at.
  • The sum of quantity_sum across all cells equals the sum of quantity over the distinct dedup keys, computed independently.
  • Swapping the accumulator to binary64 makes a fixture of 0.1-valued quantities disagree with the exact sum. That is a failing test, not a rounding note.

Follow-up

  • A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
  • The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
  • What makes a re-run over the same day produce byte-identical rollups?

Seal an hour under late data with bounded memory

Hard · watermarks · late-data · quantile-sketch · conditional-write

Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.

Approach

  1. Two clocks, two jobs. Bucket by occurred_at, because that is the hour the customer is billed for, and advance the watermark on ingested_at, because that is what the fold has consumed and what source_max_ingested_at records. Conflating them is what makes late data invisible.
  2. The global watermark is the min over partitions of each partition's committed ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past.
  3. Choose the lateness L from the measured distribution of ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters.
  4. Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
  5. Hold open hours in a min-heap keyed by hour_start. When the watermark advances, pop every hour with hour_end + L < W and seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct (tenant, workspace, sku) keys, so cap the number of simultaneously open hours and spill the oldest into usage_rollup_hourly as status='open' with a revision bump. While an hour is open the row is upsertable, so the store is your overflow.
  6. The seal itself is a conditional write: update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, and source_max_ingested_at is what proves it arrived afterwards.

Follow-up

  • A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
  • Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
  • How would you detect that L has drifted before it costs you an hour's quantity?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

  1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
  2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
  3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
  4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
  5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Worked solution (25 min)

  1. Seed a tenant with 19 running rows, then fire 8 concurrent sessions each running the select-then-insert, and count the resulting running rows.
  2. Repeat at REPEATABLE READ and confirm the count still exceeds 20.
  3. Repeat at SERIALIZABLE, count the 40001 aborts, and note that without a retry loop those requests fail rather than queue.
  4. Implement the atomic counter UPDATE, re-run the 8-way test, and confirm exactly 20 running rows with zero over-admissions.
  5. Kill a worker mid-run, let the lease expire, and check whether the slot comes back without intervention.

Expected result: Write skew identified; read committed and repeatable read both permit it; serialisable prevents it at the price of 40001 retries; the working read-committed version is update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running, whose zero-row result is the rejection, plus a reconciliation path for runs that terminate as 'lost'.

Verify it yourself

  • At 8x concurrency against a cap of 20, running rows never exceed 20.
  • The rejection returns zero rows rather than raising, so it is distinguishable from an error in the caller.
  • Killing a worker does not permanently consume a slot.
  • The counter and select count(*) from job_run where status = 'running' agree after reconciliation.

Follow-up

  • Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
  • Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
  • The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Find the join that inflates every invoice total

Medium · join fan-out · aggregation grain · sql correctness

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

Approach

  1. Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
  2. Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
  3. Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
  4. Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
  5. Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
  6. Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Follow-up

  • Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
  • Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
  • The same report is needed per tenant per month. What index makes the line CTE cheap?

How do you approach API development and documentation?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

What are the core differences between Flask and other backend frameworks?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Say what you would check first and why it is the highest-information step.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

This category evaluates your depth of knowledge in your chosen stack and your understanding of fundamental software engineering principles.

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Say what you would check first and why it is the highest-information step.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Explain the concept of threads and daemons in Java.

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

An idempotent create endpoint that returns a one-time secret

Easy · idempotency · race conditions · api design

POST /v1/api-keys inserts a tenant_api_key row (tenant_id, workspace_id, name, key_prefix, secret_hash, scopes, status, auth_version, created_at, expires_at) and returns the plaintext secret exactly once, since only its SHA-256 is stored. Write volume is tens per second and clients retry on timeout. Design the idempotency mechanism: what the key is scoped by, where the record lives, its retention, what happens when the same key arrives with a different body, what happens when a retry arrives while the first request is still in flight, and what a replay returns for the secret.

Approach

  1. Scope the key by tenant, not globally: uniqueness is on (tenant_id, idempotency_key), or one tenant's key collides with another's and the second caller receives a stored response for a request it never sent. Store a fingerprint of the request alongside it - method, path and a hash of the canonicalised body - so a mismatch is detectable.
  2. Let the unique constraint decide the race instead of application logic. In the same transaction as the credential insert, run INSERT INTO idempotency_record (tenant_id, key, request_fingerprint, status) VALUES (...) ON CONFLICT DO NOTHING RETURNING id; no returned row means this is a replay, and the existing record is then read. A select-then-insert here loses to itself under concurrency in exactly the way this endpoint is meant to prevent.
  3. Write the three replay outcomes as a decision table rather than as prose: same fingerprint and completed returns the stored response; same fingerprint and still in flight returns 409 with Retry-After, without blocking and without executing; different fingerprint returns 422, because replaying a stored response for a mutated body would tell the caller a credential was created for parameters it never sent.
  4. Handle the secret as the part that makes this endpoint different from an ordinary idempotent create. The plaintext cannot be regenerated from secret_hash, so either the stored response holds it - making the idempotency record a secret at rest whose retention is now the secret's exposure window - or a replay returns the key metadata without the secret and the documentation says a lost response is resolved by listing keys and revoking the orphan. The second is the safer default precisely because credentials are listable and revocable.
  5. Set retention from the client's retry budget, not from a round number: the record must outlive the SDK's maximum total retry duration, so twenty-four hours is defensible if that budget is minutes. After expiry the key is reusable and a very late retry creates a second credential, which is acceptable here only because the object is listable and revocable, and would not be for an unlistable side effect. Expire with a scheduled delete on an index over created_at.

Worked solution (20 min)

  1. Write the DDL for the idempotency record, including the unique constraint that makes the concurrent case impossible rather than unlikely.
  2. Write the four outcomes - fresh, replay-completed, replay-in-flight, fingerprint-mismatch - as a decision table with the HTTP status for each.
  3. Decide what a replay returns for the plaintext secret and write the exact sentence the API reference has to carry about it.
  4. Pick a retention and justify it from the client library's own maximum retry duration rather than from a round number.

Expected result: Unique (tenant_id, idempotency_key); the record inserted in the same transaction as the credential with ON CONFLICT DO NOTHING deciding the winner; 200 with the stored response, 409 for in-flight, 422 for a fingerprint mismatch; a documented rule that the plaintext secret is not replayed and a lost response is resolved by listing and revoking; retention exceeding the SDK retry budget, for example twenty-four hours.

Verify it yourself

  • Two concurrent requests with the same key must leave exactly one row in tenant_api_key. If the logic is select-then-insert, they will not.
  • A retry with a mutated body must not return 200. If it does, the fingerprint is not being compared or is not covering the fields that matter.
  • The retention you chose must be strictly longer than the client library's maximum total retry duration. Look that number up rather than assuming it.

Follow-up

  • Two requests with the same key arrive concurrently. Show the exact statements and say which one loses, and how it finds out.
  • The client receives a timeout, retries, and gets 409 in-flight. What should the client library do next, and for how long?
  • How does the design change if the created object is not listable - a one-off payout, say - so an orphan cannot be found afterwards?

Webhook workers leak until OOM and drop in-flight deliveries

Medium · memory leak · eviction · leases

webhook-delivery workers grow from 400 MB to a 2 GB limit over about 36 hours, are OOM-killed, restart, and repeat. Each restart abandons in-flight attempts, so webhook_delivery rows sit in in_flight until their leases expire and the backlog spikes. The live set measured after a forced full collection also grows. The fleet serves tens of thousands of subscriptions, several thousand of which have been failing for weeks. Give an ordered checklist, the measurement separating retention from fragmentation, and the fix.

Approach

  1. Separate the two failure shapes with one measurement: track resident set size against the live set after a forced full collection. A live set that climbs monotonically is retention; a flat live set under a rising RSS is fragmentation, off-heap or native allocation, or an allocator that never returns pages. The stated symptom puts this in the first category, which rules out allocator tuning as a fix.
  2. Characterise the curve rather than the total. Growth linear in uptime implies an unbounded structure keyed by something that keeps arriving; step growth implies buffering a large object. Correlate the slope against event rate and separately against the count of distinct subscriptions seen, because those two diverge and only one of them will fit.
  3. Diff two heap snapshots an hour apart by retained size grouped by dominant root, not by allocation count, which is dominated by short-lived objects and will point at the wrong thing.
  4. Expect a per-subscription map with no eviction: circuit-breaker or backoff state created on first failure and never removed, so the retained set grows with endpoints that have ever failed, and the several thousand permanently dead endpoints hold theirs forever.
  5. Fix in two places. Bound the in-memory structure with a size-capped LRU or a TTL keyed on last use, and move state that must survive a restart onto the subscription or webhook_delivery row, since the worker holding it in memory is exactly why a restart loses it.
  6. Repair the second-order damage separately, because it will outlive the leak: workers claim by compare-and-set with leased_until, so a bounded lease returns in_flight rows to pending on a known schedule, and a graceful shutdown releases leases instead of waiting them out.

Follow-up

  • The backlog spike after a restart is itself a thundering herd against customer endpoints. What stops the recovery from becoming a second incident?
  • Suppose the live set had been flat while RSS still climbed. Name two causes and the measurement that separates them.
  • How would you size the LRU, and what does a miss on an evicted circuit-breaker entry cost a customer whose endpoint is down?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

  • Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
  • Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
  • Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

  • Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
  • Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
  • Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

  • Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
  • Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
  • Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

  • Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
  • Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
  • Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

  • Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
  • Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
  • Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

  • Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
  • Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
  • Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

  • Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
  • Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
  • Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

The failure that was yours

Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.

Own the incident where invoices undercounted metered usage

Medium · incident response · at-least-once · billing correction · postmortem

A metering consumer acknowledged each batch before committing the fold into usage_rollup_hourly. A rolling deploy restarted consumers mid-batch for two hours; roughly 1.4M usage_event rows were acknowledged and never folded, and 61 invoices sealed against the resulting rollups before anyone noticed. Take the owner's role. Describe an incident of comparable blast radius you owned: how it surfaced, the query that sized the loss, what you stopped first, and how the money was corrected. Give a wall-clock timeline and one thing you got wrong while it was still live.

Approach

  1. Open with the invariant that broke and the direction of the error, because they determine everything else: acknowledging before committing makes the consumer at-most-once, so this loses events rather than duplicating them, and loss raises no error anywhere. A listener who hears 'we lost revenue silently' knows immediately why detection took two hours.
  2. Size it with a stated reconciliation rather than an adjective: sum(quantity) from usage_event grouped by (tenant_id, sku, hour of occurred_at) over the window, against usage_rollup_hourly.quantity_sum on the same keys, filtered to environment='production' because staging and sandbox are metered but not billed. Then bisect by hour and tenant until single cells explain the gap. Say how long that ran and whether a replica could serve it while the incident was live.
  3. Separate mitigation from fix and say which came first. Mitigation is holding the sealing job, because a sealed row is frozen by design and every minute of sealing converts a recoverable rollup into an invoice correction. The fix is moving the acknowledgement after the commit, which re-introduces duplicates that the dedup check on (tenant_id, idempotency_key) must now absorb.
  4. State the correction path in the domain's own terms: sealed periods are never edited, so each affected tenant gets an adjustment line on the next invoice with kind='adjustment' and voided_by_line_id pointing at the line it reverses, priced against the same rate tier and carrying the watermark it priced against. That is four separate numbers — tenants affected, minor units, the cycle the adjustment lands in, and when customers were told.
  5. Close on one prevention control with its cost, not five: a per-hour reconciliation comparing raw sum to rollup sum that pages above a threshold. Name the threshold and the false-page rate you accepted, because a detector nobody will keep staffed is not prevention.
  6. Name a mistake you made inside the response window — the wrong first hypothesis, a mitigation that made it worse — rather than a design mistake from six months earlier. That is the part candidates rehearse away and interviewers weight heavily.

Follow-up

  • Your fix moves the acknowledgement after the commit. What breaks now, and what absorbs it?
  • One undercharged tenant has since churned. Do you bill them, and who decides?
  • How would you have caught this in ten minutes instead of two hours, and what would that detector cost you in pages per week?

Ship metered billing with a named deduplication horizon

Medium · technical debt · deduplication · deadline pressure · detectors

Metered billing must be on in three weeks. usage_event is partitioned daily, so its unique index must include the partition key and deduplicates only within a day: a producer retry that crosses midnight, or a replay run a week later, gets through. A cross-partition dedup store is two weeks you do not have. Describe shipping with debt you named in advance: what you shipped, what you wrote down, the detector you added, the trigger and date for paying it off, and what you would have refused to ship under the same pressure.

Approach

  1. Show you can separate the two kinds of debt, because that distinction is what the question actually probes. Debt that costs engineering time later is shippable on a deadline. Debt that silently corrupts a number a customer gets charged for is not shippable unless the corruption is detectable, and detectability is the whole negotiation.
  2. Make the exposure narrow and measured rather than gestural. The hole is duplicates whose occurrences straddle a UTC day boundary, plus any replay older than partition retention. Measure it before arguing about it: how often an idempotency_key recurs at all, and the distribution of the gap between first and last occurrence. If the ninety-ninth percentile of that gap is four minutes, the residual risk is a small band around midnight and you can say so numerically.
  3. Add the detector before the feature, not after. A nightly job counting keys that appear in more than one partition is one grouped scan over recent partitions, and it converts a silent overcount into a page. State what it costs to run and what it fires on.
  4. Buy the cheap half of the real fix immediately: extend partition retention so the dedup horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. That reframes retention as a correctness parameter rather than a storage cost, which is the sentence you need on record before someone optimises the bill.
  5. Make repayment mechanical instead of aspirational: a dated entry with a named owner, plus a threshold that pulls the date forward — first detector hit above N events, or first customer dispute. Debt with a trigger gets paid; debt with only a date does not.
  6. Answer the second half honestly by naming what you would refuse under identical pressure: the sealing path, because a sealed row is frozen and a wrong number there stops being a bug and becomes an adjustment line, a dispute and an audit question.

Follow-up

  • The detector fires on forty duplicate events for one tenant, and two of their invoices have already sealed. What happens next?
  • Whom did you tell that the billing numbers had a known hole, and in what words?
  • Finance asks you to cut storage by shortening partition retention. What do you say, and to whom?

Unblock an engineer on a job run that finished twice

Easy · mentoring · fencing tokens · lease expiry · debugging method

An engineer two weeks into the team brings you a job_run row showing status succeeded with an exit_code written by a worker declared dead ten minutes earlier; the retry attempt also shows succeeded. They have spent a day adding logging and are no closer. You have twenty minutes and you do not want to take the keyboard. Describe how you unblock someone: the question you ask first, what you let them find themselves, the concept you name and when, and how you check the next day that they own the fix rather than having watched you produce it.

Approach

  1. Ask what they expect rather than what they see: which statement set status to succeeded, and what did it check before writing? That question points directly at the update's WHERE clause, which is where the answer lives, and it costs them nothing to answer, so it does not read as a test.
  2. Let them build the timeline themselves from the row: queued_at, started_at, leased_until, finished_at and worker_id, on both the original run and the retry. Two different worker_ids with a lease expiry between them tells the whole story, and they will see it before you say it.
  3. Name the concept once the evidence has earned it. A lease bounds time; it does not prevent a write. The store has to reject a stale writer, which means the update carries a fencing token the row compares — update job_run set status = 'succeeded' where run_id = $1 and lease_token = $2 and status = 'running' — and a long garbage-collection pause or a brief partition is enough to produce what they are looking at.
  4. Point at the second, less obvious half and let them decide it: 'lost' exists in the status enum precisely so a run whose worker vanished is not recorded as failed, because failed asserts an outcome nobody observed and the system then bills and retries on that assertion. Ask them what these two rows should have said.
  5. Leave them with the next step rather than the patch — a test that kills the first worker after the sandbox exits and before the row is written — and say when you are available again, so the offer is real rather than polite.
  6. Check ownership the next day by what they produced, not by asking if it went well: a test that reproduces the window proves they understood it; a test that only asserts the new WHERE clause proves they copied it. Ask them to explain it to a third person and listen for whether the explanation is theirs.

Follow-up

  • They propose a longer lease instead of a token. What do you say, and what breaks when legitimate runs last thirty minutes?
  • How can you tell whether your explanation landed or they simply deferred to you?
  • The same engineer hits a variant of this next month. What did you fail to teach the first time?