Interview guides · AfterShip

AfterShip Software Engineer Interview Questions & Guide 2026

AfterShip software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

A Software Engineer at AfterShip plays a pivotal role in maintaining and scaling the infrastructure that powers global e-commerce logistics. As a member of the engineering team, you are responsible for building robust, high-performance solutions that handle millions of tracking events, ensuring that merchants and their customers receive accurate, real-time data.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

Evolve APIs without breaking pinned SDK clients
Scope every query and cache key by tenant
Build at-least-once pipelines with explicit deduplication horizons

Overview

The work is technically demanding and strategically significant, as you are contributing to a platform that serves a massive user base across various international markets. You will work on complex distributed systems, participate in the full software development lifecycle, and collaborate closely with product and operations teams to solve real-world problems.

Success in this role requires more than just coding proficiency; it demands a deep understanding of system reliability, a commitment to engineering excellence, and the ability to thrive in a fast-paced environment. Whether you are optimizing APIs or architecting new features, your contributions directly influence the efficiency and growth of the AfterShip ecosystem.

The AfterShip interview process

AfterShip candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Assessments

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Deep-Dive Discussions

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Final Discussions

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Questions & practice

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Worked solution (30 min)

Define the four structures explicitly: queues: subscription_id -> deque[delivery], ready: min-heap of (next_attempt_at, subscription_id), inflight: set[subscription_id], breaker: min-heap of (half_open_at, subscription_id).
Write down the invariant you will assert after every operation: a subscription appears in at most one of ready, inflight and breaker, never in two.
Implement due, complete and insert, then simulate 200,000 subscriptions with Zipf-distributed queue depths totalling 20 million deliveries.
Add one endpoint that always times out after 10 seconds and one that always answers in 20 ms, then measure the fast endpoint's throughput with and without the per-endpoint breaker.
Instrument heap size across the run.

Expected result: Heap size stays at or below 200,000 regardless of the 20 million deliveries. The fast endpoint's throughput is unaffected by the dead one once the breaker trips. The dead endpoint's deliveries accumulate in their own deque and cost exactly one heap entry.

Verify it yourself

The asserted invariant never fires under concurrent workers.
Replaying the same delivery set twice produces identical per-subscription ordering, even though the interleaving across subscriptions differs.
Replacing full jitter with fixed backoff makes retry attempts cluster visibly in a histogram of attempt timestamps.

Follow-up

One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Seal an hour under late data with bounded memory

Hard · watermarks · late-data · quantile-sketch · conditional-write

Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.

Approach

Two clocks, two jobs. Bucket by occurred_at, because that is the hour the customer is billed for, and advance the watermark on ingested_at, because that is what the fold has consumed and what source_max_ingested_at records. Conflating them is what makes late data invisible.
The global watermark is the min over partitions of each partition's committed ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past.
Choose the lateness L from the measured distribution of ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters.
Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
Hold open hours in a min-heap keyed by hour_start. When the watermark advances, pop every hour with hour_end + L < W and seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct (tenant, workspace, sku) keys, so cap the number of simultaneously open hours and spill the oldest into usage_rollup_hourly as status='open' with a revision bump. While an hour is open the row is upsertable, so the store is your overflow.
The seal itself is a conditional write: update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, and source_max_ingested_at is what proves it arrived afterwards.

Follow-up

A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
How would you detect that L has drifted before it costs you an hour's quantity?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
What makes a re-run over the same day produce byte-identical rollups?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Follow-up

Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Worked solution (25 min)

Seed a tenant with 19 running rows, then fire 8 concurrent sessions each running the select-then-insert, and count the resulting running rows.
Repeat at REPEATABLE READ and confirm the count still exceeds 20.
Repeat at SERIALIZABLE, count the 40001 aborts, and note that without a retry loop those requests fail rather than queue.
Implement the atomic counter UPDATE, re-run the 8-way test, and confirm exactly 20 running rows with zero over-admissions.
Kill a worker mid-run, let the lease expire, and check whether the slot comes back without intervention.

Expected result: Write skew identified; read committed and repeatable read both permit it; serialisable prevents it at the price of 40001 retries; the working read-committed version is update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running, whose zero-row result is the rejection, plus a reconciliation path for runs that terminate as 'lost'.

Verify it yourself

At 8x concurrency against a cap of 20, running rows never exceed 20.
The rejection returns zero rows rather than raising, so it is distinguishable from an error in the caller.
Killing a worker does not permanently consume a slot.
The counter and select count(*) from job_run where status = 'running' agree after reconciliation.

Follow-up

Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

What are the fundamental differences between various data structures like Stacks, Queues, and Arrays, and when should you choose one over another?

Medium

Approach

Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Can you discuss the importance of HTTP status codes and how you ensure API reliability?

Medium

Approach

Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Metering ingest that survives a six-hour producer replay

Hard · idempotency · dedup horizon · partitioning · watermarks

metering-ingest consumes usage events at-least-once - 250M/day, 10-40k/second at peak - and folds them into usage_rollup_hourly keyed (tenant_id, workspace_id, sku, hour_start). usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). A producer outage ends in a six-hour replay that re-sends events already ingested, some of whose originals crossed midnight. Design the consumer: partitioning, where the acknowledgement sits relative to the commit, the deduplication horizon and its storage cost, and how the rollup watermark advances. Nothing may be double-counted and nothing may be silently dropped.

Approach

Choose the acknowledgement position deliberately and name what each choice costs. Acknowledging after the fold commits makes the consumer at-least-once: a crash between the two replays the batch and produces duplicates, which are ordinary and absorbable. Acknowledging first makes it at-most-once: a crash between the two drops revenue with no error raised anywhere and no way to detect it later. Take at-least-once and design everything downstream to absorb duplicates.
Put the dedup and the fold in one transaction so there is no window between them. Insert the batch into usage_event with ON CONFLICT DO NOTHING, take the rows actually inserted, and fold only those into usage_rollup_hourly with an upsert on (tenant_id, workspace_id, sku, hour_start) bucketed by occurred_at, not ingested_at. A batch of about 2,000 rows is one round trip and one index probe per event.
Attack the partition-key flaw head on: the unique index includes ingested_day because a unique index on a partitioned table must contain the partition key, so the same (tenant_id, idempotency_key) re-sent after midnight is a different index entry and passes. Deduplicate instead against a store keyed (tenant_id, idempotency_key) with no date component, whose horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. At 14 days that is 250M x 14 = 3.5 billion keys, which is a dedicated key-value store, not a larger index on the same table. The alternative - partitioning usage_event on (tenant_id, occurred_day) so the natural key is stable - fixes dedup but loses pruning on ingest time and makes retention by dropping partitions awkward.
Partition the consumer by hash of tenant_id so one tenant's replay stalls only its own partitions, and give replay traffic a separate lower-priority lane so live ingest keeps its latency. The cost is explicit: that tenant's watermark lags while the replay drains, and everything gated on the watermark waits for it.
Define the watermark as a property of committed work, not of wall-clock time: per partition it is the largest occurred_at such that every event with a smaller occurred_at has committed, and the sealing decision uses the minimum across partitions. Record source_max_ingested_at on every rollup row so any number can prove what it did and did not include, and keep restatement legal only while status = 'open' - after sealed_at the value is frozen and a late event becomes an invoice adjustment instead.

Worked solution (40 min)

Write the consumer loop in pseudocode with the acknowledgement after the commit, then annotate each line with what is lost or duplicated if the process dies exactly there.
Size the dedup store: events/day x horizon_days keys, bytes per key including the tenant prefix, and the resulting memory or disk. Compare that cost against simply extending retention on the partitioned table and say why the latter does not fix the problem.
Take one event ingested at 23:59:58 and replayed at 00:00:04 and work out its fate under (a) the partitioned unique index alone and (b) the separate dedup store.
Write the per-partition watermark formula, then what the seal uses, then what a single stalled partition does to sealing.

Expected result: An at-least-once consumer acknowledging after the commit; deduplication on (tenant_id, idempotency_key) in a store whose horizon of at least 14 days exceeds max retry window plus supported replay length, sized at roughly 3.5 billion keys; tenant-hashed partitions with replay on a separate lane; a per-partition watermark with sealing gated on the minimum across partitions; source_max_ingested_at written on every rollup row.

Verify it yourself

The midnight-crossing retry must be rejected by whatever you chose. If the table's unique index is your only dedup, it is accepted and the invoice is wrong by exactly that event.
You should be able to name a distinct failure for acknowledge-before-commit and for acknowledge-after-commit. If both sound the same, the position was not actually chosen.
quantity_sum for any (tenant, workspace, sku, hour) must be reproducible by re-aggregating usage_event filtered to ingested_at <= source_max_ingested_at. If it is not reproducible, the rollup is an opinion rather than a projection.

Follow-up

The dedup store is lost entirely. What can you still guarantee, and how do you rebuild it from what remains?
A replay delivers events for an hour that is already sealed. Trace exactly what happens to them, row by row.
One partition is stuck on a poison message, so the minimum-across-partitions watermark never advances and no tenant can be sealed. What is your escape hatch and what does it cost in correctness?

A rare job-run overwrite that logging makes disappear

Hard · fencing tokens · leases · concurrency

About one job run in fifty thousand records billable_seconds matching no observed sandbox lifetime, and a few rows show worker_id changing after finished_at was already set. It does not reproduce: debug logging around the terminal write made it vanish for two weeks before it returned. Runs last from 200 ms to 30 minutes, the lease is 60 seconds and is renewed while a run executes. Give an ordered checklist, the mechanism, and a fix that makes the illegal write impossible rather than merely rarer.

Approach

Mine the evidence instead of chasing a repro: select rows where updated_at is later than finished_at, or where a terminal status was written twice, and join them to attempt history to recover both writer identities. The defect has already happened tens of times and the rows are the recording.
State the signature before measuring it. If the cause is a lease that expired while the original worker was stalled, affected runs should cluster where the gap between the last renewal and the terminal write exceeds the lease, and should correlate with worker pause metrics rather than with workload shape.
Read the disappearance honestly. Logging inside the window changed the timing and lowered the probability; it is evidence about how narrow the window is, not a fix. Reproduce by widening the window on purpose, shortening the lease and injecting a pause between sandbox exit and the terminal write, rather than by adding more instrumentation.
Name the mechanism precisely: the lease expires during a stall such as a long garbage-collection pause or a brief partition, the run is re-dispatched, and the original worker then wakes and writes its terminal state over the new attempt's row. A lease alone cannot stop this, because the check and the write are separated by the stall.
Fix by fencing the write itself: UPDATE job_run SET status = $2, finished_at = $3, billable_seconds = $4 WHERE run_id = $1 AND status = 'running' AND lease_token = $5, with zero rows affected interpreted as having been fenced rather than as success. The token lives on the row so the store arbitrates, not the worker's memory.
Keep the state machine honest: a retry inserts a new row pointing at parent_run_id rather than resetting the old one, and a run whose worker vanished terminates as lost with billable_seconds null, because recording failure asserts an outcome nobody observed and then bills and retries on that assertion.

Follow-up

The supervisor also emits a usage event on completion. What does the fenced worker do about the event it already emitted, and how does metering absorb it?
Lease renewal is itself a network call. What happens when a renewal times out, and how does the worker decide whether it still holds the lease?
Why is lengthening the lease past the longest legitimate run the wrong lever, and what breaks if you do it anyway?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Can you describe a time you worked effectively within a team to overcome a difficult project hurdle?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Why did you leave your previous organization, and what are you looking for in your next career move?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

How do you gather requirements from stakeholders, and how do you define when a task is officially "done"?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

What work are you most proud of in your portfolio, and what was your specific contribution to that project?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?