Interview guides · Amex GBT

Amex GBT Software Engineer Interview Questions & Guide 2026

A prep plan for Amex GBT software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

A Software Engineer at Amex GBT plays a pivotal role in maintaining and evolving the complex technological ecosystem that powers global business travel. You are not just writing code; you are building scalable, reliable solutions that manage real-time travel logistics, booking workflows, and financial data for corporate clients worldwide. The work is characterized by high-stakes reliability and the need for precision, as your contributions directly impact the efficiency and experience of business travelers across the globe.

Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.

Focus areas

Make every write idempotent under client retries
Build at-least-once pipelines with explicit deduplication horizons
Keep money in integer minor units

Overview

This role offers a unique intersection of high-scale enterprise engineering and the agility of a modern, product-focused organization. You will likely work within cross-functional teams, collaborating closely with product managers and other engineers to address challenges ranging from performance optimization in booking engines to the integration of complex web services. Whether you are working on the front-end interface or the back-end processing logic, your work is critical to the company’s ability to deliver seamless, end-to-end travel management solutions.

While the interview experience can vary by region and team, candidates consistently highlight that the role requires a blend of deep technical fundamentals and the ability to articulate the "why" behind their architectural decisions.

The Amex GBT interview process

Amex GBT candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Rounds

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Managerial Discussion

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Holding money in a floating-point type, or rounding it more than once

Binary floating point cannot represent 0.01 or 0.1 exactly, so sums drift and two code paths that should agree disagree by cents nobody can trace back. The fix is integer minor units or an exact decimal type end to end, with sub-cent rates expressed as scaled integers such as micro-units, because a per-request price genuinely is smaller than a cent. The second half of the trap is rounding position: rounding each line and then summing gives a different total from summing and rounding once, and half-up and half-even diverge systematically across many lines, so rounding must happen at one named place and every downstream reader must carry the rounded value rather than recompute it from quantity and rate.

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Going silent while thinking

Narrate the candidates and why you are discarding them, even in fragments: sorting first would make this a two-pointer scan, but it destroys the original indices, which the output needs. From the other side of the table, a candidate thinking hard and a candidate stuck are indistinguishable until one of them speaks.

Tests that assert on the implementation rather than the behaviour

Assert on what a caller can observe, not on the number of internal calls or the shape of a private field. A test that breaks on every refactor but still passes when the answer is wrong costs more than it protects.

Questions & practice

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Order a job dependency graph and find its critical path

Medium · topological-sort · dag-longest-path · cycle-detection · critical-path

A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.

Approach

Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order: earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E).
Second pass in reverse topological order for latest_finish, then slack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain. slack[v] = 0 is exactly the statement that some longest path runs through v; equivalently, the longest path through v has length T - slack[v].
The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by min(d, T - L_avoid(v)), where L_avoid(v) is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, while min(10, 15 - 14) predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan is max(T - d, L_avoid(v)).
Compute L_avoid(v) the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, since L_avoid(v) only ever matters through that max: set duration[v] := 0, recompute the makespan as T0(v) = max(T - duration[v], L_avoid(v)), and the gain is min(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.

Follow-up

Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Worked solution (25 min)

Write both key tuples down before any code: dedup key (tenant_id, idempotency_key), cell key (tenant_id, workspace_id, sku, hour_start), with hour_start derived from occurred_at in UTC.
Build a 10,000-row fixture containing one event duplicated three times under the same idempotency_key, two events sharing an idempotency_key across different tenant_id values, one event whose occurred_at is two hours before its ingested_at, and one staging event inside an otherwise production cell.
Fold it and assert each of those four expectations separately rather than eyeballing a grand total.
Re-run with the input shuffled and diff the output files.
Size the dedup set for 250M keys using the load-factor arithmetic and write the number down next to the fixture.

Expected result: The triplicate contributes one event and its quantity once. The two same-key, different-tenant events both count, because the dedup key is the pair. The late event lands in the hour of its occurred_at while that cell's source_max_ingested_at advances to the later timestamp. The staging event is included or excluded per the stated filter and never silently.

Verify it yourself

Shuffled input produces byte-identical cells, including event_count and source_max_ingested_at.
The sum of quantity_sum across all cells equals the sum of quantity over the distinct dedup keys, computed independently.
Swapping the accumulator to binary64 makes a fixture of 0.1-valued quantities disagree with the exact sum. That is a failing test, not a rounding note.

Follow-up

A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
What makes a re-run over the same day produce byte-identical rollups?

Find the join that inflates every invoice total

Medium · join fan-out · aggregation grain · sql correctness

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

Approach

Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Follow-up

Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
The same report is needed per tenant per month. What index makes the line CTE cheap?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Worked solution (25 min)

Write the DDL: unique (invoice_id, tenant_id) on invoice, the composite FK from the line, and a comment on each denormalised column saying whether it is a snapshot or a cache.
Attempt to insert a line whose tenant_id differs from its invoice's and confirm the foreign key rejects it.
Rename a tenant, re-render a sealed invoice, and confirm the rendered name is the one stored on the header.
Recompute amount_minor from quantity times unit_price_micros for a thousand synthetic lines rounding half-up, sum both ways, and record the divergence from the stored half-even values.

Expected result: tenant_id kept and enforced by a composite foreign key to (invoice_id, tenant_id); amount_minor kept because rounding happens once with a stated mode; legal name and billing address snapshotted onto the invoice header and never updated; mutable operational attributes left normalised. Each decision paired with its read pattern, its anomaly, and the mechanism that prevents the anomaly.

Verify it yourself

A line cannot be attached to an invoice belonging to another tenant, and the rejection comes from the database rather than the application.
Re-rendering a sealed invoice after editing the tenant record produces identical output.
Rounding per line and rounding once at the end differ measurably over a thousand lines, which is the argument for storing the value.
No column on the line is recomputed by any reader.

Follow-up

Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

What is the difference between a composite key and a unique key?

Medium · system design and architecture

Approach

Name the failure you are designing for, then the recovery path.
Name the read and write paths separately; they rarely have the same bottleneck.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

How would you go about solving a workflow performance problem?

Medium · system design and architecture

Approach

Name the read and write paths separately; they rarely have the same bottleneck.
Choose a partition key and say what query it makes expensive.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

Explain the architecture of a SaaS product you have worked on and the critical features you implemented.

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
State the consistency you need, and where you are willing to be stale.
Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

These questions assess your ability to design scalable systems and understand the implications of your architectural choices.

Medium · system design and architecture

Approach

Fix the scope first: who calls this, how often, and what they do when it fails.
State the consistency you need, and where you are willing to be stale.
Choose a partition key and say what query it makes expensive.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

These questions test your command of language-specific features, data structures, and fundamental engineering principles.

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Write a program to output the Fibonacci series given an array.

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Design the batch ingest endpoint metering agents retry into

Hard · batch api · partial failure · deduplication

A customer-run agent posts usage events in batches of up to 1,000 to metering-ingest with a 30-second timeout and at-least-once retry of the whole batch. Each event carries event_id, idempotency_key, sku, quantity and occurred_at; the server adds ingested_at, and usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). Design the endpoint: the request shape, what the response says when 900 events are new, 90 are duplicates and 10 are malformed, the status code, and the agent's algorithm on timeout. Then state the deduplication horizon and justify it against that unique constraint.

Approach

Fix the per-item outcome taxonomy first, because the status code follows from it: accepted, duplicate, and rejected with a permanent code. A duplicate is a success; reporting it as an error makes the agent either re-send revenue it already delivered or drop it.
Allow only permanent failures per item. A transient per-item failure inside a 200 invites the agent to discard that event, so anything transient escalates to a 5xx for the entire batch. A 200 is then a promise that every event not marked rejected is committed and durable.
Return 200 with a results array aligned by index and carrying the event id, so the agent can retry precisely the subset that needs it and quarantine the ten malformed events instead of hard-looping a poison batch forever. Cap the batch at 1,000 items and a byte size, with 413 beyond it and 429 with Retry-After for backpressure.
Deduplicate per event, never per batch: the agent may split, merge or reorder a retried batch, so a batch-level key matches nothing on the second attempt. The key is (tenant_id, idempotency_key), and the tenant comes from the resolved credential; a tenant id present in the body is compared against it, never trusted.
Size the horizon as a correctness parameter. The unique index includes the partition key, so it deduplicates only within one day: a retry that crosses midnight, or a replay run a week later, passes straight through it. A separate dedup store keyed (tenant_id, idempotency_key) with a TTL exceeding the agent's maximum retry window plus the longest replay you intend to support is what actually enforces the invariant, which makes its retention a correctness setting rather than a cost knob.
Order the commit against the response and the acknowledgement: commit then respond at the endpoint, and downstream commit the fold then acknowledge the message. Acknowledging first turns a crash into silently lost revenue with no error raised anywhere.

Worked solution (40 min)

Write the request body schema with the batch envelope and one event, and state which fields the server assigns rather than accepts.
Write the 200 response for the 900/90/10 case, showing three result entries, one of each outcome, with the rejected one carrying a permanent code.
Write the rule separating per-item rejection from whole-batch failure, and list which conditions fall on each side.
Compute the dedup horizon from the agent's retry window plus the replay window you support, and say where the dedup state lives and how it ages out.
Write the agent's pseudocode for timeout, 5xx, 429 and 200-with-rejections, four branches, and mark which branch may drop an event.
Trace the crash between commit and response, and between fold and acknowledgement, and say what each produces.

Expected result: A 200 with an index-aligned results array where duplicates count as success and rejections are permanent-only, transient problems escalated to a whole-batch 5xx, per-event deduplication keyed on (tenant_id, idempotency_key), and a published horizon larger than retry plus replay because the partitioned unique index only covers one day.

Verify it yourself

The agent can distinguish 'already counted' from 'never counted' for every event in the batch.
Exactly one branch of the agent's algorithm ever discards an event, and it is the permanent-rejection branch.
A retry crossing midnight is still deduplicated, and the mechanism named is not the unique index.
No crash point in the pipeline both commits work and loses it without an error.

Follow-up

The agent times out at 30 seconds having received nothing. What exactly does it do next, and what in your design makes that safe?
Ten events are rejected every hour for a week and nobody notices. What does the endpoint owe the customer beyond a per-item 4xx code?
A replay pushes 40 million events through this endpoint in an hour. Which part of your design degrades first?

One tenant's counter writes stall the whole connection pool

Medium · lock contention · connection pool · multi-tenancy

A change that made a per-tenant usage counter correct now produces site-wide latency whenever one large tenant writes: unrelated endpoints time out waiting for a connection while database CPU stays low and no statement is slow. The change wraps the counter update in a transaction that takes SELECT ... FOR UPDATE on one row, calls an external pricing service, then updates and commits. Give an ordered checklist, the arithmetic that bounds that tenant's write rate, and three repairs with the cost each one accepts.

Approach

Separate waiting from working. Low database CPU alongside high application latency points at a queue, so instrument connection-acquisition wait separately from query execution time; that queue forms in the application and is invisible in database metrics, which is why the database looks healthy throughout.
Confirm the lock rather than assuming it: sample waiting sessions and group by wait event, relation and tuple. Contention concentrated on one tuple belonging to one tenant is the signature; a deadlock would instead show the database aborting transactions after its detection timeout, which is not happening here.
Do the arithmetic out loud. Throughput on a serialised row is one divided by the lock hold time, and the hold spans the external call, so a 20 ms pricing call caps that tenant near 50 writes per second no matter how many pods run. Every waiter also holds a pooled connection while it queues, so the shared pool drains and unrelated tenants fail at acquisition.
Repair one: shrink the critical section to a single statement with the price resolved before the transaction opens. Cost is a stale price for the duration of one request and a second round trip; benefit is a hold time measured in the database's own execution time.
Repairs two and three change where the contention lives rather than how long it is held. Sharding the counter into per-(tenant, bucket) rows and summing on read multiplies write throughput by the shard count, at the cost of an aggregate on every read and a shard count you must size against the largest tenant rather than the median. Accumulating in memory and flushing periodically removes the per-write round trip entirely, paid for with a bounded loss window on crash, which is acceptable for a rate limiter and not for a billing counter.
Contain independently of which repair wins: a separate pool or per-tenant concurrency cap for this write class, a statement timeout low enough that a pathological query dies before it accumulates waiters, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks.

Follow-up

What would a genuine deadlock look like here, which two code paths would produce one, and how does the database's response differ from what you observed?
If a transaction-pooling proxy sits in front of the database, which of your three repairs changes behaviour, and what stops working that would have worked on a direct connection?
The counter also enforces a quota. Why is SELECT the count and then INSERT still wrong after you have fixed the contention?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

Run the same sub-skill decomposition on the second-largest gap in half the time.
Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

Tell us about the projects you have handled and your specific contributions.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

How do you solve conflicts in a work environment?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Unblock an engineer on a job run that finished twice

Easy · mentoring · fencing tokens · lease expiry · debugging method

An engineer two weeks into the team brings you a job_run row showing status succeeded with an exit_code written by a worker declared dead ten minutes earlier; the retry attempt also shows succeeded. They have spent a day adding logging and are no closer. You have twenty minutes and you do not want to take the keyboard. Describe how you unblock someone: the question you ask first, what you let them find themselves, the concept you name and when, and how you check the next day that they own the fix rather than having watched you produce it.

Approach

Ask what they expect rather than what they see: which statement set status to succeeded, and what did it check before writing? That question points directly at the update's WHERE clause, which is where the answer lives, and it costs them nothing to answer, so it does not read as a test.
Let them build the timeline themselves from the row: queued_at, started_at, leased_until, finished_at and worker_id, on both the original run and the retry. Two different worker_ids with a lease expiry between them tells the whole story, and they will see it before you say it.
Name the concept once the evidence has earned it. A lease bounds time; it does not prevent a write. The store has to reject a stale writer, which means the update carries a fencing token the row compares — update job_run set status = 'succeeded' where run_id = $1 and lease_token = $2 and status = 'running' — and a long garbage-collection pause or a brief partition is enough to produce what they are looking at.
Point at the second, less obvious half and let them decide it: 'lost' exists in the status enum precisely so a run whose worker vanished is not recorded as failed, because failed asserts an outcome nobody observed and the system then bills and retries on that assertion. Ask them what these two rows should have said.
Leave them with the next step rather than the patch — a test that kills the first worker after the sandbox exits and before the row is written — and say when you are available again, so the offer is real rather than polite.
Check ownership the next day by what they produced, not by asking if it went well: a test that reproduces the window proves they understood it; a test that only asserts the new WHERE clause proves they copied it. Ask them to explain it to a third person and listen for whether the explanation is theirs.

Follow-up

They propose a longer lease instead of a token. What do you say, and what breaks when legitimate runs last thirty minutes?
How can you tell whether your explanation landed or they simply deferred to you?
The same engineer hits a variant of this next month. What did you fail to teach the first time?