Morgan Stanley Software Engineer Interview Guide

The 60-second brief

As a Software Engineer at Morgan Stanley, you sit at the intersection of cutting-edge technology and global finance. The firm relies on its engineering teams to build, scale, and secure platforms that handle billions of dollars in daily transactions, drive high-frequency trading networks, manage wealth for millions of clients, and power complex risk-analytics engines. Software engineers here do not merely write code to meet specifications; they design resilient, low-latency infrastructure capable of operating flawlessly under high-volume market conditions.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

• Bound blast radius with per-tenant concurrency limits
• Evolve APIs without breaking pinned SDK clients
• Scope every query and cache key by tenant

Overview

Engineers at Morgan Stanley work across a diverse set of technical domains. You might find yourself architecting core microservices in Java or Spring Boot, building real-time data pipelines using Kafka and Spark, or implementing low-latency algorithms in C++. Teams span critical divisions, including Wealth Management Technology, Institutional Securities, Risk Analytics and Platform Delivery (RAPD), and emerging business units developing enterprise blockchain and digital asset tokenization.

The impact of your work in this role is direct and measurable. A minor optimization in your code can result in millisecond improvements for order execution or secure multi-million-dollar trades against unexpected market volatility. Morgan Stanley places a strong emphasis on architectural rigor, system design, data structure efficiency, and code quality, offering engineers a deep, technically challenging environment where scalable problem-solving is the top priority.

The Morgan Stanley interview process

Morgan Stanley candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening Call

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

• Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
• Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
• Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

• Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
• Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
• Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Online Assessment

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Core Technical Rounds

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

• Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
• Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
• Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
• Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

• From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
• Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
• Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Super Day

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

• Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
• Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
• Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
• Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

• Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
• Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
• Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Holding money in a floating-point type, or rounding it more than once

Binary floating point cannot represent 0.01 or 0.1 exactly, so sums drift and two code paths that should agree disagree by cents nobody can trace back. The fix is integer minor units or an exact decimal type end to end, with sub-cent rates expressed as scaled integers such as micro-units, because a per-request price genuinely is smaller than a cent. The second half of the trap is rounding position: rounding each line and then summing gives a different total from summing and rounding once, and half-up and half-even diverge systematically across many lines, so rounding must happen at one named place and every downstream reader must carry the rounded value rather than recompute it from quantity and rate.

Paginating a growing table with limit and offset

Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.

Hardcoding to the sample inputs

Solve the stated problem rather than the two examples; special-casing a literal to make a sample pass is obvious immediately and reads as either a misunderstanding or an attempt to fake progress. If you genuinely cannot generalise yet, say which part is a stub and what would replace it.

Saying 'eventually consistent' without naming the anomaly a user would see

Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.

Questions & practice

Implement an LRU Cache with $O(1)$ time complexity for read and write operations.

Medium · data structures and algorithms

Approach

1. Walk one small example through your approach before writing the whole thing.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Find the shortest path between two nodes in a social network graph using Breadth-First Search (BFS).

Medium · data structures and algorithms

Approach

1. Walk one small example through your approach before writing the whole thing.
2. Choose the data structure from the access pattern, not from familiarity.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• How does this change if the input no longer fits in memory?
• Which test case would catch an off-by-one here?

Given an array of integers and a target sum, return the indices of the two numbers that add up to the target in $O(n)$ time.

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. State the target complexity and say which constraint rules the naive version out.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• How does this change if the input no longer fits in memory?
• Which test case would catch an off-by-one here?

Given a string containing upper and lower case letters, write a function to return uppercase characters that have a corresponding lowercase counterpart present.

Medium · data structures and algorithms

Approach

1. Name the brute-force solution and its complexity before improving on it.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Explain the internal working of HashMap. How are collisions handled, and how does it change in Java 8?

Medium · languages, concurrency and fundamentals

Approach

1. Reach for the cheapest primitive that closes the race, not the broadest lock.
2. Identify the window where an invariant is briefly untrue.
3. Name what is shared across threads and what owns each piece of state.

Follow-up

• What happens if two callers reach this at the same time?
• How would you prove the race exists rather than suspect it?

Explain SOLID principles in detail and show how each is implemented in your preferred programming language.

Medium · languages, concurrency and fundamentals

Approach

1. Identify the window where an invariant is briefly untrue.
2. Distinguish a value from a reference to it, and say which one you handed out.
3. Name what is shared across threads and what owns each piece of state.

Follow-up

• How would you prove the race exists rather than suspect it?
• Where could this allocate more than you expect?

Hold a tenant to a trailing sixty-second request limit

Medium · sliding-window · two-pointer · rate-limiting · tenant-skew

The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.

Approach

1. Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request.
2. Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
3. Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate, prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact.
4. Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (tokens, last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual.
5. Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
6. Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.

Worked solution (25 min)

1. Implement the deque version and instrument the per-request pop count, then confirm total pops equal total pushes over a run.
2. Generate a burst that places R requests in the last 100 ms of one minute and R more in the first 100 ms of the next.
3. Run that burst through the exact deque, a fixed 60-second counter, and the weighted two-bucket estimate, recording admissions in the trailing 60 seconds at every instant.
4. Size the memory as R x active tenants x 8 bytes at R = 1,000 and 200,000 tenants, and compare it against what a token bucket would need.

Expected result: The exact deque never admits more than R in any trailing 60-second window. The fixed counter admits close to 2R across the boundary. The weighted estimate lands between the two on this burst and approaches 2R once the previous window's requests are packed at its end.

Verify it yourself

• Total pops equal total pushes, which is what makes the amortised O(1) claim true rather than merely asserted.
• Sliding the whole burst by 30 seconds changes the fixed counter's result and leaves the deque's unchanged.
• Running two processes each limiting to R/2, against a tenant that sends everything to one of them, shows that tenant throttled at half its contractual limit.

Follow-up

• One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
• Quotas rather than rate limits: the check is select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes.
• How do you return an accurate Retry-After from the exact algorithm without a second scan?

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

1. Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
2. Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
3. Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
4. Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
5. Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Worked solution (20 min)

1. Load about 50k deliveries for one tenant, then walk them with the OFFSET query while a writer inserts 10 rows/second, collecting every returned delivery_id.
2. Compare the distinct ids collected against the set of ids that existed when the walk started, and record the shortfall.
3. Repeat the walk with the keyset query and confirm every pre-existing id is returned exactly once.
4. Run explain (analyze, buffers) on page 1 and page 500 of each form and compare shared buffer hits.

Expected result: A keyset query using the row comparison (created_at, delivery_id) < ($2, $3) with order by created_at desc, delivery_id desc limit 100, backed by (tenant_id, created_at, delivery_id) scanned backwards; plus the statement that OFFSET drops rows under concurrent insert and costs time proportional to the offset, and that updated_at is disqualified because it mutates on retry.

Verify it yourself

• Buffer counts for page 500 are the same order of magnitude as page 1.
• Every id present before the walk is returned exactly once when only inserts run concurrently.
• The query plan shows no Sort node above the index scan.
• Two rows sharing a created_at value to the microsecond are both returned across the page boundary.

Follow-up

• The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
• A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
• The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

Rebuild an hourly rollup with deduplication and late-arrival accounting

Hard · window functions · deduplication · event time vs ingest time · watermarks

From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.

Approach

1. Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed: row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs.
2. Bucket on occurred_at and nothing else, and pin the timezone explicitly. date_trunc('hour', timestamptz) truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argument date_trunc('hour', occurred_at, 'UTC') on PostgreSQL 16 or later, or date_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC' before that. Filter environment = 'production' explicitly, since metering covers three environments and billing covers one.
3. Aggregate to the grain with sum(quantity), count(*) and max(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include.
4. Compute the late share inside the dedup-and-aggregate step as a conditional aggregate, sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows: sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back.
5. Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
6. Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.

Follow-up

• A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
• Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
• Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?

How would you process and find the most frequent word in a massive log file that exceeds available RAM?

Medium · system design and architecture

Approach

1. State the consistency you need, and where you are willing to be stale.
2. Choose a partition key and say what query it makes expensive.
3. Name the failure you are designing for, then the recovery path.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

Explain the overall architecture of a complex system you currently work on, including its failure points and data flow.

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. Choose a partition key and say what query it makes expensive.

Follow-up

• What breaks first when traffic grows ten times?
• How does this behave when that dependency is down for an hour?

An idempotent create endpoint that returns a one-time secret

Easy · idempotency · race conditions · api design

POST /v1/api-keys inserts a tenant_api_key row (tenant_id, workspace_id, name, key_prefix, secret_hash, scopes, status, auth_version, created_at, expires_at) and returns the plaintext secret exactly once, since only its SHA-256 is stored. Write volume is tens per second and clients retry on timeout. Design the idempotency mechanism: what the key is scoped by, where the record lives, its retention, what happens when the same key arrives with a different body, what happens when a retry arrives while the first request is still in flight, and what a replay returns for the secret.

Approach

1. Scope the key by tenant, not globally: uniqueness is on (tenant_id, idempotency_key), or one tenant's key collides with another's and the second caller receives a stored response for a request it never sent. Store a fingerprint of the request alongside it - method, path and a hash of the canonicalised body - so a mismatch is detectable.
2. Let the unique constraint decide the race instead of application logic. In the same transaction as the credential insert, run INSERT INTO idempotency_record (tenant_id, key, request_fingerprint, status) VALUES (...) ON CONFLICT DO NOTHING RETURNING id; no returned row means this is a replay, and the existing record is then read. A select-then-insert here loses to itself under concurrency in exactly the way this endpoint is meant to prevent.
3. Write the three replay outcomes as a decision table rather than as prose: same fingerprint and completed returns the stored response; same fingerprint and still in flight returns 409 with Retry-After, without blocking and without executing; different fingerprint returns 422, because replaying a stored response for a mutated body would tell the caller a credential was created for parameters it never sent.
4. Handle the secret as the part that makes this endpoint different from an ordinary idempotent create. The plaintext cannot be regenerated from secret_hash, so either the stored response holds it - making the idempotency record a secret at rest whose retention is now the secret's exposure window - or a replay returns the key metadata without the secret and the documentation says a lost response is resolved by listing keys and revoking the orphan. The second is the safer default precisely because credentials are listable and revocable.
5. Set retention from the client's retry budget, not from a round number: the record must outlive the SDK's maximum total retry duration, so twenty-four hours is defensible if that budget is minutes. After expiry the key is reusable and a very late retry creates a second credential, which is acceptable here only because the object is listable and revocable, and would not be for an unlistable side effect. Expire with a scheduled delete on an index over created_at.

Worked solution (20 min)

1. Write the DDL for the idempotency record, including the unique constraint that makes the concurrent case impossible rather than unlikely.
2. Write the four outcomes - fresh, replay-completed, replay-in-flight, fingerprint-mismatch - as a decision table with the HTTP status for each.
3. Decide what a replay returns for the plaintext secret and write the exact sentence the API reference has to carry about it.
4. Pick a retention and justify it from the client library's own maximum retry duration rather than from a round number.

Expected result: Unique (tenant_id, idempotency_key); the record inserted in the same transaction as the credential with ON CONFLICT DO NOTHING deciding the winner; 200 with the stored response, 409 for in-flight, 422 for a fingerprint mismatch; a documented rule that the plaintext secret is not replayed and a lost response is resolved by listing and revoking; retention exceeding the SDK retry budget, for example twenty-four hours.

Verify it yourself

• Two concurrent requests with the same key must leave exactly one row in tenant_api_key. If the logic is select-then-insert, they will not.
• A retry with a mutated body must not return 200. If it does, the fingerprint is not being compared or is not covering the fields that matter.
• The retention you chose must be strictly longer than the client library's maximum total retry duration. Look that number up rather than assuming it.

Follow-up

• Two requests with the same key arrive concurrently. Show the exact statements and say which one loses, and how it finds out.
• The client receives a timeout, retries, and gets 409 in-flight. What should the client library do next, and for how long?
• How does the design change if the created object is not listable - a one-off payout, say - so an orphan cannot be found afterwards?

Webhook workers leak until OOM and drop in-flight deliveries

Medium · memory leak · eviction · leases

webhook-delivery workers grow from 400 MB to a 2 GB limit over about 36 hours, are OOM-killed, restart, and repeat. Each restart abandons in-flight attempts, so webhook_delivery rows sit in in_flight until their leases expire and the backlog spikes. The live set measured after a forced full collection also grows. The fleet serves tens of thousands of subscriptions, several thousand of which have been failing for weeks. Give an ordered checklist, the measurement separating retention from fragmentation, and the fix.

Approach

1. Separate the two failure shapes with one measurement: track resident set size against the live set after a forced full collection. A live set that climbs monotonically is retention; a flat live set under a rising RSS is fragmentation, off-heap or native allocation, or an allocator that never returns pages. The stated symptom puts this in the first category, which rules out allocator tuning as a fix.
2. Characterise the curve rather than the total. Growth linear in uptime implies an unbounded structure keyed by something that keeps arriving; step growth implies buffering a large object. Correlate the slope against event rate and separately against the count of distinct subscriptions seen, because those two diverge and only one of them will fit.
3. Diff two heap snapshots an hour apart by retained size grouped by dominant root, not by allocation count, which is dominated by short-lived objects and will point at the wrong thing.
4. Expect a per-subscription map with no eviction: circuit-breaker or backoff state created on first failure and never removed, so the retained set grows with endpoints that have ever failed, and the several thousand permanently dead endpoints hold theirs forever.
5. Fix in two places. Bound the in-memory structure with a size-capped LRU or a TTL keyed on last use, and move state that must survive a restart onto the subscription or webhook_delivery row, since the worker holding it in memory is exactly why a restart loses it.
6. Repair the second-order damage separately, because it will outlive the leak: workers claim by compare-and-set with leased_until, so a bounded lease returns in_flight rows to pending on a known schedule, and a graceful shutdown releases leases instead of waiting them out.

Follow-up

• The backlog spike after a restart is itself a thundering herd against customer endpoints. What stops the recovery from becoming a second incident?
• Suppose the live set had been flat while RSS still climbed. Name two causes and the measurement that separates them.
• How would you size the LRU, and what does a miss on an evicted circuit-breaker entry cost a customer whose endpoint is down?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

• Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
• For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
• Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

• Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
• After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
• Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

• Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
• Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
• Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

• Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
• Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
• Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

• Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
• After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
• Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

• Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
• Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
• Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

• Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
• Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
• Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

'We rewrote it in a better framework' is not a story

A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.

Tell me about a time you had to deliver a project with ambiguous or changing client requirements.

Medium · behavioural and engineering judgement

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Give the blast radius: what could have broken, and what you measured.
3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

• What did you decide not to do, and why?
• How did you know your change caused the improvement?

Disclose a cross-tenant webhook delivery to affected customers

Medium · cross-tenant leak · disclosure · blast radius · authorisation checks

An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.

Approach

1. Bound the population before saying anything externally. The affected set is deliveries in the window where the event's tenant and the subscription's tenant differ; the ones that actually left are those with delivered_at set and a 2xx in last_response_code. Attempted and delivered are two different counts and a disclosure has to use the right one in the right sentence.
2. Separate what the records prove from what they do not, and say both halves rather than the flattering one. They prove which payloads were signed, where they went, and — through payload_digest — exactly which bytes. They do not prove what the receiving system did with them, and they do not bound the window more precisely than your deploy timestamps do.
3. Communicate on the facts you hold, with the scope stated as an upper bound: 'at most eleven payloads, four recipient endpoints, these fields, this window' is more useful and more honest than waiting a day for certainty. The field list matters more than the event count, because a customer cannot assess exposure from 'an event'.
4. Name the code change precisely, because this class never originates in the delivery worker. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before the payload is signed, and make the second comparison drop the delivery rather than log a warning. Say why one check is insufficient: the enqueue check protects against the bug you know about, the pre-signing check protects the boundary itself.
5. Run the history question in parallel and say so: a query over historical deliveries for the same mismatch tells you whether this was nineteen minutes or a year, and you would rather find the second case yourself than have a customer find it after your disclosure.
6. Split the response into workstreams with owners — recipients asked to delete, affected customers notified, the check landed with a test, history swept — and say which you personally drove and which you handed off. Claiming all four is not credible and claiming none is not ownership.

Follow-up

• The historical sweep finds two more instances from last year. What changes in what you have already told people?
• Who approves the wording, and what do you do when you are asked to soften the scope?
• A customer asks you to prove a redelivery contained the same bytes as the original. What do you show them?

Estimate a tenant-leading index migration you have never run

Hard · estimation · online migration · index builds · uncertainty

Someone needs a date. usage_event carries an index on (occurred_at) and needs (tenant_id, occurred_at); the largest tenant holds roughly a hundred times the median tenant's rows, the table is partitioned daily with years of retention, and you have never run a migration on a table this large. Give an estimate you would defend: how you decompose the work, the two or three numbers you would go and measure first, the range and confidence you state, and what you commit to when the person asking needs a single date today.

Approach

1. Refuse the bare number and then give one anyway, in the form that is actually useful: a range plus the measurement that collapses it. 'Four to eleven days; one afternoon building this index on a restored copy of the largest partition takes that to within a day' is an answer, while 'it depends' is not.
2. Decompose by failure mode rather than into equal chunks, because that is where estimates go wrong. On a partitioned parent you create the index ON ONLY the parent, build each partition's index with CREATE INDEX CONCURRENTLY, then ALTER INDEX ... ATTACH PARTITION, at which point the parent index becomes valid. CONCURRENTLY does not block writes but scans each partition twice, waits out older transactions, cannot run inside a transaction block, and on failure leaves an invalid index you must drop concurrently and retry.
3. Name the two unknowns that dominate and price them: build time on one restored partition of realistic size, and whether the planner actually chooses the new index for the skewed tenant, since selectivity for a tenant holding most of the rows is a different question from selectivity for the median tenant. Both are half-day measurements against a replica, and both are cheaper than being wrong by a week.
4. State the assumptions the range is conditional on, because that is what makes a slip a re-estimate instead of a credibility event: no partition above a stated row count, one concurrent build at a time so it does not compete with ingest for I/O, and an ingest backlog that can absorb the added write amplification while both indexes exist.
5. Budget the step nobody budgets: verification and the old index's removal. Dropping the old index is fast, but deciding it is safe to drop means confirming no plan still uses it, and that confirmation waits on real traffic across a full weekly cycle rather than on your patience.
6. Answer the single-date request honestly. Commit to a date for the first checkpoint — the measured build number from the replica — and to re-estimating on that date, and say plainly what you are not committing to yet. A date with a scheduled re-estimate is worth more to the asker than a confident wrong one, and you should say why in those words.

Follow-up

• The concurrent build fails half way through the largest partition. What is the state of the database and what do you do next?
• Your estimate slips by sixty percent. Which assumption broke, and at what point would you have known?
• The person asking needs the date for a customer commitment. Does your answer change?