cswork / libraryOpen practice workspace

Interview guides · Bristol Myers Squibb

Bristol Myers Squibb Software Engineer Interview Questions & Guide 2026

Bristol Myers Squibb software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

As a Software Engineer at Bristol Myers Squibb, you play a vital role in building and scaling the digital and technical infrastructure that powers life-saving pharmaceutical research, clinical operations, and global business processes. This position is responsible for designing robust backend systems, integrating complex third-party tools, and delivering high-performance applications that handle critical healthcare data securely and efficiently. Your daily work directly impacts researchers, clinicians, and patients by ensuring that software solutions operate with high availability, strict compliance, and absolute reliability.

If the seat owns a service boundary, scope your preparation toward failure behaviour rather than topology. Retrying over an at-least-once channel produces duplicates by construction, so a retry policy is only as safe as the idempotency key underneath it.

Focus areas

  • Build at-least-once pipelines with explicit deduplication horizons
  • Bound blast radius with per-tenant concurrency limits
  • Evolve APIs without breaking pinned SDK clients

Overview

The scope of this role spans across multiple domains, including cloud architecture, full-stack development, API integrations, and specialized data pipelines. Whether you are modernizing internal data workflows, implementing secure Public Key Infrastructure (PKI) frameworks, or collaborating across cross-functional teams on global deployments, your technical expertise drives innovation in a highly regulated industry. Working at Bristol Myers Squibb means balancing engineering excellence with a deep commitment to compliance, quality, and patient-focused outcomes.

Candidates stepping into this role should expect an environment that values both technical precision and collaborative problem-solving. While the technical stacks are modern—incorporating cloud platforms, Python, advanced data frameworks, and containerization—the culture emphasizes accountability, continuous improvement, and alignment with core organizational values. You will find yourself collaborating closely with product managers, validation engineers, and global stakeholders to turn complex business needs into elegant, scalable software solutions.

The Bristol Myers Squibb interview process

Bristol Myers Squibb candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interview

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

  • Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
  • Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
  • Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

  • Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
  • Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
  • Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Panel Interview

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

One shared connection pool for every tenant and every query class

A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Questions & practice

Seal an hour under late data with bounded memory

Hard · watermarks · late-data · quantile-sketch · conditional-write

Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.

Approach

  1. Two clocks, two jobs. Bucket by occurred_at, because that is the hour the customer is billed for, and advance the watermark on ingested_at, because that is what the fold has consumed and what source_max_ingested_at records. Conflating them is what makes late data invisible.
  2. The global watermark is the min over partitions of each partition's committed ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past.
  3. Choose the lateness L from the measured distribution of ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters.
  4. Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
  5. Hold open hours in a min-heap keyed by hour_start. When the watermark advances, pop every hour with hour_end + L < W and seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct (tenant, workspace, sku) keys, so cap the number of simultaneously open hours and spill the oldest into usage_rollup_hourly as status='open' with a revision bump. While an hour is open the row is upsertable, so the store is your overflow.
  6. The seal itself is a conditional write: update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, and source_max_ingested_at is what proves it arrived afterwards.

Follow-up

  • A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
  • Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
  • How would you detect that L has drifted before it costs you an hour's quantity?

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

  1. Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
  2. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
  3. complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
  4. Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
  5. Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
  6. All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Follow-up

  • One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
  • A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
  • The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Hold a tenant to a trailing sixty-second request limit

Medium · sliding-window · two-pointer · rate-limiting · tenant-skew

The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.

Approach

  1. Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request.
  2. Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
  3. Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate, prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact.
  4. Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (tokens, last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual.
  5. Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
  6. Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.

Worked solution (25 min)

  1. Implement the deque version and instrument the per-request pop count, then confirm total pops equal total pushes over a run.
  2. Generate a burst that places R requests in the last 100 ms of one minute and R more in the first 100 ms of the next.
  3. Run that burst through the exact deque, a fixed 60-second counter, and the weighted two-bucket estimate, recording admissions in the trailing 60 seconds at every instant.
  4. Size the memory as R x active tenants x 8 bytes at R = 1,000 and 200,000 tenants, and compare it against what a token bucket would need.

Expected result: The exact deque never admits more than R in any trailing 60-second window. The fixed counter admits close to 2R across the boundary. The weighted estimate lands between the two on this burst and approaches 2R once the previous window's requests are packed at its end.

Verify it yourself

  • Total pops equal total pushes, which is what makes the amortised O(1) claim true rather than merely asserted.
  • Sliding the whole burst by 30 seconds changes the fixed counter's result and leaves the deque's unchanged.
  • Running two processes each limiting to R/2, against a tenant that sends everything to one of them, shows that tenant throttled at half its contractual limit.

Follow-up

  • One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
  • Quotas rather than rate limits: the check is select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes.
  • How do you return an accurate Retry-After from the exact algorithm without a second scan?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

  1. Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
  2. Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
  3. Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
  4. Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
  5. Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
  6. Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Follow-up

  • Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
  • A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
  • Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

  1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
  2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
  3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
  4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
  5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
  6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

  1. On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
  2. Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
  3. Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
  4. Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
  5. Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
  6. Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

  • No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
  • The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
  • After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
  • Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

  • A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
  • The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
  • Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

How would you approach roadmap design and third-party vendor management?

Medium · system design and architecture

Approach

  1. Choose a partition key and say what query it makes expensive.
  2. Name the failure you are designing for, then the recovery path.
  3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

Define for me the key concepts of MRO management in relation to planning, scheduling, and overall impact to the maintenance department.

Medium · system design and architecture

Approach

  1. Name the failure you are designing for, then the recovery path.
  2. State the consistency you need, and where you are willing to be stale.
  3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

How do you manage a system around the globe of the organization and how do you manage the work around it?

Medium · system design and architecture

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. Fix the scope first: who calls this, how often, and what they do when it fails.
  3. Name the failure you are designing for, then the recovery path.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What breaks first when traffic grows ten times?

These questions assess your ability to manage distributed systems, handle global deployments, and scale applications.

Medium · system design and architecture

Approach

  1. Fix the scope first: who calls this, how often, and what they do when it fails.
  2. Name the failure you are designing for, then the recovery path.
  3. State the consistency you need, and where you are willing to be stale.

Follow-up

  • What breaks first when traffic grows ten times?
  • What would you drop to keep the system up under load?

Are you interviewing with any other companies?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. State your assumptions explicitly before working the problem.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

These questions test your core engineering stack knowledge, ranging from language specifics to systems architecture.

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Specify webhook signature verification a customer can implement

Easy · webhooks · hmac · replay protection

The webhook-delivery service signs each payload before POSTing it to a customer endpoint. Write the signature specification a customer implements in their own language: the header format, exactly which bytes are signed, the algorithm, how replay is bounded, and how a signing secret rotates without a delivery gap. Then write the verification steps the customer performs, in order, including what they compare and what they return on failure. Constraint: most customers reach for their web framework's parsed JSON body by default. Deliverable: the spec section plus reference pseudocode.

Approach

  1. Sign the concatenation of the timestamp and the raw body, t + "." + body, and emit a header of the form t=<unix seconds>,v1=<hex>. The timestamp has to be inside the MAC, or an attacker re-stamps a captured body and the tolerance window buys nothing.
  2. Require the raw request bytes. A framework that parses JSON and re-serialises it changes key order, whitespace and number formatting, so the spec must tell the customer to capture the body before the parser runs and give the middleware note for each common framework.
  3. Use HMAC-SHA256, not sha256(secret || body): SHA-256 is a Merkle-Damgard construction, so the naive form admits length extension. Require a constant-time comparison as well, since a short-circuiting byte compare leaks the expected prefix under repeated probing.
  4. Bound replay in two layers: reject when |now - t| exceeds a stated tolerance such as 300 seconds, then deduplicate on the event identifier header. The tolerance is what makes the customer's dedup store finite rather than unbounded.
  5. Rotate by allowing two live secrets and emitting both signatures in one header (v1=<old>,v1=<new>); the customer accepts if any candidate matches, so neither side needs an instantaneous cutover. A failed verification returns 400 and the body is not processed.

Worked solution (15 min)

  1. Write the header grammar and one real example line with a plausible timestamp and hex digest.
  2. Write the signed string construction explicitly as a byte concatenation, and add the sentence telling the customer where in their framework to obtain the raw body.
  3. Write the five verification steps in order: extract t and candidates, check the tolerance, recompute the HMAC over t + '.' + raw body, compare in constant time against each candidate, then deduplicate on the event identifier.
  4. Add the rotation paragraph: two active secrets, both signatures sent, overlap window stated in the dashboard.
  5. State the failure response and the fact that the payload is not processed, plus what the sender does with that 400.

Expected result: A spec where the signed input is the timestamp joined to the exact raw bytes, the algorithm is HMAC-SHA256 compared in constant time, replay is bounded by a stated tolerance plus event-id deduplication, and rotation is an overlap of two accepted signatures rather than a cutover.

Verify it yourself

  • The timestamp is inside the signed string, not merely sent alongside it.
  • The spec names raw bytes and says where the customer gets them before parsing.
  • Rotation requires no coordinated instant on either side.
  • The reason for HMAC over sha256(secret || body) is stated as length extension, not as a preference.

Follow-up

  • A customer's verification passes locally and fails in production behind a proxy that re-encodes the response body. Where do you look first?
  • Why sign with a per-endpoint secret rather than the tenant's API key?

Hourly rollups merge one hour and lose another

Easy · timezone · rollups · reconciliation

Reconciliation flags one tenant on one day. Summing usage_event.quantity by hour of occurred_at gives 24 non-empty hours, but usage_rollup_hourly holds 23 rows for that tenant, workspace and SKU, one of which carries roughly the sum of two adjacent hours. Other days reconcile exactly, and the affected date matches a civil-time transition. hour_start is documented as truncated to the hour in UTC. You have both tables, the rollup job source, and its runtime environment. Give an ordered checklist, the mechanism, and the correction path for a day that may already be sealed.

Approach

  1. Bisect by dimension until one cell explains the whole difference: tenant, then day, then SKU, then hour. A defect confined to a single transition date already rules out deduplication and late arrival, both of which are indifferent to which hour an event lands in.
  2. Read the truncation with its precondition stated: date_trunc on a timestamptz value is evaluated in the session TimeZone, not in UTC. If the job connects without pinning that setting, it inherits the server or container default.
  3. Follow that to the collision: in a zone that observes daylight saving, two distinct UTC hours map to the same local wall-clock label at the autumn transition, so both fold into one key under the unique constraint on (tenant_id, workspace_id, sku, hour_start) and their quantities sum into one row. At the spring transition a label never occurs and the row is simply absent.
  4. Confirm from data rather than from reading code: run the same aggregate twice, once with the session pinned to UTC and once with the job host zone, and check that the second reproduces the stored rollup exactly.
  5. Fix at the source by pinning the connection to UTC explicitly, or by truncating on occurred_at AT TIME ZONE 'UTC', rather than relying on a default that differs between a developer machine, CI and production.
  6. Correct according to status, not convenience: an open hour is recomputed with revision incremented, a sealed hour is frozen and the difference becomes an adjustment line on the next invoice with voided_by_line_id pointing at the line it reverses.

Follow-up

  • The same job also emits a daily figure for a dashboard. Why can a correct hourly rollup still produce a wrong day, and what does the tenant's billing timezone have to do with it?
  • How would you detect this class automatically rather than waiting for reconciliation, given that it only manifests twice a year per zone?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

  • Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
  • Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
  • Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

  • Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
  • Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
  • Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

  • Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
  • Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
  • Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

  • For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
  • Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
  • Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
  • Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

  • Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
  • Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
  • Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
  • Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

  • Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
  • Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
  • Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

  • Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
  • Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
  • Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

The failure that was yours

Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.

Tell me about your experience with backend systems and API integrations.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Close with what you would do differently, concretely.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Describe a time when you had something go wrong at work and how did you resolve it.

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. Give the blast radius: what could have broken, and what you measured.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Resolve a review disagreement over a quota check

Easy · code review · isolation levels · write skew · disagreement

A colleague's pull request enforces a per-tenant quota by selecting the current count and then inserting when it is under the limit. You flag it as a race. They reply that the transaction already runs at repeatable read, so the snapshot makes it safe, and the tests pass. Walk through taking that disagreement to a resolution: what you write in the review, what you demonstrate rather than assert, which fix you propose and why, and what you do if they still disagree after all of it.

Approach

  1. Answer the claim precisely instead of restating your objection, because they have made a specific technical argument. In PostgreSQL, repeatable read is snapshot isolation; this is write skew, which snapshot isolation permits by design. Both transactions read a count that is stable within their own snapshot, insert disjoint rows that the other cannot see, and both commit, so the limit is exceeded by exactly the concurrency.
  2. Demonstrate rather than cite. Two psql sessions, both BEGIN ISOLATION LEVEL REPEATABLE READ, both select the count, both insert, both commit: it succeeds. Repeat at SERIALIZABLE and the second commit fails with serialization_failure, SQLSTATE 40001. That takes two minutes, ends the argument without anyone conceding a position, and leaves an artefact for the next reviewer.
  3. Offer the options with their costs rather than a verdict. Serialisable plus a retry loop on 40001 is correct but obliges every caller to retry and degrades under contention. An increment-and-compare on a counter row — update tenant_quota set used = used + 1 where tenant_id = $1 and used < limit returning used — is safe even at read committed, because a blocked updater re-evaluates the WHERE clause against the row version it finally locks, and zero rows returned means full. A unique or exclusion constraint that makes the surplus write fail is the third.
  4. Name the plausible non-fix explicitly, since it is what usually gets merged instead: folding the count into the insert as insert ... select ... where (select count(*) ...) < limit is still racy under read committed, because the subquery cannot see the other transaction's uncommitted rows. It looks atomic and is not.
  5. Say what you do if they still disagree: escalate the decision rather than the disagreement. Attach the reproduction, hand it to the service owner or a third reviewer, and state that you will not block the merge if the owner accepts the risk knowingly — and that you want that acceptance written down.
  6. Close with the general lesson worth leaving in the review thread: a passing suite is weak evidence for a concurrency claim because it runs one request at a time. Ask for a test that runs two.

Follow-up

  • Write the counter-row version. Does your answer change if the quota counts child rows rather than a column?
  • Under serialisable, who performs the retry, and what does the API client see if the retry also fails?
  • This is the third disagreement with the same reviewer this month. What changes in how you review?