Interview guides · Cloudera

Cloudera Software Engineer Interview Questions & Guide 2026

Caching practice for Cloudera software engineer prep: settle the staleness budget before choosing an invalidation strategy.

The 60-second brief

As a Software Engineer at Cloudera, you play a crucial role in designing and developing cutting-edge data management and analytics solutions. Your work directly influences how organizations manage and derive insights from their data, making it essential to the success of numerous industries. You will collaborate with cross-functional teams to build scalable and efficient software that meets the diverse needs of Cloudera's customers.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

Evolve APIs without breaking pinned SDK clients
Build at-least-once pipelines with explicit deduplication horizons
Keep money in integer minor units

Overview

The Software Engineer position is pivotal in driving innovation within Cloudera's suite of products, including its cloud-based solutions and data platforms. You will tackle complex challenges related to distributed systems, data processing, and real-time analytics. This role not only offers the opportunity to work on exciting and impactful projects but also allows you to grow your skills in a dynamic and supportive environment.

In this position, you will engage with state-of-the-art technologies and methodologies, contributing to the development of solutions that empower businesses to leverage their data effectively. The complexity and scale of the projects at Cloudera provide a stimulating work atmosphere where you can make significant contributions while advancing your career.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Holding money in a floating-point type, or rounding it more than once

Binary floating point cannot represent 0.01 or 0.1 exactly, so sums drift and two code paths that should agree disagree by cents nobody can trace back. The fix is integer minor units or an exact decimal type end to end, with sub-cent rates expressed as scaled integers such as micro-units, because a per-request price genuinely is smaller than a cent. The second half of the trap is rounding position: rounding each line and then summing gives a different total from summing and rounding once, and half-up and half-even diverge systematically across many lines, so rounding must happen at one named place and every downstream reader must carry the rounded value rather than recompute it from quantity and rate.

Paginating a growing table with limit and offset

Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.

Finishing a solution without stating its complexity

Give time and space in the same breath as the code, and define n explicitly when there are two sizes, since n nodes and m edges are not interchangeable. Space is the half that gets skipped: count the auxiliary structures you allocate and the recursion stack at its deepest, not only the answer you hand back.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Questions & practice

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Follow-up

One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Worked solution (35 min)

Reproduce the shape locally: generate 2 million events for one synthetic tenant over 5 days, fold them, then inject three defects, namely 0.2% of events bucketed by ingested_at, a handful of duplicate idempotency_key values whose retries cross midnight, and a block of events ingested after the seal.
Write the watermark-bounded query first and record how much of the gap it explains on its own.
Bisect by day, then SKU, then hour, recording the probe count and the rows each probe touches.
For each located cell, write down the predicted signature before querying it, then check whether the data matches the prediction.
Time a full re-fold of the 2 million rows and extrapolate to 90 million.

Expected result: The watermark-bounded query accounts for the post-seal block entirely and removes it from the investigation. The ingested_at bucketing appears as adjacent hours off by equal and opposite amounts. The midnight-crossing duplicates appear as one (tenant_id, idempotency_key) present in two ingested_day partitions. Bisection reaches each cell in fewer than 70 probes.

Verify it yourself

The three injected defects sum to the original gap with no unexplained remainder.
Each probe's row count is bounded by one day-partition for one tenant, not by the period.
Removing the source_max_ingested_at bound from the first query makes a gap appear, and grow, while ingest runs, even after every defect is removed.

Follow-up

The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
Ingest is still running while you investigate. What makes your two numbers comparable at all?

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Worked solution (20 min)

Load about 50k deliveries for one tenant, then walk them with the OFFSET query while a writer inserts 10 rows/second, collecting every returned delivery_id.
Compare the distinct ids collected against the set of ids that existed when the walk started, and record the shortfall.
Repeat the walk with the keyset query and confirm every pre-existing id is returned exactly once.
Run explain (analyze, buffers) on page 1 and page 500 of each form and compare shared buffer hits.

Expected result: A keyset query using the row comparison (created_at, delivery_id) < ($2, $3) with order by created_at desc, delivery_id desc limit 100, backed by (tenant_id, created_at, delivery_id) scanned backwards; plus the statement that OFFSET drops rows under concurrent insert and costs time proportional to the offset, and that updated_at is disqualified because it mutates on retry.

Verify it yourself

Buffer counts for page 500 are the same order of magnitude as page 1.
Every id present before the walk is returned exactly once when only inserts run concurrently.
The query plan shows no Sort node above the index scan.
Two rows sharing a created_at value to the microsecond are both returned across the page boundary.

Follow-up

The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Explain how you would approach scaling a web application.

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
Fix the scope first: who calls this, how often, and what they do when it fails.
State the consistency you need, and where you are willing to be stale.

Follow-up

How does this behave when that dependency is down for an hour?
What breaks first when traffic grows ten times?

How would you design a URL shortening service?

Medium · system design and architecture

Approach

Name the read and write paths separately; they rarely have the same bottleneck.
Fix the scope first: who calls this, how often, and what they do when it fails.
Name the failure you are designing for, then the recovery path.

Follow-up

How does this behave when that dependency is down for an hour?
What breaks first when traffic grows ten times?

Discuss the considerations for designing a distributed cache.

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
Name the failure you are designing for, then the recovery path.
State the consistency you need, and where you are willing to be stale.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

Explain the difference between a stack and a queue.

Medium

Approach

State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How do you ensure code quality and reliability in your projects?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

What is the time complexity of binary search?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Write the delivery guarantee and replay API for outbound events

Medium · webhooks · at-least-once · ordering

The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.

Approach

Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.

Worked solution (30 min)

Write the guarantee sentence and the one-sentence reason exactly-once is not available over HTTP.
List the headers a customer needs to build their own dedup table: event id, delivery id, attempt number, subscription id, signature and timestamp.
Write the response-code table with four classes and the action for each, including the 410 case.
Write the retry schedule with concrete numbers, the terminal condition, and the breaker's threshold and its effect on the delivery row's status.
Specify the redelivery request and response, and say which columns change and which do not.

Expected result: At-least-once with a stable event id, ordering offered only as an opt-in single-in-flight mode with head-of-line blocking stated, a four-class response table, a jittered capped schedule with a terminal state, and redelivery implemented as new rows at an incremented sequence carrying identical bytes.

Verify it yourself

The customer can deduplicate using headers alone, without parsing the payload.
Nothing in the contract requires the sender to know whether the customer committed.
A day-long outage has a defined end state rather than unbounded retries.
Redelivery leaves the original delivery rows readable as history.

Follow-up

A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?

Regional error rate explodes after a dependency merely slows

Hard · retry storm · circuit breaker · partial outage

A control-plane read replica in one region degrades from 4 ms to 120 ms. Within ninety seconds that region's gateway error rate rises from 0.01% to 40% and its p99 becomes bimodal, one mode near the old p99 and one at the client timeout. The other two regions are unaffected. The gateway retries control-plane reads three times with exponential backoff and no jitter. Give an ordered checklist that separates trigger from amplifier, the offered-load arithmetic, and the controls that break the loop.

Approach

Split the incident into three questions before touching a control: what started it, what amplified it, and what would make recovery slow. Here they are the replica slowdown, the retry policy interacting with queueing, and a synchronised unjittered herd at recovery. They are different mechanisms and each needs its own control.
Read the distribution rather than the mean. A bimodal p99 with one mode pinned at the client timeout is two populations, not one degraded path; split latency by cache hit and miss and confirm the fast mode is hits and the timeout mode is misses that reached the replica.
Do the load arithmetic. Three retries turn one client request into up to four upstream requests, so offered load reaches roughly 4x on a dependency that is already slower, and it arrives at the worst moment. With utilisation approaching one, queueing delay grows superlinearly, which is why a 30x latency increase upstream does not produce a 30x increase downstream, it produces timeouts.
Break the loop with controls that bound offered load rather than with more attempts: a concurrency limit on the control-plane client so at most N calls are in flight and the remainder fail fast, a circuit breaker scoped per dependency and region, and a retry budget capping retries at a small fraction of base traffic so amplification has a ceiling that does not depend on how many clients are retrying.
Add full jitter to whatever retries survive, sleeping uniformly in [0, min(cap, base x 2^attempt)], so attempts de-correlate instead of arriving in waves aligned to the moment of failure.
Decide the unreachable-dependency behaviour in advance, because it is the actual product decision underneath: serving from an expired credential cache keeps the product available while extending a revoked key's life past the stated bound, and failing closed converts a dependency degradation into a total outage. State the mode and the staleness number rather than letting the timeout choose.

Follow-up

The replica recovers. Describe what happens in the first ten seconds with your controls in place versus without them.
Which single metric would have paged before the error rate moved, and why is upstream latency by itself not it?
Requests that fail fast under the concurrency limit still need an answer. What does the gateway return, and what does it do to the usage event it would otherwise have emitted?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

Own the incident where invoices undercounted metered usage

Medium · incident response · at-least-once · billing correction · postmortem

A metering consumer acknowledged each batch before committing the fold into usage_rollup_hourly. A rolling deploy restarted consumers mid-batch for two hours; roughly 1.4M usage_event rows were acknowledged and never folded, and 61 invoices sealed against the resulting rollups before anyone noticed. Take the owner's role. Describe an incident of comparable blast radius you owned: how it surfaced, the query that sized the loss, what you stopped first, and how the money was corrected. Give a wall-clock timeline and one thing you got wrong while it was still live.

Approach

Open with the invariant that broke and the direction of the error, because they determine everything else: acknowledging before committing makes the consumer at-most-once, so this loses events rather than duplicating them, and loss raises no error anywhere. A listener who hears 'we lost revenue silently' knows immediately why detection took two hours.
Size it with a stated reconciliation rather than an adjective: sum(quantity) from usage_event grouped by (tenant_id, sku, hour of occurred_at) over the window, against usage_rollup_hourly.quantity_sum on the same keys, filtered to environment='production' because staging and sandbox are metered but not billed. Then bisect by hour and tenant until single cells explain the gap. Say how long that ran and whether a replica could serve it while the incident was live.
Separate mitigation from fix and say which came first. Mitigation is holding the sealing job, because a sealed row is frozen by design and every minute of sealing converts a recoverable rollup into an invoice correction. The fix is moving the acknowledgement after the commit, which re-introduces duplicates that the dedup check on (tenant_id, idempotency_key) must now absorb.
State the correction path in the domain's own terms: sealed periods are never edited, so each affected tenant gets an adjustment line on the next invoice with kind='adjustment' and voided_by_line_id pointing at the line it reverses, priced against the same rate tier and carrying the watermark it priced against. That is four separate numbers — tenants affected, minor units, the cycle the adjustment lands in, and when customers were told.
Close on one prevention control with its cost, not five: a per-hour reconciliation comparing raw sum to rollup sum that pages above a threshold. Name the threshold and the false-page rate you accepted, because a detector nobody will keep staffed is not prevention.
Name a mistake you made inside the response window — the wrong first hypothesis, a mitigation that made it worse — rather than a design mistake from six months earlier. That is the part candidates rehearse away and interviewers weight heavily.

Follow-up

Your fix moves the acknowledgement after the commit. What breaks now, and what absorbs it?
One undercharged tenant has since churned. Do you bill them, and who decides?
How would you have caught this in ten minutes instead of two hours, and what would that detector cost you in pages per week?

Ship metered billing with a named deduplication horizon

Medium · technical debt · deduplication · deadline pressure · detectors

Metered billing must be on in three weeks. usage_event is partitioned daily, so its unique index must include the partition key and deduplicates only within a day: a producer retry that crosses midnight, or a replay run a week later, gets through. A cross-partition dedup store is two weeks you do not have. Describe shipping with debt you named in advance: what you shipped, what you wrote down, the detector you added, the trigger and date for paying it off, and what you would have refused to ship under the same pressure.

Approach

Show you can separate the two kinds of debt, because that distinction is what the question actually probes. Debt that costs engineering time later is shippable on a deadline. Debt that silently corrupts a number a customer gets charged for is not shippable unless the corruption is detectable, and detectability is the whole negotiation.
Make the exposure narrow and measured rather than gestural. The hole is duplicates whose occurrences straddle a UTC day boundary, plus any replay older than partition retention. Measure it before arguing about it: how often an idempotency_key recurs at all, and the distribution of the gap between first and last occurrence. If the ninety-ninth percentile of that gap is four minutes, the residual risk is a small band around midnight and you can say so numerically.
Add the detector before the feature, not after. A nightly job counting keys that appear in more than one partition is one grouped scan over recent partitions, and it converts a silent overcount into a page. State what it costs to run and what it fires on.
Buy the cheap half of the real fix immediately: extend partition retention so the dedup horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. That reframes retention as a correctness parameter rather than a storage cost, which is the sentence you need on record before someone optimises the bill.
Make repayment mechanical instead of aspirational: a dated entry with a named owner, plus a threshold that pulls the date forward — first detector hit above N events, or first customer dispute. Debt with a trigger gets paid; debt with only a date does not.
Answer the second half honestly by naming what you would refuse under identical pressure: the sealing path, because a sealed row is frozen and a wrong number there stops being a bug and becomes an adjustment line, a dispute and an audit question.

Follow-up

The detector fires on forty duplicate events for one tenant, and two of their invoices have already sealed. What happens next?
Whom did you tell that the billing numbers had a known hole, and in what words?
Finance asks you to cut storage by shortening partition retention. What do you say, and to whom?

Unblock an engineer on a job run that finished twice

Easy · mentoring · fencing tokens · lease expiry · debugging method

An engineer two weeks into the team brings you a job_run row showing status succeeded with an exit_code written by a worker declared dead ten minutes earlier; the retry attempt also shows succeeded. They have spent a day adding logging and are no closer. You have twenty minutes and you do not want to take the keyboard. Describe how you unblock someone: the question you ask first, what you let them find themselves, the concept you name and when, and how you check the next day that they own the fix rather than having watched you produce it.

Approach

Ask what they expect rather than what they see: which statement set status to succeeded, and what did it check before writing? That question points directly at the update's WHERE clause, which is where the answer lives, and it costs them nothing to answer, so it does not read as a test.
Let them build the timeline themselves from the row: queued_at, started_at, leased_until, finished_at and worker_id, on both the original run and the retry. Two different worker_ids with a lease expiry between them tells the whole story, and they will see it before you say it.
Name the concept once the evidence has earned it. A lease bounds time; it does not prevent a write. The store has to reject a stale writer, which means the update carries a fencing token the row compares — update job_run set status = 'succeeded' where run_id = $1 and lease_token = $2 and status = 'running' — and a long garbage-collection pause or a brief partition is enough to produce what they are looking at.
Point at the second, less obvious half and let them decide it: 'lost' exists in the status enum precisely so a run whose worker vanished is not recorded as failed, because failed asserts an outcome nobody observed and the system then bills and retries on that assertion. Ask them what these two rows should have said.
Leave them with the next step rather than the patch — a test that kills the first worker after the sandbox exits and before the row is written — and say when you are available again, so the offer is real rather than polite.
Check ownership the next day by what they produced, not by asking if it went well: a test that reproduces the window proves they understood it; a test that only asserts the new WHERE clause proves they copied it. Ask them to explain it to a third person and listen for whether the explanation is theirs.

Follow-up

They propose a longer lease instead of a token. What do you say, and what breaks when legitimate runs last thirty minutes?
How can you tell whether your explanation landed or they simply deferred to you?
The same engineer hits a variant of this next month. What did you fail to teach the first time?