Interview guides · Advaiya
Advaiya Software Engineer Interview Questions & Guide 2026
Advaiya software engineer prep on databases: indexes, query plans, and the anomalies an isolation level still permits.
The 60-second brief
A Software Engineer at Advaiya serves as a vital bridge between complex business challenges and scalable technological solutions. As an organization focused on digital transformation, data infrastructure, and cloud services, Advaiya relies on engineers who can translate ambiguous client requirements into robust, functional code. You will be expected to contribute to projects that span full-stack architecture, cloud service optimization, and data-driven systems.
The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.
Focus areas
- Build at-least-once pipelines with explicit deduplication horizons
- Keep money in integer minor units
- Bound blast radius with per-tenant concurrency limits
Overview
This role is critical because it directly impacts how clients manage their digital ecosystems. Whether you are working as a Sr. Consultant or a Cloud Service Engineer, your work will influence the efficiency and reliability of enterprise-level applications. You will operate in a dynamic environment where technical proficiency is balanced with the ability to navigate diverse project scopes, making this an ideal position for engineers who enjoy variety and high-impact problem-solving.
The Advaiya interview process
Advaiya candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.
Initial Screening
Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.
What to demonstrate
- Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
- Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
- Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not
How to prepare
- Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
- Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
- If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window
Technical Evaluation
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Where candidates lose points
Paginating a growing table with limit and offset
Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.
Checking a quota with a select and then writing
Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.
Comparing floating-point values for equality, or holding money in them
Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.
Treating a network call as though it were a local function call
A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.
Questions & practice
Hold a tenant to a trailing sixty-second request limit
Medium · sliding-window · two-pointer · rate-limiting · tenant-skew
The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.
Approach
- Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while
front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request. - Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
- Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate,
prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact. - Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (
tokens,last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual. - Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
- Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.
Follow-up
- One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
- Quotas rather than rate limits: the check is
select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes. - How do you return an accurate
Retry-Afterfrom the exact algorithm without a second scan?
Order a job dependency graph and find its critical path
Medium · topological-sort · dag-longest-path · cycle-detection · critical-path
A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.
Approach
- Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
- Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
- Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order:
earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E). - Second pass in reverse topological order for
latest_finish, thenslack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain.slack[v] = 0is exactly the statement that some longest path runs through v; equivalently, the longest path through v has lengthT - slack[v]. - The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by
min(d, T - L_avoid(v)), whereL_avoid(v)is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, whilemin(10, 15 - 14)predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan ismax(T - d, L_avoid(v)). - Compute
L_avoid(v)the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, sinceL_avoid(v)only ever matters through that max: setduration[v] := 0, recompute the makespan asT0(v) = max(T - duration[v], L_avoid(v)), and the gain ismin(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.
Follow-up
- Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
- Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
- Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?
Schedule ordered webhook retries with a heap of subscription queues
Medium · heap · scheduling · backoff · head-of-line-blocking
Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.
Approach
- Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's
next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, incrementattempt_countand setnext_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.- Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
- Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
- All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on
(subscription_id, next_attempt_at) where status in ('pending','failed_retryable')claimed withFOR UPDATE SKIP LOCKED, and the write-back must be fenced onlease_tokenso a worker that stalled and resumed cannot overwrite a newer attempt.
Worked solution (30 min)
- Define the four structures explicitly:
queues: subscription_id -> deque[delivery],ready: min-heap of (next_attempt_at, subscription_id),inflight: set[subscription_id],breaker: min-heap of (half_open_at, subscription_id). - Write down the invariant you will assert after every operation: a subscription appears in at most one of
ready,inflightandbreaker, never in two. - Implement
due,completeandinsert, then simulate 200,000 subscriptions with Zipf-distributed queue depths totalling 20 million deliveries. - Add one endpoint that always times out after 10 seconds and one that always answers in 20 ms, then measure the fast endpoint's throughput with and without the per-endpoint breaker.
- Instrument heap size across the run.
Expected result: Heap size stays at or below 200,000 regardless of the 20 million deliveries. The fast endpoint's throughput is unaffected by the dead one once the breaker trips. The dead endpoint's deliveries accumulate in their own deque and cost exactly one heap entry.
Verify it yourself
- The asserted invariant never fires under concurrent workers.
- Replaying the same delivery set twice produces identical per-subscription ordering, even though the interleaving across subscriptions differs.
- Replacing full jitter with fixed backoff makes retry attempts cluster visibly in a histogram of attempt timestamps.
Follow-up
- One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
- A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
- The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?
Find the join that inflates every invoice total
Medium · join fan-out · aggregation grain · sql correctness
invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.
Approach
- Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row:
sum(l.amount_minor)is the true total multiplied by the attempt count, andcount(p.attempt_id)is attempts times lines. Three lines and two attempts report double the money and six attempts. - Reject the reflex repair.
count(distinct p.attempt_id)does fix the count, because attempt_id is unique.sum(distinct l.amount_minor)does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match. - Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
- Keep invoices with no attempts by making the attempt branch a LEFT JOIN with
coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about. - Push each filter to its own grain:
where l.voided_at is nullbelongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants. - Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.
Worked solution (25 min)
- Create one invoice with three lines of 1000, 1000 and 500 minor units and two payment attempts, then run the original query.
- Confirm it reports 5000 and 6 rather than 2500 and 2.
- Apply
sum(distinct l.amount_minor)and confirm the total becomes 1500, which is worse rather than better. - Write the two-CTE version with a LEFT JOIN and coalesce, and confirm 2500 and 2.
- Add a second invoice with lines and no attempts and confirm it still appears.
Expected result: The original sum equals the true line total multiplied by the number of payment attempts and the count equals attempts times lines; the corrected query pre-aggregates each branch to invoice grain in separate CTEs and LEFT JOINs them with coalesce, with voided lines excluded inside the line CTE and the tenant predicate applied to both branches.
Verify it yourself
- The money total is unchanged by adding or removing payment attempts.
- An invoice with zero attempts appears with its total and a zero count.
- Two lines with identical amounts both contribute to the total.
- Voiding a line changes the money figure and leaves the attempt count alone.
Follow-up
- Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with
filter (where ...)be better? - Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
- The same report is needed per tenant per month. What index makes the line CTE cheap?
Migrate a live partitioned event table without blocking ingest
Hard · online migration · ddl locks · expand and contract · partitioning
usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.
Approach
- Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so
alter column ... typerewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement. - Expand: add
quantity_numeric numeric(20,6)andenvironmentwith its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each withlock_timeoutset to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage. - Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
- Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
- Make NOT NULL cheap with the three-step form:
add constraint ... check (environment is not null) not valid(brief ACCESS EXCLUSIVE, no scan), thenvalidate constraint(SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), thenset not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent. - Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.
Follow-up
- A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
- The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
- Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.
Leasing sandboxed job runs without double execution
Medium · leases · fencing · state machines · metering
job-runner leases work from a queue and executes customer workloads in sandboxes with CPU, memory, wall-clock and egress limits. Run durations span 200 ms to 30 minutes, thousands are concurrent, and job_run.status must reach exactly one terminal state among succeeded, failed, timed_out, cancelled and lost. A worker can pause 45 seconds for garbage collection, or be briefly partitioned, after its sandbox has exited but before it writes the outcome. Design the lease duration, the fencing, and timeout ownership, and state how billable_seconds is computed for a run whose worker never returns.
Approach
- Resolve the lease-duration conflict rather than picking a compromise. A lease shorter than the longest legitimate run re-dispatches work that is still executing; a lease long enough for a 30-minute run leaves a crashed 200 ms run undetected for half an hour. Use a short lease - about 30 seconds - renewed by a heartbeat every 10 seconds while the supervisor is alive, each renewal pushing leased_until to now() + 30 s. Detection latency is then one lease, not one heartbeat: a dead worker's last successful heartbeat landed at most 10 s before it died, so leased_until expires 20 to 30 s after the death, plus whatever interval the reaper scans on. What the 3:1 ratio of lease to heartbeat buys is headroom - one or two missed heartbeats change nothing, and a stall only becomes a re-dispatch once it outlives the lease remaining when it began, between 20 and 30 s here. Both costs follow from those two numbers: heartbeat write load proportional to concurrent runs (3,000 runs at a 10-second interval is about 300 updates/second), and any pause longer than the lease - the 45-second GC - re-dispatching a run that is perfectly healthy.
- Fence the outcome write so the store, not the worker's memory, arbitrates: UPDATE job_run SET status = $1, finished_at = $2, exit_code = $3 WHERE run_id = $4 AND status = 'running' AND lease_token = $5, and require exactly one row affected. Zero rows means this worker was fenced while it was paused, and its correct behaviour is to discard the result, not to retry - the retry is how a resumed worker overwrites a newer attempt.
- Make the status machine monotone with a check constraint or a trigger so no terminal row can move, and model a retry as a new row with parent_run_id set and attempt incremented rather than a reset of the old one. That is what keeps how many times did this actually execute answerable afterwards and keeps each attempt's resource usage attributable to itself.
- Put the wall-clock timeout in the supervisor, never in the workload: a customer-supplied program asked to time itself out will not. Distinguish timed_out, which the supervisor observed and caused, from lost, which nobody observed at all, and leave exit_code null unless the supervisor actually saw the process exit. Recording lost as failed asserts an outcome no one witnessed and then bills and retries on that assertion.
- Decide billable_seconds for the unobserved case explicitly, because it cannot be derived. It is not now() - started_at, which bills queue and pause time the customer never consumed. The two defensible policies are leaving it null while status = 'lost' and billing nothing, or flooring it at the last heartbeat's observed running time as a stated lower bound; pick one and write it down. Add a per-tenant concurrency cap on sandbox slots so one tenant cannot occupy the whole fleet while these questions are being answered.
Follow-up
- A worker returns from a 45-second GC pause and writes succeeded. Walk through what the database does statement by statement.
- Heartbeats are now a few hundred writes a second against job_run. How do you keep that off the path that matters, and what do you lose by moving it?
- The re-dispatched workload has an external side effect the first attempt already performed. What does your design owe the customer, and what can it not fix?
A resumable usage export that never skips a row
Easy · pagination · indexing · tenant scoping · visibility
Customers pull their own rows from usage_event through GET /v1/usage to reconcile against their own systems. The table is append-only and partitioned daily on ingested_at; a large tenant adds millions of rows a day while the export is being walked, and a client may pause for hours and resume. Specify the cursor, the index it requires, the tenant scoping, the ordering guarantee you can honestly offer, the per-page cost as the walk deepens, and what the client must do to avoid missing rows.
Approach
- Rule out LIMIT/OFFSET on two independent grounds and say both, because fixing only one leaves the other. Correctness: rows inserted between page requests shift the window, so a walking client skips rows and repeats others, which for a reconciliation consumer is silent data loss rather than an error anyone sees. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N x page_size and a deep page degrades from milliseconds to seconds.
- Use keyset pagination over a stable, unique, indexed ordering: WHERE tenant_id = $1 AND (ingested_at, event_id) > ($2, $3) ORDER BY ingested_at, event_id LIMIT $4, carrying the last row's pair as the cursor. The row-value comparison navigates a composite btree directly, so each page is O(log n + page_size) and stays constant as the walk deepens. The precondition is that the cursor columns never change value for a row, which ingested_at satisfies and updated_at would not.
- Lead the index with tenant_id - (tenant_id, ingested_at, event_id) - which is simultaneously the correctness guard and the plan choice. An index on (ingested_at) alone forces a filter across every tenant's rows, and on a table where one tenant holds most of them that is fine only for that tenant and terrible for everyone else. Because ingested_at is also the partition key, a resumed cursor prunes to the partitions from the cursor forward.
- Name the visibility hazard rather than assuming it away: in PostgreSQL now() is transaction start time, so a transaction that starts at T, inserts, and commits at T+8 s writes a row whose ingested_at is T but which becomes visible only at T+8 s. A walker that has already passed T never returns it. The gap equals the writer's longest transaction, so the mitigation is either a safety lag - serve only rows older than now() minus the longest permitted transaction - or a client that re-walks a trailing overlap window and deduplicates on event_id, which is stable and unique.
- Offer the guarantee you can actually keep: ordering by (ingested_at, event_id) with no claim whatsoever about occurred_at order, and completeness only behind the safety lag or with the documented overlap-and-dedup obligation on the client. Saying this in the API reference is part of the design, because the client's reconciliation logic is what has to absorb it.
Worked solution (20 min)
- Write the keyset query and the exact index it needs, then read the query plan and confirm there is no Sort node.
- Insert rows concurrently while walking with a LIMIT/OFFSET pager and count the distinct rows returned against the rows that exist; repeat with the keyset pager and compare.
- Construct the out-of-order commit case by hand: open a transaction, insert, hold it open while the walker passes that timestamp, then commit, and check whether the walker ever returns that row.
- Choose the mitigation - safety lag or client overlap plus event_id dedup - and write down the number it depends on.
Expected result: Keyset pagination on (tenant_id, ingested_at, event_id) with a matching composite index, constant per-page cost and no sort; the guarantee limited to ingested_at ordering with nothing claimed about occurred_at; completeness delivered by a safety lag equal to the longest permitted write transaction, or by a documented client overlap window with deduplication on event_id.
Verify it yourself
- The plan for a page should be an index scan with no Sort node. A Sort means the index column order does not match the ORDER BY and the constant-cost claim is false.
- Under concurrent inserts the offset pager and the keyset pager return different sets. Count the difference rather than reasoning about it.
- Your safety lag must be at least the longest transaction the writing service permits. If you cannot name that number, the completeness guarantee is not enforceable and should not be documented.
Follow-up
- The customer wants to reconcile by occurred_at instead of ingested_at. What breaks, and what would you offer them in its place?
- One tenant starts a full-history export. How do you keep it from occupying every connection in the pool?
- A customer reports a missing row. What do you check first, and what would each answer tell you?
One tenant's counter writes stall the whole connection pool
Medium · lock contention · connection pool · multi-tenancy
A change that made a per-tenant usage counter correct now produces site-wide latency whenever one large tenant writes: unrelated endpoints time out waiting for a connection while database CPU stays low and no statement is slow. The change wraps the counter update in a transaction that takes SELECT ... FOR UPDATE on one row, calls an external pricing service, then updates and commits. Give an ordered checklist, the arithmetic that bounds that tenant's write rate, and three repairs with the cost each one accepts.
Approach
- Separate waiting from working. Low database CPU alongside high application latency points at a queue, so instrument connection-acquisition wait separately from query execution time; that queue forms in the application and is invisible in database metrics, which is why the database looks healthy throughout.
- Confirm the lock rather than assuming it: sample waiting sessions and group by wait event, relation and tuple. Contention concentrated on one tuple belonging to one tenant is the signature; a deadlock would instead show the database aborting transactions after its detection timeout, which is not happening here.
- Do the arithmetic out loud. Throughput on a serialised row is one divided by the lock hold time, and the hold spans the external call, so a 20 ms pricing call caps that tenant near 50 writes per second no matter how many pods run. Every waiter also holds a pooled connection while it queues, so the shared pool drains and unrelated tenants fail at acquisition.
- Repair one: shrink the critical section to a single statement with the price resolved before the transaction opens. Cost is a stale price for the duration of one request and a second round trip; benefit is a hold time measured in the database's own execution time.
- Repairs two and three change where the contention lives rather than how long it is held. Sharding the counter into per-(tenant, bucket) rows and summing on read multiplies write throughput by the shard count, at the cost of an aggregate on every read and a shard count you must size against the largest tenant rather than the median. Accumulating in memory and flushing periodically removes the per-write round trip entirely, paid for with a bounded loss window on crash, which is acceptable for a rate limiter and not for a billing counter.
- Contain independently of which repair wins: a separate pool or per-tenant concurrency cap for this write class, a statement timeout low enough that a pathological query dies before it accumulates waiters, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks.
Follow-up
- What would a genuine deadlock look like here, which two code paths would produce one, and how does the database's response differ from what you observed?
- If a transaction-pooling proxy sits in front of the database, which of your three repairs changes behaviour, and what stops working that would have worked on a direct connection?
- The counter also enforces a quota. Why is SELECT the count and then INSERT still wrong after you have fixed the contention?
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
'We rewrote it in a better framework' is not a story
A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.
Can you describe a challenging technical project you led and the outcome?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Why do you want to switch jobs at this stage in your career?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What did you decide not to do, and why?
- How did you know your change caused the improvement?
How do you handle data consistency in a distributed system?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?