Interview guides · Kovai

Kovai Software Engineer Interview Questions & Guide 2026

API design practice for Kovai software engineer candidates: compatibility, versioning, and changes you cannot take back.

The 60-second brief

As a Software Engineer at Kovai, you play a central role in building, scaling, and maintaining enterprise-grade SaaS products that serve thousands of global business users. Kovai is widely recognized for its high-impact product portfolio—most notably Document360, an award-winning knowledge base and documentation platform, alongside specialized cloud management tools. In this position, you contribute directly to product architecture, end-to-end feature development, performance optimization, and seamless cloud deployments.

The behavioural round is a technical round in narrative form. Prepare it by collecting specifics you actually owned, such as a design you argued against, an incident you diagnosed, or a decision you later reversed, rather than by rehearsing phrasing.

Focus areas

Bound blast radius with per-tenant concurrency limits
Build at-least-once pipelines with explicit deduplication horizons
Keep money in integer minor units

Overview

The engineering organization at Kovai values hands-on practical execution, clean code standards, and modern development practices. You will work within cross-functional product teams using modern full-stack web technologies like.NET Core, Angular, TypeScript, and Microsoft Azure. The work involves building robust RESTful Web APIs, engineering high-concurrency cloud services, designing intuitive web interfaces, and integrating modern AI capability frameworks to keep products ahead of market trends.

This role requires a blend of deep foundational technical knowledge and practical problem-solving. Whether you are re-architecting APIs for low-latency data access, optimizing complex database queries, or participating in machine coding rounds during the hiring process, Kovai evaluates how effectively you transform complex technical challenges into clean, maintainable, and scalable software solutions.

The Kovai interview process

Kovai candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Evaluations

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Team Discussions

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

HR Discussion

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Going silent while thinking

Narrate the candidates and why you are discarding them, even in fragments: sorting first would make this a two-pointer scan, but it destroys the original indices, which the output needs. From the other side of the table, a candidate thinking hard and a candidate stuck are indistinguishable until one of them speaks.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Questions & practice

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Worked solution (20 min)

Write the null policy as three lines of prose first, one per case, and keep them beside the output.
Emit 2n endpoint tuples (timestamp, delta, run_id) and sort on the key (timestamp, delta) so -1 precedes +1.
Sweep, tracking cur, peak, peak_at updated only on a strict increase, and the first run_id whose +1 takes cur to C+1.
Build a fixture with two runs where one ends exactly when the next starts, three genuinely overlapping runs, one run with a null finished_at and status running, and one with status lost and a 300-second wall_clock_limit_seconds.
Re-run with every timestamp shifted by a constant and confirm the peak is unchanged while peak_at shifts by the same constant.

Expected result: Peak is 3, from the overlapping trio. The exact-handoff pair yields a peak of 1, not 2. peak_at is the start instant of the third overlapping run. The lost run occupies exactly [started_at, started_at + 300s) under the stated policy.

Verify it yourself

Setting peak_at on >= rather than > moves the reported instant to the last time the peak was attained. Flip it and watch the fixture change.
Sorting by timestamp alone, without the delta tie-break, raises the reported peak on the handoff fixture.
The deltas over the whole sweep sum to zero once every open-ended run has been clipped.

Follow-up

Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Hold a tenant to a trailing sixty-second request limit

Medium · sliding-window · two-pointer · rate-limiting · tenant-skew

The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.

Approach

Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request.
Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate, prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact.
Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (tokens, last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual.
Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.

Follow-up

One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
Quotas rather than rate limits: the check is select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes.
How do you return an accurate Retry-After from the exact algorithm without a second scan?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Follow-up

The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

What is a Content Delivery Network (CDN), and how does it improve front-end asset delivery and knowledge base loading speeds?

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
Name the failure you are designing for, then the recovery path.
State the consistency you need, and where you are willing to be stale.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

How do you design and implement role-based access control (RBAC) and user management across a distributed web application?

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Name the read and write paths separately; they rarely have the same bottleneck.
Choose a partition key and say what query it makes expensive.

Follow-up

What breaks first when traffic grows ten times?
How does this behave when that dependency is down for an hour?

How would you re-architect a single-node API to scale for millions of concurrent requests while maintaining low latency and data consistency?

Medium · system design and architecture

Approach

Choose a partition key and say what query it makes expensive.
Name the failure you are designing for, then the recovery path.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

How does this behave when that dependency is down for an hour?
What would you drop to keep the system up under load?

Explain cloud computing delivery models (IaaS, PaaS, SaaS) and describe how automated cloud deployment pipelines should be structured.

Medium · system design and architecture

Approach

Fix the scope first: who calls this, how often, and what they do when it fails.
Name the read and write paths separately; they rarely have the same bottleneck.
Choose a partition key and say what query it makes expensive.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

Explain how you integrated Generative AI tools or APIs into your application code, including how you validated generated output and handled edge cases.

Medium · api and integration design

Approach

Define the identity of a request so a retry cannot double-apply it.
Separate accepted, pending, failed and confirmed; they are different facts.
Design the error taxonomy before the success shape; callers branch on it.

Follow-up

What does a partial failure look like to the caller?
How does a client discover it is on an old version of this contract?

Describe how you handle API calls using HTTP clients (such as Axios or Angular HttpClient) and manage error boundaries in front-end applications.

Medium · api and integration design

Approach

Design the error taxonomy before the success shape; callers branch on it.
State how the contract changes without breaking existing clients.
Define the identity of a request so a retry cannot double-apply it.

Follow-up

How does a client discover it is on an old version of this contract?
What happens if the caller retries after a timeout?

Explain asynchronous programming using async and await in modern backend frameworks, and discuss how it improves request throughput.

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Explain the role of caching layers and how object-relational mapping (ORM) performance issues like N+1 queries can be avoided.

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Metering ingest that survives a six-hour producer replay

Hard · idempotency · dedup horizon · partitioning · watermarks

metering-ingest consumes usage events at-least-once - 250M/day, 10-40k/second at peak - and folds them into usage_rollup_hourly keyed (tenant_id, workspace_id, sku, hour_start). usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). A producer outage ends in a six-hour replay that re-sends events already ingested, some of whose originals crossed midnight. Design the consumer: partitioning, where the acknowledgement sits relative to the commit, the deduplication horizon and its storage cost, and how the rollup watermark advances. Nothing may be double-counted and nothing may be silently dropped.

Approach

Choose the acknowledgement position deliberately and name what each choice costs. Acknowledging after the fold commits makes the consumer at-least-once: a crash between the two replays the batch and produces duplicates, which are ordinary and absorbable. Acknowledging first makes it at-most-once: a crash between the two drops revenue with no error raised anywhere and no way to detect it later. Take at-least-once and design everything downstream to absorb duplicates.
Put the dedup and the fold in one transaction so there is no window between them. Insert the batch into usage_event with ON CONFLICT DO NOTHING, take the rows actually inserted, and fold only those into usage_rollup_hourly with an upsert on (tenant_id, workspace_id, sku, hour_start) bucketed by occurred_at, not ingested_at. A batch of about 2,000 rows is one round trip and one index probe per event.
Attack the partition-key flaw head on: the unique index includes ingested_day because a unique index on a partitioned table must contain the partition key, so the same (tenant_id, idempotency_key) re-sent after midnight is a different index entry and passes. Deduplicate instead against a store keyed (tenant_id, idempotency_key) with no date component, whose horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. At 14 days that is 250M x 14 = 3.5 billion keys, which is a dedicated key-value store, not a larger index on the same table. The alternative - partitioning usage_event on (tenant_id, occurred_day) so the natural key is stable - fixes dedup but loses pruning on ingest time and makes retention by dropping partitions awkward.
Partition the consumer by hash of tenant_id so one tenant's replay stalls only its own partitions, and give replay traffic a separate lower-priority lane so live ingest keeps its latency. The cost is explicit: that tenant's watermark lags while the replay drains, and everything gated on the watermark waits for it.
Define the watermark as a property of committed work, not of wall-clock time: per partition it is the largest occurred_at such that every event with a smaller occurred_at has committed, and the sealing decision uses the minimum across partitions. Record source_max_ingested_at on every rollup row so any number can prove what it did and did not include, and keep restatement legal only while status = 'open' - after sealed_at the value is frozen and a late event becomes an invoice adjustment instead.

Worked solution (40 min)

Write the consumer loop in pseudocode with the acknowledgement after the commit, then annotate each line with what is lost or duplicated if the process dies exactly there.
Size the dedup store: events/day x horizon_days keys, bytes per key including the tenant prefix, and the resulting memory or disk. Compare that cost against simply extending retention on the partitioned table and say why the latter does not fix the problem.
Take one event ingested at 23:59:58 and replayed at 00:00:04 and work out its fate under (a) the partitioned unique index alone and (b) the separate dedup store.
Write the per-partition watermark formula, then what the seal uses, then what a single stalled partition does to sealing.

Expected result: An at-least-once consumer acknowledging after the commit; deduplication on (tenant_id, idempotency_key) in a store whose horizon of at least 14 days exceeds max retry window plus supported replay length, sized at roughly 3.5 billion keys; tenant-hashed partitions with replay on a separate lane; a per-partition watermark with sealing gated on the minimum across partitions; source_max_ingested_at written on every rollup row.

Verify it yourself

The midnight-crossing retry must be rejected by whatever you chose. If the table's unique index is your only dedup, it is accepted and the invoice is wrong by exactly that event.
You should be able to name a distinct failure for acknowledge-before-commit and for acknowledge-after-commit. If both sound the same, the position was not actually chosen.
quantity_sum for any (tenant, workspace, sku, hour) must be reproducible by re-aggregating usage_event filtered to ingested_at <= source_max_ingested_at. If it is not reproducible, the rollup is an opinion rather than a projection.

Follow-up

The dedup store is lost entirely. What can you still guarantee, and how do you rebuild it from what remains?
A replay delivers events for an hour that is already sealed. Trace exactly what happens to them, row by row.
One partition is stuck on a poison message, so the minimum-across-partitions watermark never advances and no tenant can be sealed. What is your escape hatch and what does it cost in correctness?

Regional error rate explodes after a dependency merely slows

Hard · retry storm · circuit breaker · partial outage

A control-plane read replica in one region degrades from 4 ms to 120 ms. Within ninety seconds that region's gateway error rate rises from 0.01% to 40% and its p99 becomes bimodal, one mode near the old p99 and one at the client timeout. The other two regions are unaffected. The gateway retries control-plane reads three times with exponential backoff and no jitter. Give an ordered checklist that separates trigger from amplifier, the offered-load arithmetic, and the controls that break the loop.

Approach

Split the incident into three questions before touching a control: what started it, what amplified it, and what would make recovery slow. Here they are the replica slowdown, the retry policy interacting with queueing, and a synchronised unjittered herd at recovery. They are different mechanisms and each needs its own control.
Read the distribution rather than the mean. A bimodal p99 with one mode pinned at the client timeout is two populations, not one degraded path; split latency by cache hit and miss and confirm the fast mode is hits and the timeout mode is misses that reached the replica.
Do the load arithmetic. Three retries turn one client request into up to four upstream requests, so offered load reaches roughly 4x on a dependency that is already slower, and it arrives at the worst moment. With utilisation approaching one, queueing delay grows superlinearly, which is why a 30x latency increase upstream does not produce a 30x increase downstream, it produces timeouts.
Break the loop with controls that bound offered load rather than with more attempts: a concurrency limit on the control-plane client so at most N calls are in flight and the remainder fail fast, a circuit breaker scoped per dependency and region, and a retry budget capping retries at a small fraction of base traffic so amplification has a ceiling that does not depend on how many clients are retrying.
Add full jitter to whatever retries survive, sleeping uniformly in [0, min(cap, base x 2^attempt)], so attempts de-correlate instead of arriving in waves aligned to the moment of failure.
Decide the unreachable-dependency behaviour in advance, because it is the actual product decision underneath: serving from an expired credential cache keeps the product available while extending a revoked key's life past the stated bound, and failing closed converts a dependency degradation into a total outage. State the mode and the staleness number rather than letting the timeout choose.

Follow-up

The replica recovers. Describe what happens in the first ten seconds with your controls in place versus without them.
Which single metric would have paged before the error rate moved, and why is upstream latency by itself not it?
Requests that fail fast under the concurrency limit still need an answer. What does the gateway return, and what does it do to the usage event it would otherwise have emitted?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

The failure that was yours

Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.

Resolve a review disagreement over a quota check

Easy · code review · isolation levels · write skew · disagreement

A colleague's pull request enforces a per-tenant quota by selecting the current count and then inserting when it is under the limit. You flag it as a race. They reply that the transaction already runs at repeatable read, so the snapshot makes it safe, and the tests pass. Walk through taking that disagreement to a resolution: what you write in the review, what you demonstrate rather than assert, which fix you propose and why, and what you do if they still disagree after all of it.

Approach

Answer the claim precisely instead of restating your objection, because they have made a specific technical argument. In PostgreSQL, repeatable read is snapshot isolation; this is write skew, which snapshot isolation permits by design. Both transactions read a count that is stable within their own snapshot, insert disjoint rows that the other cannot see, and both commit, so the limit is exceeded by exactly the concurrency.
Demonstrate rather than cite. Two psql sessions, both BEGIN ISOLATION LEVEL REPEATABLE READ, both select the count, both insert, both commit: it succeeds. Repeat at SERIALIZABLE and the second commit fails with serialization_failure, SQLSTATE 40001. That takes two minutes, ends the argument without anyone conceding a position, and leaves an artefact for the next reviewer.
Offer the options with their costs rather than a verdict. Serialisable plus a retry loop on 40001 is correct but obliges every caller to retry and degrades under contention. An increment-and-compare on a counter row — update tenant_quota set used = used + 1 where tenant_id = $1 and used < limit returning used — is safe even at read committed, because a blocked updater re-evaluates the WHERE clause against the row version it finally locks, and zero rows returned means full. A unique or exclusion constraint that makes the surplus write fail is the third.
Name the plausible non-fix explicitly, since it is what usually gets merged instead: folding the count into the insert as insert ... select ... where (select count(*) ...) < limit is still racy under read committed, because the subquery cannot see the other transaction's uncommitted rows. It looks atomic and is not.
Say what you do if they still disagree: escalate the decision rather than the disagreement. Attach the reproduction, hand it to the service owner or a third reviewer, and state that you will not block the merge if the owner accepts the risk knowingly — and that you want that acceptance written down.
Close with the general lesson worth leaving in the review thread: a passing suite is weak evidence for a concurrency claim because it runs one request at a time. Ask for a test that runs two.

Follow-up

Write the counter-row version. Does your answer change if the quota counts child rows rather than a column?
Under serialisable, who performs the retry, and what does the API client see if the retry also fails?
This is the third disagreement with the same reviewer this month. What changes in how you review?

Own the incident where invoices undercounted metered usage

Medium · incident response · at-least-once · billing correction · postmortem

A metering consumer acknowledged each batch before committing the fold into usage_rollup_hourly. A rolling deploy restarted consumers mid-batch for two hours; roughly 1.4M usage_event rows were acknowledged and never folded, and 61 invoices sealed against the resulting rollups before anyone noticed. Take the owner's role. Describe an incident of comparable blast radius you owned: how it surfaced, the query that sized the loss, what you stopped first, and how the money was corrected. Give a wall-clock timeline and one thing you got wrong while it was still live.

Approach

Open with the invariant that broke and the direction of the error, because they determine everything else: acknowledging before committing makes the consumer at-most-once, so this loses events rather than duplicating them, and loss raises no error anywhere. A listener who hears 'we lost revenue silently' knows immediately why detection took two hours.
Size it with a stated reconciliation rather than an adjective: sum(quantity) from usage_event grouped by (tenant_id, sku, hour of occurred_at) over the window, against usage_rollup_hourly.quantity_sum on the same keys, filtered to environment='production' because staging and sandbox are metered but not billed. Then bisect by hour and tenant until single cells explain the gap. Say how long that ran and whether a replica could serve it while the incident was live.
Separate mitigation from fix and say which came first. Mitigation is holding the sealing job, because a sealed row is frozen by design and every minute of sealing converts a recoverable rollup into an invoice correction. The fix is moving the acknowledgement after the commit, which re-introduces duplicates that the dedup check on (tenant_id, idempotency_key) must now absorb.
State the correction path in the domain's own terms: sealed periods are never edited, so each affected tenant gets an adjustment line on the next invoice with kind='adjustment' and voided_by_line_id pointing at the line it reverses, priced against the same rate tier and carrying the watermark it priced against. That is four separate numbers — tenants affected, minor units, the cycle the adjustment lands in, and when customers were told.
Close on one prevention control with its cost, not five: a per-hour reconciliation comparing raw sum to rollup sum that pages above a threshold. Name the threshold and the false-page rate you accepted, because a detector nobody will keep staffed is not prevention.
Name a mistake you made inside the response window — the wrong first hypothesis, a mitigation that made it worse — rather than a design mistake from six months earlier. That is the part candidates rehearse away and interviewers weight heavily.

Follow-up

Your fix moves the acknowledgement after the commit. What breaks now, and what absorbs it?
One undercharged tenant has since churned. Do you bill them, and who decides?
How would you have caught this in ten minutes instead of two hours, and what would that detector cost you in pages per week?

Reverse a webhook ordering decision after measuring its cost

Medium · reversing decisions · head-of-line blocking · at-least-once · api contracts

You argued for strict per-subscription ordering in webhook-delivery, which means one in-flight attempt per subscription. It shipped. Three months later a single unresponsive endpoint holds one subscription's queue at a six-hour backlog, and two customers report events arriving out of order anyway once their own retries are counted. Describe a decision you reversed: what you originally optimised for, the measurement that changed your mind, what the reversal cost in engineering time and customer change, and how you told the people who had already built on the original guarantee.

Approach

State the original decision as a trade you made knowingly. Ordering across a network requires a single in-flight attempt per subscription, and its price is head-of-line blocking whenever one endpoint is slow. 'We priced it wrong' is a much stronger opening than 'we did not realise', and it is usually the true one.
Bring the measurement that flipped it, not the anecdote: backlog age at the ninety-ninth percentile per subscription, the share of subscriptions where one slow endpoint gated an otherwise healthy queue, and the delivery throughput lost to serialisation. A reversal justified by complaints is indistinguishable from a reversal justified by fatigue.
Name what you learned about the guarantee itself, which is the engineering content of this story. At-least-once delivery means a retried event already arrives after newer ones and the consumer already must be idempotent, so a guarantee the customer has to defend against anyway was never worth what it cost to provide.
Describe the migration, because reversing a published contract is the hard half and the part candidates skip. Parallel attempts behind a per-subscription flag, a monotonically increasing sequence number added to the envelope so order-sensitive consumers can sort or discard, documentation that states at-least-once and unordered in those words, and a deprecation measured in quarters because the client is a pinned SDK inside a build pipeline you cannot see or redeploy.
Give the cost in the two currencies that matter: engineer-weeks, and how many customers had to change code. Then say who you told before it shipped rather than in a changelog afterwards, and which large customer you left on the old behaviour and for how long.
Close with the signal you now weight differently, stated as something you would do earlier next time: measuring the blocking cost on the slowest decile of endpoints before committing to the guarantee, rather than after a customer noticed.

Follow-up

A customer insists they need ordering. What do you offer them that is not global serialisation?
How did you choose the deprecation window given that you cannot see or redeploy the clients?
What would have to be true for you to reverse back?