Interview guides · INCATech

INCATech Software Engineer Interview Questions & Guide 2026

A prep plan for INCATech software engineer candidates: answer first, read the solution second, repeat the misses later.

The 60-second brief

A Software Engineer at INCATech is a vital contributor to the company’s mission of delivering transformational technical solutions for the Department of Defense and the Intelligence Community. You are not just writing code; you are building systems that support critical national security missions, often working within the specialized domains of Cybersecurity or Geospatial Information Systems.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

Bound blast radius with per-tenant concurrency limits
Build at-least-once pipelines with explicit deduplication horizons
Keep money in integer minor units

Overview

This role is inherently collaborative and requires a balance of deep technical rigor and a service-oriented mindset. Whether you are hardening systems against vulnerabilities, automating complex geospatial workflows, or integrating security into DevSecOps pipelines, your work directly influences the effectiveness of the government programs you support. You will be expected to navigate classified environments and communicate effectively with stakeholders, making this an ideal role for engineers who thrive on high-stakes, mission-driven problem solving.

The INCATech interview process

INCATech candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Discussions

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Leadership Discussions

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
Whether you can be redirected in the middle of an answer without restarting it from the beginning
Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Where candidates lose points

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

One shared connection pool for every tenant and every query class

A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.

Abandoning working code to chase the optimal solution

Get the straightforward version correct, state its complexity, and only then optimise, keeping the working version until the faster one passes the same cases. A correct quadratic solution with a stated path to linear beats a half-written optimal one that never ran.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Questions & practice

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Worked solution (30 min)

Define the four structures explicitly: queues: subscription_id -> deque[delivery], ready: min-heap of (next_attempt_at, subscription_id), inflight: set[subscription_id], breaker: min-heap of (half_open_at, subscription_id).
Write down the invariant you will assert after every operation: a subscription appears in at most one of ready, inflight and breaker, never in two.
Implement due, complete and insert, then simulate 200,000 subscriptions with Zipf-distributed queue depths totalling 20 million deliveries.
Add one endpoint that always times out after 10 seconds and one that always answers in 20 ms, then measure the fast endpoint's throughput with and without the per-endpoint breaker.
Instrument heap size across the run.

Expected result: Heap size stays at or below 200,000 regardless of the 20 million deliveries. The fast endpoint's throughput is unaffected by the dead one once the breaker trips. The dead endpoint's deliveries accumulate in their own deque and cost exactly one heap entry.

Verify it yourself

The asserted invariant never fires under concurrent workers.
Replaying the same delivery set twice produces identical per-subscription ordering, even though the interleaving across subscriptions differs.
Replacing full jitter with fixed backoff makes retry attempts cluster visibly in a histogram of attempt timestamps.

Follow-up

One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Follow-up

The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
Ingest is still running while you investigate. What makes your two numbers comparable at all?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
What makes a re-run over the same day produce byte-identical rollups?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Model credential revocation so history survives the delete

Easy · soft delete · partial indexes · check constraints · audit history

tenant_api_key stores key_id, tenant_id, workspace_id, name, key_prefix, secret_hash, scopes text[], status (active, revoked, expired, compromised), auth_version, created_at, expires_at, last_used_at, revoked_at, revoked_reason. Rotation inserts a new row and revocation never deletes, because an incident review asks which credential served a request last quarter. Write the constraints that enforce: a label is unique only among a tenant's live keys, revoked_at and status can never disagree, and scopes is never empty. Then write the authentication lookup predicate, and name one column in this table that must stay out of it.

Approach

Reach for a partial unique index rather than a plain UNIQUE: create unique index on tenant_api_key (tenant_id, name) where revoked_at is null. Any number of revoked rows may share a label, the live namespace stays unique per tenant, and the revoked majority is not in the index at all, so it stays small on a table that only grows.
Tie the nullable timestamp to the enum so the two cannot drift: check ((revoked_at is not null) = (status in ('revoked','compromised'))) and check ((revoked_at is null) = (revoked_reason is null)). A revocation that records no reason is the one an incident review cannot use.
Write the emptiness check as check (cardinality(scopes) > 0), not array_length(scopes, 1) > 0. array_length returns NULL for an empty array, a CHECK constraint passes when its expression is NULL, so the array_length version accepts exactly the value it was written to reject.
Make the lookup a single index probe with every liveness condition inside it: where secret_hash = $1 and revoked_at is null and (expires_at is null or expires_at > now()) and auth_version = $2, backed by a unique index on secret_hash. Nothing is filtered in application code, so there is no path that forgets a clause.
Keep last_used_at out of that predicate. It is written asynchronously and is allowed to lag by a minute, so it is a usage signal; feeding it into an authorisation decision makes the decision depend on a write that may be late, batched away or lost.
Flag the modelling smell while you are here: expired is derivable from expires_at < now(), so storing it as a status obliges a job to keep it true and guarantees the column is wrong between the expiry instant and that job's next run. Derive it in the predicate; keep the stored status for states that are decisions rather than clock readings.

Worked solution (20 min)

Create the table with all three constraints on a scratch database and insert two revoked rows sharing (tenant_id, name); the partial index should accept both.
Insert a second live row with that same name and confirm the violation names the partial index.
Run update tenant_api_key set revoked_at = now() leaving status = 'active' and confirm the CHECK rejects it; then try insert ... scopes = '{}' against both the cardinality and the array_length forms and note that only one rejects it.
Run explain (analyze, buffers) on the lookup predicate for a live key and confirm an index scan on secret_hash with rows removed by filter equal to zero.

Expected result: A partial unique index on (tenant_id, name) where revoked_at is null; check ((revoked_at is not null) = (status in ('revoked','compromised'))); check ((revoked_at is null) = (revoked_reason is null)); check (cardinality(scopes) > 0); a single-probe lookup on secret_hash plus revoked_at, expires_at and auth_version; last_used_at excluded, with a note that 'expired' should be derived rather than stored.

Verify it yourself

Two revoked keys with the same name insert cleanly; a second live one does not.
The array_length form of the emptiness check accepts an empty array and the cardinality form rejects it.
No statement anywhere in the design deletes or updates a revoked row.
Revoking a key touches exactly one row and takes effect on the next lookup with no application-side filtering.

Follow-up

Rotation issues a replacement while the old key stays live for a 30-day overlap. What does the uniqueness rule become, and what does the UI show to tell two same-named keys apart?
A password reset bumps the principal's auth_version. No row in this table changed. How does the next request fail, and what query counts how many keys that bump just killed?
A key turns up in a public repository. Which columns let you find it, and what do you write to the row?

How do you approach dealing with customers who are very detail-oriented?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

How do you work in team-based environments?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Thoughts about working for a small business.

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Ship three breaking-looking changes without breaking pinned SDKs

Hard · versioning · backward compatibility · deprecation

GET /v1/runs/{id} returns status from the set queued, leased, running, succeeded, failed, timed_out, cancelled. You must add the terminal state lost, add an optional billable_seconds field, and start rejecting name values over 64 characters that were previously accepted. Clients are generated SDKs pinned inside customer build pipelines you cannot redeploy or reliably contact, some with strict deserialisers. Classify each change as compatible or breaking and justify it mechanically, then give the rollout: the versioning scheme, the transform old clients receive, the telemetry, and the sunset process.

Approach

Classify against a deserialiser rather than against intuition. A new enum value breaks a generated client that validates the enum or switches exhaustively; a new optional field breaks one generated with additionalProperties false; tightening validation breaks every caller already sending an 80-character name even though no field moved. All three are breaking for some real client, and that is the answer, not a technicality.
Choose the versioning axis on maintenance cost. A path version forks the handler, and each subsequent change multiplies the fork. A date-pinned version header, defaulted per credential at first use and pinned thereafter, keeps one internal model and a chain of ordered response transforms, so the handler stays single and the transforms compose.
Write the downgrade honestly. Older versions have no lost, so the transform maps it to the least-wrong existing value and the changelog says which; mapping it to failed asserts an outcome nobody observed, so pair it with a reason field the old version already exposes rather than inventing a field, and keep the mapping in one documented table instead of a conditional inside the serialiser.
Sequence the validation tightening separately from the payload changes: first measure how many tenants send names over 64 characters, then warn for a stated window with a Deprecation header and a non-fatal warning code, then reject only for versions dated after the change, never retroactively for a pinned client.
Deprecate from evidence, not from announcement: per-version, per-tenant, per-credential request counts from the gateway, a Sunset header carrying the date, direct contact for the handful of accounts that make up most of the tail, and short scheduled brownouts before the cutoff so the clients nobody can reach fail on your calendar rather than during their release.

Worked solution (40 min)

Write the three-row classification table with the client-side mechanism that breaks in each case, not just a verdict.
Pick the versioning scheme and write the header, the default-and-pin rule for a new credential, and where the transform chain sits relative to the handler.
Write the lost mapping for pre-change versions as an explicit table entry plus the changelog sentence a customer reads.
Write the validation timeline with four dated steps, and mark which one is visible to clients that never change their version.
Write the deprecation plan: the telemetry query by version and tenant, the headers, the brownout schedule, and the fallback for the largest holdout.

Expected result: All three changes classified breaking with the mechanism named, a date-pinned version header with one internal model and ordered downgrade transforms, a documented lost mapping, validation tightened only for versions after the change, and a sunset driven by per-version traffic with brownouts.

Verify it yourself

No change is called additive without naming the client behaviour that makes it safe.
A client that never changes its version header sees no behaviour change from any of the three.
The plan names a measurement that tells you when sunset is safe.
The lost mapping is stated once, in a table, not implied by serialiser branches.

Follow-up

Which step of your rollout cannot be rolled back cleanly once a client has depended on it?
A tenant pinned to a sunset version breaks their build the morning after the cutoff. What in your plan already lets you resolve it within an hour?
Your transform chain is now six versions deep. What stops it becoming the fork you were avoiding?

Webhook workers leak until OOM and drop in-flight deliveries

Medium · memory leak · eviction · leases

webhook-delivery workers grow from 400 MB to a 2 GB limit over about 36 hours, are OOM-killed, restart, and repeat. Each restart abandons in-flight attempts, so webhook_delivery rows sit in in_flight until their leases expire and the backlog spikes. The live set measured after a forced full collection also grows. The fleet serves tens of thousands of subscriptions, several thousand of which have been failing for weeks. Give an ordered checklist, the measurement separating retention from fragmentation, and the fix.

Approach

Separate the two failure shapes with one measurement: track resident set size against the live set after a forced full collection. A live set that climbs monotonically is retention; a flat live set under a rising RSS is fragmentation, off-heap or native allocation, or an allocator that never returns pages. The stated symptom puts this in the first category, which rules out allocator tuning as a fix.
Characterise the curve rather than the total. Growth linear in uptime implies an unbounded structure keyed by something that keeps arriving; step growth implies buffering a large object. Correlate the slope against event rate and separately against the count of distinct subscriptions seen, because those two diverge and only one of them will fit.
Diff two heap snapshots an hour apart by retained size grouped by dominant root, not by allocation count, which is dominated by short-lived objects and will point at the wrong thing.
Expect a per-subscription map with no eviction: circuit-breaker or backoff state created on first failure and never removed, so the retained set grows with endpoints that have ever failed, and the several thousand permanently dead endpoints hold theirs forever.
Fix in two places. Bound the in-memory structure with a size-capped LRU or a TTL keyed on last use, and move state that must survive a restart onto the subscription or webhook_delivery row, since the worker holding it in memory is exactly why a restart loses it.
Repair the second-order damage separately, because it will outlive the leak: workers claim by compare-and-set with leased_until, so a bounded lease returns in_flight rows to pending on a known schedule, and a graceful shutdown releases leases instead of waiting them out.

Follow-up

The backlog spike after a restart is itself a thundering herd against customer endpoints. What stops the recovery from becoming a second incident?
Suppose the live set had been flat while RSS still climbed. Name two causes and the measurement that separates them.
How would you size the LRU, and what does a miss on an evicted circuit-breaker entry cost a customer whose endpoint is down?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

Disclose a cross-tenant webhook delivery to affected customers

Medium · cross-tenant leak · disclosure · blast radius · authorisation checks

An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.

Approach

Bound the population before saying anything externally. The affected set is deliveries in the window where the event's tenant and the subscription's tenant differ; the ones that actually left are those with delivered_at set and a 2xx in last_response_code. Attempted and delivered are two different counts and a disclosure has to use the right one in the right sentence.
Separate what the records prove from what they do not, and say both halves rather than the flattering one. They prove which payloads were signed, where they went, and — through payload_digest — exactly which bytes. They do not prove what the receiving system did with them, and they do not bound the window more precisely than your deploy timestamps do.
Communicate on the facts you hold, with the scope stated as an upper bound: 'at most eleven payloads, four recipient endpoints, these fields, this window' is more useful and more honest than waiting a day for certainty. The field list matters more than the event count, because a customer cannot assess exposure from 'an event'.
Name the code change precisely, because this class never originates in the delivery worker. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before the payload is signed, and make the second comparison drop the delivery rather than log a warning. Say why one check is insufficient: the enqueue check protects against the bug you know about, the pre-signing check protects the boundary itself.
Run the history question in parallel and say so: a query over historical deliveries for the same mismatch tells you whether this was nineteen minutes or a year, and you would rather find the second case yourself than have a customer find it after your disclosure.
Split the response into workstreams with owners — recipients asked to delete, affected customers notified, the check landed with a test, history swept — and say which you personally drove and which you handed off. Claiming all four is not credible and claiming none is not ownership.

Follow-up

The historical sweep finds two more instances from last year. What changes in what you have already told people?
Who approves the wording, and what do you do when you are asked to soften the scope?
A customer asks you to prove a redelivery contained the same bytes as the original. What do you show them?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?

Ship metered billing with a named deduplication horizon

Medium · technical debt · deduplication · deadline pressure · detectors

Metered billing must be on in three weeks. usage_event is partitioned daily, so its unique index must include the partition key and deduplicates only within a day: a producer retry that crosses midnight, or a replay run a week later, gets through. A cross-partition dedup store is two weeks you do not have. Describe shipping with debt you named in advance: what you shipped, what you wrote down, the detector you added, the trigger and date for paying it off, and what you would have refused to ship under the same pressure.

Approach

Show you can separate the two kinds of debt, because that distinction is what the question actually probes. Debt that costs engineering time later is shippable on a deadline. Debt that silently corrupts a number a customer gets charged for is not shippable unless the corruption is detectable, and detectability is the whole negotiation.
Make the exposure narrow and measured rather than gestural. The hole is duplicates whose occurrences straddle a UTC day boundary, plus any replay older than partition retention. Measure it before arguing about it: how often an idempotency_key recurs at all, and the distribution of the gap between first and last occurrence. If the ninety-ninth percentile of that gap is four minutes, the residual risk is a small band around midnight and you can say so numerically.
Add the detector before the feature, not after. A nightly job counting keys that appear in more than one partition is one grouped scan over recent partitions, and it converts a silent overcount into a page. State what it costs to run and what it fires on.
Buy the cheap half of the real fix immediately: extend partition retention so the dedup horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. That reframes retention as a correctness parameter rather than a storage cost, which is the sentence you need on record before someone optimises the bill.
Make repayment mechanical instead of aspirational: a dated entry with a named owner, plus a threshold that pulls the date forward — first detector hit above N events, or first customer dispute. Debt with a trigger gets paid; debt with only a date does not.
Answer the second half honestly by naming what you would refuse under identical pressure: the sealing path, because a sealed row is frozen and a wrong number there stops being a bug and becomes an adjustment line, a dispute and an audit question.

Follow-up

The detector fires on forty duplicate events for one tenant, and two of their invoices have already sealed. What happens next?
Whom did you tell that the billing numbers had a known hole, and in what words?
Finance asks you to cut storage by shortening partition retention. What do you say, and to whom?