ConnectWise Software Engineer Interview Questions & Guide 2026

The 60-second brief

A Software Engineer at ConnectWise is a foundational role responsible for building and maintaining the robust software ecosystem that powers IT solution providers globally. You will work on complex, high-impact products—ranging from business automation and remote monitoring to cybersecurity solutions—that serve as the backbone for thousands of managed service providers (MSPs). Your contributions directly influence the operational efficiency and security posture of businesses worldwide, requiring a balance of technical rigor and a user-centric mindset.

Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.

Focus areas

• Bound blast radius with per-tenant concurrency limits
• Scope every query and cache key by tenant
• Evolve APIs without breaking pinned SDK clients

Overview

This role is both challenging and rewarding, as you will navigate a diverse technology stack and a high-scale environment. You will be expected to solve intricate architectural problems, optimize database performance, and deliver scalable code that meets the demands of ConnectWise's partners. Whether you are working on backend services, frontend interfaces, or cloud infrastructure, your work will be critical to sustaining the company’s mission of providing comprehensive, integrated software solutions.

The ConnectWise interview process

ConnectWise candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

• Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
• Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
• Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
• Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

• From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
• Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
• Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Technical Interviews

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

• In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
• In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
• Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

• Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
• Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
• For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Managerial or HR Discussion

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

• Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
• Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
• Whether you can be redirected in the middle of an answer without restarting it from the beginning
• Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

• Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
• Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
• Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Where candidates lose points

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Treating a network call as though it were a local function call

A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Questions & practice

Describe how to detect a cycle in a linked list and return the start of that cycle.

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Walk one small example through your approach before writing the whole thing.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• How does this change if the input no longer fits in memory?

How do you implement a function to traverse a binary tree and check if a path sum equals a target value?

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. Choose the data structure from the access pattern, not from familiarity.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• How does this change if the input no longer fits in memory?

Explain the time complexity of your solution for finding the first non-repeating character in a string.

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Given an array, how would you sort it or find the second-highest element with optimal complexity?

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Choose the data structure from the access pattern, not from familiarity.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

1. Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
2. Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
3. Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
5. If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Worked solution (20 min)

1. Write the null policy as three lines of prose first, one per case, and keep them beside the output.
2. Emit 2n endpoint tuples (timestamp, delta, run_id) and sort on the key (timestamp, delta) so -1 precedes +1.
3. Sweep, tracking cur, peak, peak_at updated only on a strict increase, and the first run_id whose +1 takes cur to C+1.
4. Build a fixture with two runs where one ends exactly when the next starts, three genuinely overlapping runs, one run with a null finished_at and status running, and one with status lost and a 300-second wall_clock_limit_seconds.
5. Re-run with every timestamp shifted by a constant and confirm the peak is unchanged while peak_at shifts by the same constant.

Expected result: Peak is 3, from the overlapping trio. The exact-handoff pair yields a peak of 1, not 2. peak_at is the start instant of the third overlapping run. The lost run occupies exactly [started_at, started_at + 300s) under the stated policy.

Verify it yourself

• Setting peak_at on >= rather than > moves the reported instant to the last time the peak was attained. Flip it and watch the fixture change.
• Sorting by timestamp alone, without the delta tie-break, raises the reported peak on the handoff fixture.
• The deltas over the whole sweep sum to zero once every open-ended run has been clipped.

Follow-up

• Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
• The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
• How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Follow-up

• A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
• The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
• Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

Model credential revocation so history survives the delete

Easy · soft delete · partial indexes · check constraints · audit history

tenant_api_key stores key_id, tenant_id, workspace_id, name, key_prefix, secret_hash, scopes text[], status (active, revoked, expired, compromised), auth_version, created_at, expires_at, last_used_at, revoked_at, revoked_reason. Rotation inserts a new row and revocation never deletes, because an incident review asks which credential served a request last quarter. Write the constraints that enforce: a label is unique only among a tenant's live keys, revoked_at and status can never disagree, and scopes is never empty. Then write the authentication lookup predicate, and name one column in this table that must stay out of it.

Approach

1. Reach for a partial unique index rather than a plain UNIQUE: create unique index on tenant_api_key (tenant_id, name) where revoked_at is null. Any number of revoked rows may share a label, the live namespace stays unique per tenant, and the revoked majority is not in the index at all, so it stays small on a table that only grows.
2. Tie the nullable timestamp to the enum so the two cannot drift: check ((revoked_at is not null) = (status in ('revoked','compromised'))) and check ((revoked_at is null) = (revoked_reason is null)). A revocation that records no reason is the one an incident review cannot use.
3. Write the emptiness check as check (cardinality(scopes) > 0), not array_length(scopes, 1) > 0. array_length returns NULL for an empty array, a CHECK constraint passes when its expression is NULL, so the array_length version accepts exactly the value it was written to reject.
4. Make the lookup a single index probe with every liveness condition inside it: where secret_hash = $1 and revoked_at is null and (expires_at is null or expires_at > now()) and auth_version = $2, backed by a unique index on secret_hash. Nothing is filtered in application code, so there is no path that forgets a clause.
5. Keep last_used_at out of that predicate. It is written asynchronously and is allowed to lag by a minute, so it is a usage signal; feeding it into an authorisation decision makes the decision depend on a write that may be late, batched away or lost.
6. Flag the modelling smell while you are here: expired is derivable from expires_at < now(), so storing it as a status obliges a job to keep it true and guarantees the column is wrong between the expiry instant and that job's next run. Derive it in the predicate; keep the stored status for states that are decisions rather than clock readings.

Worked solution (20 min)

1. Create the table with all three constraints on a scratch database and insert two revoked rows sharing (tenant_id, name); the partial index should accept both.
2. Insert a second live row with that same name and confirm the violation names the partial index.
3. Run update tenant_api_key set revoked_at = now() leaving status = 'active' and confirm the CHECK rejects it; then try insert ... scopes = '{}' against both the cardinality and the array_length forms and note that only one rejects it.
4. Run explain (analyze, buffers) on the lookup predicate for a live key and confirm an index scan on secret_hash with rows removed by filter equal to zero.

Expected result: A partial unique index on (tenant_id, name) where revoked_at is null; check ((revoked_at is not null) = (status in ('revoked','compromised'))); check ((revoked_at is null) = (revoked_reason is null)); check (cardinality(scopes) > 0); a single-probe lookup on secret_hash plus revoked_at, expires_at and auth_version; last_used_at excluded, with a note that 'expired' should be derived rather than stored.

Verify it yourself

• Two revoked keys with the same name insert cleanly; a second live one does not.
• The array_length form of the emptiness check accepts an empty array and the cardinality form rejects it.
• No statement anywhere in the design deletes or updates a revoked row.
• Revoking a key touches exactly one row and takes effect on the next lookup with no application-side filtering.

Follow-up

• Rotation issues a replacement while the old key stays live for a 30-day overlap. What does the uniqueness rule become, and what does the UI show to tell two same-named keys apart?
• A password reset bumps the principal's auth_version. No row in this table changed. How does the next request fail, and what query counts how many keys that bump just killed?
• A key turns up in a public repository. Which columns let you find it, and what do you write to the row?

What are the four pillars of OOP, and how have you applied them in your projects?

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. State the consistency you need, and where you are willing to be stale.
3. Choose a partition key and say what query it makes expensive.

Follow-up

• What would you drop to keep the system up under load?
• How does this behave when that dependency is down for an hour?

Describe the difference between a process and a thread, and how you manage thread safety.

Medium · system design and architecture

Approach

1. Choose a partition key and say what query it makes expensive.
2. Name the read and write paths separately; they rarely have the same bottleneck.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

How do React hooks like useState and useEffect work under the hood?

Medium

Approach

1. State your assumptions explicitly before working the problem.
2. Work from the requirement backwards to the design.
3. Clarify what is being asked and what a complete answer contains.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Sealing a billing period against late-arriving usage

Medium · idempotency · immutable ledger · rounding · conditional writes

billing seals a tenant's period once the metering watermark passes the period end, prices the sealed rollups against the plan (included allowance, tier boundaries, negotiated discount) and writes invoice_line_item rows: quantity numeric(28,6), unit_price_micros bigint, amount_minor bigint, currency char(3), source_rollup_watermark timestamptz. The job is re-run after failures and two workers may attempt the same tenant. Events legitimately arrive with occurred_at inside the period and ingested_at after it. Specify the seal transition, the idempotency key for every write including the payment-processor call, the rounding position and mode, and the fate of a post-seal event.

Approach

1. Make sealing a conditional write rather than a read followed by a write: UPDATE usage_rollup_hourly SET status = 'sealed', sealed_at = now() WHERE tenant_id = $1 AND hour_start >= $2 AND hour_start < $3 AND status = 'open' RETURNING rollup_id. Two concurrent sealers cannot both win because the loser's predicate no longer matches, and the loser learns it lost from an empty result rather than from a lock timeout.
2. Gate the seal on the watermark, not on the clock - the period closes when the fold is trustworthy past period_end - but cap the wait, because an unsealed period blocks the whole billing run. Seal anyway after a stated maximum (six hours past period end is a reasonable default) and let whatever arrives afterwards become an adjustment. The trade-off is named in one line: adjustments are cheap, a missed billing cycle is not.
3. Make every write converge under re-run. Line items are idempotent on (invoice_id, sku, rate_tier) with ON CONFLICT DO UPDATE permitted only while the invoice is draft; the payment-processor call carries (tenant_id, billing_period_start) as its idempotency key, because a timeout there is an unknown outcome, not a failed one, and a plain retry is how a customer gets charged twice.
4. Keep the arithmetic exact and round exactly once. quantity stays numeric, the rate is an integer unit_price_micros in millionths of a minor unit because per-request prices are genuinely below a cent, and amount_minor = round_half_even(quantity x unit_price_micros / 1,000,000) is computed once per line and stored. Tiering walks the boundaries in order emitting one line per (sku, rate_tier) with tier 0 as the included allowance at price zero; the invoice total is the sum of stored amount_minor values and is never recomputed downstream from quantity and rate.
5. Handle the post-seal event as a forward-only correction. The sealed rollup is frozen, so the difference between the sealed value and the restated value becomes a new invoice_line_item with kind = 'adjustment' and voided_by_line_id pointing at the line it reverses, carrying its own source_rollup_watermark. Nothing is edited in place, because the original line is the only evidence of what the customer was actually charged and is precisely what a dispute, a refund or an audit asks to see.

Worked solution (35 min)

1. Write the seal as a single conditional UPDATE ... WHERE status = 'open' RETURNING and state exactly what the losing worker sees.
2. Price one line by hand: quantity 4,318,904.250000 at unit_price_micros 1200 gives 5,182.6851 minor units and 5,183 after a single half-even rounding. Then do quantity 2,500,000.000000 at unit_price_micros 1, which lands exactly on 2.5 and rounds to 2 under half-even but 3 under half-up.
3. Take four hundred synthetic lines with fractional minor units and compute the total two ways - sum of per-line rounded amounts, and a single rounding of the summed exact amounts - then record the gap.
4. Trace one event with occurred_at inside the period and ingested_at two days after the seal all the way to the row that eventually reflects it.

Expected result: Seal as a conditional UPDATE from status = 'open' gated on the metering watermark with a bounded maximum wait; line writes idempotent on (invoice_id, sku, rate_tier) and the processor call idempotent on (tenant_id, billing_period_start); amount_minor rounded once per line with half-even; late data becoming an adjustment line that points at the line it reverses and records its own watermark, with no in-place edit anywhere.

Verify it yourself

• Re-running the entire billing job on an already-invoiced period must produce identical line rows and must not call the payment processor a second time. If it charges again, the idempotency key was on the retry rather than on the business fact.
• Over four hundred lines with uniformly distributed fractional parts the two totals should differ by a handful of minor units - standard deviation around sqrt(400/12), about 5.8, with a worst case of 200. If your two totals match exactly, the test data has no fractional parts and has not exercised the trap.
• Every line carries a source_rollup_watermark, and two lines for the same SKU with different watermarks should explain their difference without opening any other table.

Follow-up

• Two workers start the same tenant's seal a millisecond apart and the winner crashes after sealing but before writing any line. What does the second worker observe, and is the resulting invoice correct?
• Show the divergence between rounding each line and rounding the total once over four hundred lines, and say which direction it goes.
• A customer disputes a charge from two quarters ago. Which rows answer it, and which single design decision made that answer possible?

Webhook workers leak until OOM and drop in-flight deliveries

Medium · memory leak · eviction · leases

webhook-delivery workers grow from 400 MB to a 2 GB limit over about 36 hours, are OOM-killed, restart, and repeat. Each restart abandons in-flight attempts, so webhook_delivery rows sit in in_flight until their leases expire and the backlog spikes. The live set measured after a forced full collection also grows. The fleet serves tens of thousands of subscriptions, several thousand of which have been failing for weeks. Give an ordered checklist, the measurement separating retention from fragmentation, and the fix.

Approach

1. Separate the two failure shapes with one measurement: track resident set size against the live set after a forced full collection. A live set that climbs monotonically is retention; a flat live set under a rising RSS is fragmentation, off-heap or native allocation, or an allocator that never returns pages. The stated symptom puts this in the first category, which rules out allocator tuning as a fix.
2. Characterise the curve rather than the total. Growth linear in uptime implies an unbounded structure keyed by something that keeps arriving; step growth implies buffering a large object. Correlate the slope against event rate and separately against the count of distinct subscriptions seen, because those two diverge and only one of them will fit.
3. Diff two heap snapshots an hour apart by retained size grouped by dominant root, not by allocation count, which is dominated by short-lived objects and will point at the wrong thing.
4. Expect a per-subscription map with no eviction: circuit-breaker or backoff state created on first failure and never removed, so the retained set grows with endpoints that have ever failed, and the several thousand permanently dead endpoints hold theirs forever.
5. Fix in two places. Bound the in-memory structure with a size-capped LRU or a TTL keyed on last use, and move state that must survive a restart onto the subscription or webhook_delivery row, since the worker holding it in memory is exactly why a restart loses it.
6. Repair the second-order damage separately, because it will outlive the leak: workers claim by compare-and-set with leased_until, so a bounded lease returns in_flight rows to pending on a known schedule, and a graceful shutdown releases leases instead of waiting them out.

Follow-up

• The backlog spike after a restart is itself a thundering herd against customer endpoints. What stops the recovery from becoming a second incident?
• Suppose the live set had been flat while RSS still climbed. Name two causes and the measurement that separates them.
• How would you size the LRU, and what does a miss on an evicted circuit-breaker entry cost a customer whose endpoint is down?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

• Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
• Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
• Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

• Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
• Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
• Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

• Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
• Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
• Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

• For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
• Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
• Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
• Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

• Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
• Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
• Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
• Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

• Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
• Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
• Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

• Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
• Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
• Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

How do you handle concurrent transactions in a database to avoid deadlocks?

Medium · behavioural and engineering judgement

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Pick a story where you made the decision, not one where you watched it.
3. Name the disagreement and how you resolved it with evidence.

Follow-up

• How did you know your change caused the improvement?
• What would you do differently if you ran that again?

Unblock an engineer on a job run that finished twice

Easy · mentoring · fencing tokens · lease expiry · debugging method

An engineer two weeks into the team brings you a job_run row showing status succeeded with an exit_code written by a worker declared dead ten minutes earlier; the retry attempt also shows succeeded. They have spent a day adding logging and are no closer. You have twenty minutes and you do not want to take the keyboard. Describe how you unblock someone: the question you ask first, what you let them find themselves, the concept you name and when, and how you check the next day that they own the fix rather than having watched you produce it.

Approach

1. Ask what they expect rather than what they see: which statement set status to succeeded, and what did it check before writing? That question points directly at the update's WHERE clause, which is where the answer lives, and it costs them nothing to answer, so it does not read as a test.
2. Let them build the timeline themselves from the row: queued_at, started_at, leased_until, finished_at and worker_id, on both the original run and the retry. Two different worker_ids with a lease expiry between them tells the whole story, and they will see it before you say it.
3. Name the concept once the evidence has earned it. A lease bounds time; it does not prevent a write. The store has to reject a stale writer, which means the update carries a fencing token the row compares — update job_run set status = 'succeeded' where run_id = $1 and lease_token = $2 and status = 'running' — and a long garbage-collection pause or a brief partition is enough to produce what they are looking at.
4. Point at the second, less obvious half and let them decide it: 'lost' exists in the status enum precisely so a run whose worker vanished is not recorded as failed, because failed asserts an outcome nobody observed and the system then bills and retries on that assertion. Ask them what these two rows should have said.
5. Leave them with the next step rather than the patch — a test that kills the first worker after the sandbox exits and before the row is written — and say when you are available again, so the offer is real rather than polite.
6. Check ownership the next day by what they produced, not by asking if it went well: a test that reproduces the window proves they understood it; a test that only asserts the new WHERE clause proves they copied it. Ask them to explain it to a third person and listen for whether the explanation is theirs.

Follow-up

• They propose a longer lease instead of a token. What do you say, and what breaks when legitimate runs last thirty minutes?
• How can you tell whether your explanation landed or they simply deferred to you?
• The same engineer hits a variant of this next month. What did you fail to teach the first time?

Own the incident where invoices undercounted metered usage

Medium · incident response · at-least-once · billing correction · postmortem

A metering consumer acknowledged each batch before committing the fold into usage_rollup_hourly. A rolling deploy restarted consumers mid-batch for two hours; roughly 1.4M usage_event rows were acknowledged and never folded, and 61 invoices sealed against the resulting rollups before anyone noticed. Take the owner's role. Describe an incident of comparable blast radius you owned: how it surfaced, the query that sized the loss, what you stopped first, and how the money was corrected. Give a wall-clock timeline and one thing you got wrong while it was still live.

Approach

1. Open with the invariant that broke and the direction of the error, because they determine everything else: acknowledging before committing makes the consumer at-most-once, so this loses events rather than duplicating them, and loss raises no error anywhere. A listener who hears 'we lost revenue silently' knows immediately why detection took two hours.
2. Size it with a stated reconciliation rather than an adjective: sum(quantity) from usage_event grouped by (tenant_id, sku, hour of occurred_at) over the window, against usage_rollup_hourly.quantity_sum on the same keys, filtered to environment='production' because staging and sandbox are metered but not billed. Then bisect by hour and tenant until single cells explain the gap. Say how long that ran and whether a replica could serve it while the incident was live.
3. Separate mitigation from fix and say which came first. Mitigation is holding the sealing job, because a sealed row is frozen by design and every minute of sealing converts a recoverable rollup into an invoice correction. The fix is moving the acknowledgement after the commit, which re-introduces duplicates that the dedup check on (tenant_id, idempotency_key) must now absorb.
4. State the correction path in the domain's own terms: sealed periods are never edited, so each affected tenant gets an adjustment line on the next invoice with kind='adjustment' and voided_by_line_id pointing at the line it reverses, priced against the same rate tier and carrying the watermark it priced against. That is four separate numbers — tenants affected, minor units, the cycle the adjustment lands in, and when customers were told.
5. Close on one prevention control with its cost, not five: a per-hour reconciliation comparing raw sum to rollup sum that pages above a threshold. Name the threshold and the false-page rate you accepted, because a detector nobody will keep staffed is not prevention.
6. Name a mistake you made inside the response window — the wrong first hypothesis, a mitigation that made it worse — rather than a design mistake from six months earlier. That is the part candidates rehearse away and interviewers weight heavily.

Follow-up

• Your fix moves the acknowledgement after the commit. What breaks now, and what absorbs it?
• One undercharged tenant has since churned. Do you bill them, and who decides?
• How would you have caught this in ten minutes instead of two hours, and what would that detector cost you in pages per week?