cswork / libraryOpen practice workspace

Interview guides · Centre for Development Communication

Centre for Development Communication Software Engineer Interview Questions & Guide 2026

Caching practice for Centre for Development Communication software engineer prep: settle the staleness budget before choosing an invalidation strategy.

The 60-second brief

The Software Engineer role at the Centre for Development Communication is a vital position focused on bridging the gap between complex technical solutions and effective communication strategies. You will be responsible for developing and maintaining the digital infrastructure that enables the organization to disseminate critical information efficiently. Your work directly impacts how the Centre for Development Communication interacts with its audience, requiring a blend of technical proficiency and a deep understanding of user-centric design.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

  • Detect concurrent edits instead of losing writes
  • Trace a symptom to a mechanism under load
  • Bound every outbound call with a timeout

Overview

This role is particularly interesting because it sits at the intersection of traditional software development and communication-driven goals. You will likely work on projects ranging from front-end interface development to specialized email development, ensuring that every piece of software built serves the core mission of the organization. Success in this role requires not just coding ability, but the ability to translate organizational needs into scalable, reliable, and accessible technical solutions.

The Centre for Development Communication interview process

Centre for Development Communication candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Interaction

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Discussion Preparation

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

  • Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
  • Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
  • Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
  • Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

  • Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
  • Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
  • Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Trusting input because it came from your own front end

Anything crossing a trust boundary is hostile: parameterise queries instead of building SQL by concatenation, validate against an allow-list rather than a deny-list, and bound the size of anything you allocate from a request. Raising this unprompted in an API or design question is a cheap and unusually strong signal.

Questions & practice

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

  1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
  2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
  3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
  4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
  5. Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
  6. Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Worked solution (30 min)

  1. Write the edge-loading query with the tenant predicate on both endpoints and state what it does with a cross-tenant edge.
  2. Implement iterative Tarjan with an explicit stack and confirm on a three-node cycle that it emits one component of size three.
  3. Build the transpose restricted to the visited set and mark every node with an in-edge from outside it as refused, carrying the referrer id.
  4. Run Kahn over the condensation and verify the emitted order against the referrer-before-referenced rule.
  5. Size the CSR arrays for 2,000,000 nodes and 8,000,000 edges and compare against a boxed adjacency map.

Expected result: An iterative O(V+E) traversal over a tenant-scoped CSR subgraph, SCC condensation so cycles archive atomically as one component, a transpose-based refusal list naming the external referrer for each blocked resource, and a Kahn topological order over the condensation, with recursion replaced by an explicit stack because of graph depth rather than style.

Verify it yourself

  • A two-node cycle with no external referrers archives as one unit, and adding a single external referrer moves both nodes to the refusal list.
  • No resource appears in the archive order before something that references it, except within its own component.
  • A counter on edge visits matches the loaded edge count exactly, confirming each edge is traversed once.

Follow-up

  • The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
  • The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
  • An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

  1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
  2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
  3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
  4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
  5. Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
  6. Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).

Follow-up

  • The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
  • How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
  • Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

  1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
  2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
  3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
  4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
  5. Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
  6. Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Follow-up

  • The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
  • Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
  • How would you run this continuously at low cost instead of only as incident response?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

  1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
  2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
  3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
  4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
  5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
  6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Worked solution (25 min)

  1. Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
  2. Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
  3. Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
  4. Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
  5. Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.

Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.

Verify it yourself

  • COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
  • Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
  • Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.

Follow-up

  • A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
  • Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
  • Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

  1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
  2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
  3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
  4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
  5. State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
  6. Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Follow-up

  • The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
  • Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
  • What does the cursor do when the row it points at has since been deleted?

Shard by tenant when one tenant outgrows a single shard

Hard · sharding · hot partition · routing · online migration

One primary holds resource, resource_revision, outbox_event and idempotency_key for every tenant and is at its write ceiling at 1.2k writes/second. tenant_id leads every index. Shard across eight primaries. One tenant holds 22% of all rows and by itself exceeds a single shard's write capacity. Design the routing, the split of that tenant, and the online move of a tenant between shards with writes continuing. State what breaks for queries that are tenant-scoped today, and exactly what a write must do when it arrives at the old shard after the cutover.

Approach

  1. Route on a unit smaller than a tenant from the start. Make the routing key (tenant_id, bucket) with a fixed bucket count - 64 over eight shards - and keep a directory mapping each (tenant_id, bucket) to a shard, carrying a version and cached in every service. An ordinary tenant has all 64 buckets pointing at one shard and behaves exactly as it does today; only the hot tenant has its buckets spread. Hashing tenant_id alone spreads tenants evenly, gives you no way to move one, and has no answer at all for a tenant larger than a node. The bucket count is the part you cannot change later without rehashing rows, so pick it well above the shard count and rebalance by moving buckets, not by re-bucketing.
  2. For the tenant that exceeds one node, its buckets must land on different shards - that is the whole point of bucketing it, and buckets confined to its own shard would rename the rows while leaving every write on the node whose ceiling it already exceeds. Size it from measured numbers rather than from its row share: 22% of rows says nothing about write rate. The current primary tops out near 1.2k writes/second on this hardware and workload, so a tenant peaking at W writes/second needs its buckets spread over at least ceil(W / headroom-per-shard) shards, where the divisor is the share of each shard's ceiling you are willing to give it while that shard still serves other tenants - not the full 1.2k. Size on its peak, not its mean.
  3. Fix the co-location invariant at the right grain. What must commit in one transaction is a resource, its resource_revision row and its outbox_event row, so the bucket is a property of the resource: derive it once at creation and stamp it into resource_id, and every later revision and event routes with its parent for free. Per-tenant co-location was never the requirement, and mistaking it for one is what makes a tenant look unsplittable. What genuinely breaks is an invariant spanning two resources of one tenant - a per-tenant counter, uniqueness across its resources - which now needs either a home-shard table or two-phase commit, and 2PC at this write rate is not a serious option.
  4. Keep the idempotency constraint arbitrating, because it is now enforced per shard. PRIMARY KEY (tenant_id, idempotency_key) only continues to reject a retry if the same key always lands on the same shard, so derive the create-path bucket from hash(tenant_id, idempotency_key) and mint the new resource_id inside that bucket, which also puts the key row and the resource it guards in one transaction. A bucket chosen from anything that differs between a request and its retry - a timestamp, the worker id, a client-supplied resource id - splits one key across two shards, both inserts succeed, and the write endpoint's retry safety is silently gone.
  5. State what the split costs the hot tenant's reads. Its listing, one 21-entry index scan today, becomes a scatter-gather: every bucket-shard returns 21 rows, a coordinator merges and discards the surplus, latency becomes the slowest shard's rather than the median's, and the keyset cursor has to carry a position per bucket instead of one (updated_at, resource_id) pair. Counts over that tenant fan out the same way. The relay becomes one leader per shard; consumers are unaffected because their ordering guarantee was always per aggregate and a resource's events never leave its bucket.
  6. Move one bucket at a time, reversibly, and fence the straggler at the shard rather than at the caller. Copy from a snapshot while the bucket stays read-write, tail changes until the remaining delta is a few seconds of writes, fence writes for that (tenant_id, bucket) alone with a retryable status, apply the final delta, bump the routing version. Scoping the fence to a bucket is what makes a seconds-long freeze affordable. Then have each shard store the routing epoch it believes it holds for each (tenant_id, bucket) and reject any write carrying an older one: without that token, a service on a stale map commits successfully to a database nothing will ever read again, and the loss stays invisible for days. Outside the data path, anything that aggregated across tenants in one query - admin reporting, the A-Z index, global counters - becomes a fan-out across eight shards with a merge, and per-tenant uniqueness survives only on tables that stay whole on the tenant's home shard.

Follow-up

  • The fence lasts 90 seconds instead of 4 because the final delta keeps growing. What is happening, and what do you do while the tenant is fenced?
  • Two tenants must merge into one account. What does that cost under this scheme, and which step is not reversible?
  • A shard is lost entirely. Which tenants are affected, and what is the source of truth for rebuilding them?

Design the bulk write endpoint a migration script retries blindly

Medium · batch api · partial failure · idempotency

A customer's migration script pushes 2 million resources through POST /v1/resources:batch, up to 500 items per call, and retries any call that errors or times out. Within a call some items fail validation, some collide with rows that already exist, and some succeed. Specify the request and response shape, whether a batch is atomic or per-item, how idempotency works for the call and for each item, the status code for a mixed outcome, the size and item-count limits with their error codes, and exactly what the script does after a timeout mid-batch.

Approach

  1. Choose atomicity deliberately and price it. All-or-nothing means one transaction holding locks for the whole batch on a primary already absorbing about 1.2k writes/second, which bounds batch size by lock duration, and it turns one bad row into 499 rejections the script must resend. Per-item partial success is the right default for a migration, and the contract's job is then to make a partial outcome impossible to miss.
  2. Require a client-supplied id on every item and echo it in every result. Deriving a per-item key from the array index breaks the first time the script resends a batch with the failures removed: the indices shift, previously-succeeded items acquire new keys, and they are created a second time.
  3. Key the effects at two levels. The call's Idempotency-Key covers an exact resend of the same bytes; per-item keys of (tenant_id, client_item_id) make a partially-applied batch safe to resend whole. Resending an identical batch must reproduce the same per-item results, not 500 conflicts the script has to interpret.
  4. Answer a mixed outcome with one status plus per-item detail: 200, or 207 borrowed from WebDAV if you prefer it - document whichever you pick - carrying an array of client_item_id, per-item status, and either resource_id or an error code from the same taxonomy the single-item endpoint uses. Reserve 4xx for the request as a whole: unparseable body, too many items, payload over the limit (413). Put a failed count at the top level so that even a script checking only the cheapest thing cannot conclude success while rows were dropped.
  5. Cap the request before doing any of it: item count, total bytes, and concurrent batches per tenant, since one tenant's migration otherwise consumes write capacity everyone shares. Anything that cannot finish inside the request deadline belongs on job_run behind a 202, not in a synchronous call that will time out halfway.
  6. Script behaviour after a timeout: the outcome is unknown and no results were received, so resend the identical batch with the same keys and read the results. Never resend 'only the items I have no result for' - a timeout yields no results at all, and that rule silently means resend everything anyway.

Worked solution (30 min)

  1. Write the request schema with the per-item client id, and the response schema with per-item status and a top-level failed count.
  2. Write the atomicity decision and the sentence of justification that names the lock cost or the resend cost.
  3. Define both key levels and trace a resend of a half-applied batch through them, item by item.
  4. List the whole-request rejections with their status codes and limits.
  5. Write the script's timeout rule and the pacing it should apply between calls.

Expected result: A per-item batch with client-supplied item ids, per-item idempotency plus a call-level key, a mixed-outcome response that a status-only check cannot misread, explicit request caps, and a resend rule that is correct when no results were received.

Verify it yourself

  • Resending an identical batch twice leaves exactly the same set of resource rows as sending it once.
  • Resending a batch with the successful items removed does not create duplicates of anything.
  • A caller that reads only the top-level fields can still tell that some items failed.

Follow-up

  • At 500 items per call, what is the wall-clock time for 2 million rows, and what pacing do you publish so the migration does not become an incident?
  • One item in every batch fails with the same code. How does the script discover that without a human reading logs?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles · tail latency

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

  1. Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
  2. Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
  3. Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
  4. Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.
  5. Price the fix. This is a fourth index on a table taking about 1.2k writes/second, and every insert and version bump maintains it. Justify it against the query it serves, and check whether it makes an existing index redundant, which here it does not, since the original still serves the status-filtered default listing.
  6. Re-measure per tenant-size bucket rather than in aggregate. A fleet-wide p99 can improve while the largest tenant is still on the old plan.

Follow-up

  • The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
  • How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?
  • If a fourth index were unacceptable on write grounds, what else could serve this query?

Suggested practice checklist

Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.

Day 1 — Fix the scope and take a cold baseline

  • Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
  • Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
  • Set the week's rule: one thing finished every night, including the night you only have forty minutes.

Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One pattern, written three times from blank

  • Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
  • On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
  • Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.

Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.

Day 3 — One design, only to the depth you can defend

  • Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
  • Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
  • Write the one tradeoff you are choosing against and the observation that would make you reverse it.

Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.

Day 4 — Only the fundamentals you will have to defend

  • Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
  • Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
  • Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.

Deliverable: Three written answers, each under two hundred words and each timed aloud.

Day 5 — Your own work, timed

  • Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
  • Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
  • Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in the weekend block

  • Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
  • Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
  • Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

  • Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
  • Re-read only your own notes from this week and open no new material.
  • Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

What motivates you to apply your technical skills to the mission of the Centre for Development Communication?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Name the disagreement and how you resolved it with evidence.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Why do you want to transition into this role from your previous position?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

How do you handle feedback on your code or design work from non-technical stakeholders?

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. Name the disagreement and how you resolved it with evidence.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Can you provide an example of a time you had to adapt your technical approach to meet a communication goal?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Pick a story where you made the decision, not one where you watched it.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?