cswork / libraryOpen practice workspace

Interview guides · Aspora

Aspora Software Engineer Interview Guide

Aspora Software Engineer interview prep: ledger design, optimistic locking vs event sourcing, machine coding a ledger, and resume deep-dive answers.

The 60-second brief

Candidates describe the Software Engineer work at Aspora as backend infrastructure: ledger implementations, system architecture, and services that have to keep data correct when many writes compete for the same records. The requirements candidates describe ask for at least one backend language, database design, and experience with distributed or transactional systems; event sourcing, CQRS and high-concurrency work are listed as a plus. In practice that means the job sits close to money-like data, where a lost update or a double-applied transaction is a correctness bug, not a performance issue, and the reported interview questions follow that thread from design through to code.

This guide is for backend-leaning Software Engineer candidates in the Aspora loop. It covers the four reported rounds (Initial Screening, Resume Deep-Dive, System Design Assessment, Machine Coding Round), gives question-specific answers for all eleven reported questions, from walking through your own project architecture to implementing and testing a ledger, and lays out a seven-day plan that ends with a full ledger build and a mock design session. It assumes you can already write production code in one backend language and focuses on consistency, concurrency control, data integrity and clean, testable implementation.

Candidates report four rounds over roughly 3-5 weeks: Initial Screening, Resume Deep-Dive, System Design Assessment and a Machine Coding Round. The reported technical topics concentrate on ledger systems, consistency and concurrency control.

Focus areas

  • Ledger systems
  • System Design
  • HLD (High-Level Design)

Overview

The reported Aspora questions form an unusually tight cluster. Ledger systems appear in the design questions (a scalable ledger with strong consistency, high-contention writes, data integrity across services) and again in the coding questions (implement a functional ledger, make it extensible, test its edge cases). Two example scenarios candidates mention point the same way: handling concurrent financial transactions without losing data, and moving a monolithic data structure to a distributed model. If you prepare one domain deeply, double-entry bookkeeping with correct concurrency control, you are preparing for most of the technical loop at once.

That domain has a small set of ideas you need to be able to use, not just name: double-entry postings that sum to zero, append-only entries with compensating reversals, idempotency keys, integer minor units for amounts, row locking versus version checks versus conditional updates, lock ordering to avoid deadlocks, the transactional outbox, and the difference between a strongly consistent balance check and an eventually consistent read model. Event sourcing and CQRS are listed as a plus in the requirements candidates describe, and one reported question asks you to compare optimistic locking with event sourcing directly, so know where each one fits and what it costs.

The resume questions are the other half. Candidates report being asked to walk through the architecture of their most recent project, name their biggest trade-offs, describe how they would rebuild for ten times the scale, and explain the internals of a framework or language on their CV. Candidates also report that interviewers push on the limits of what is listed and challenge design assumptions. Preparation here is concrete: a diagram you can redraw from memory, two trade-offs with the rejected option named, and runtime internals for every technology you list. Anything on your CV you cannot explain one level below the API is a liability, so cut it or study it.

The Aspora interview process

The four stages below are what candidates report about the Aspora Software Engineer process; the descriptions say what each stage is reported to cover and how to prepare for it.

Initial Screening

Candidates report an initial screen on basic qualifications and fit. Nothing more specific is reported about its format, so treat it as the place to establish that your background matches what candidates say the role asks for: a backend language, database design, and distributed or transactional systems work. Lead your introduction with the project closest to that profile (anything involving payments, balances, inventory counts, bookings or other contended writes) and use the end of the call to ask how the later rounds run, in particular which languages are allowed in the machine coding round and whether you code in your own environment.

What to demonstrate

  • Basic qualifications against the reported requirements: a backend language, database design, distributed or transactional systems
  • Fit for the role and how clearly you describe your own work

How to prepare

  • Write a short introduction that opens with your most transactional or concurrency-heavy project and names your main backend language and database.
  • Map each reported must-have (backend language, database design, distributed or transactional systems) to one concrete line of evidence from your CV.
  • List the questions you will ask about the remaining rounds: machine coding language and setup, and whether the design round centres on one problem.

Resume Deep-Dive

Candidates report a thorough examination of the resume, and the questions they file under resume and experience are: walk through the architecture of your most recent project; your most significant technical trade-offs and why; how you would approach your last project if it had to scale tenfold; and the internal workings of a framework or language on your CV. Candidates also report that the deep-dive probes how far your knowledge on listed technologies goes and that being able to name the limitations of your past projects matters. Approach it as a technical defence of your own work: every box on your diagram should have a reason, every trade-off should have a rejected alternative, and every listed technology should be explainable below the API surface.

What to demonstrate

  • Depth of understanding of the systems you say you built, including their failure modes
  • Trade-off reasoning with named alternatives and the constraint that decided
  • Internals of the frameworks and languages listed on your CV
  • Honest assessment of what you would change in past work

How to prepare

  • Redraw your most recent project's architecture from memory, then trace one write request and one read request through it end to end.
  • Write two trade-offs in the form: options considered, constraint that decided, cost paid, what happened afterwards.
  • For a 10x version of that project, name the first three components to break and the change you would make for each.
  • For every language and framework on your CV, write one paragraph on its runtime model (concurrency, memory, transactions) and delete any item you cannot.

System Design Assessment

Candidates report this stage assesses how you design systems and articulate your decisions, with topics centred on consistency models (eventual versus strong), concurrency control (locking versus non-blocking approaches) and data integrity in transactional systems such as ledgers. The reported design questions cover a scalable ledger with strong consistency, optimistic locking versus event sourcing under high concurrency, data integrity in distributed systems, and high-contention writes; candidates also mention scenarios on concurrent financial transactions without data loss and migrating a monolithic data structure to a distributed model. Candidates report that design choices get challenged, so state the invariant first (balances never go negative, postings always sum to zero, every request applies at most once), then show how each component protects it and what you give up.

What to demonstrate

  • Choice of consistency model per data path and the reasoning behind it
  • Concurrency control under contended writes: pessimistic, optimistic, conditional updates, serialisation per key
  • Data integrity mechanisms: idempotency, outbox, constraints, reconciliation
  • Explaining and defending design decisions against alternatives

How to prepare

  • Design a double-entry ledger end to end: schema, transfer transaction, idempotency, balance reads, partitioning, and cross-partition transfers.
  • Build a comparison table of row locks, version-column optimistic locking, conditional atomic updates and event-sourced streams: conflict behaviour, throughput under a hot key, audit trail, and read-side cost.
  • Walk through the concurrent financial transactions scenario: two simultaneous withdrawals on one account, and opposite-direction transfers that could deadlock.
  • Outline a live migration from one database to a partitioned model: dual writes via outbox, backfill, verification, cut-over and rollback.

Machine Coding Round

Candidates describe this as a significant coding round focused on practical engineering problems. The questions they file under machine coding are implementing a functional ledger that supports standard transaction operations, structuring that code so it can absorb future business requirements, and describing how you would test its edge cases. Treat it as building a small, runnable service rather than solving a puzzle: agree on the operations first, write the domain model and the core transfer path, get it running with a few calls, then add validation, idempotency, thread safety and tests in that order, saying what you are deferring as you go.

What to demonstrate

  • Working, runnable code for the reported problem type
  • Code structure: separated domain model, service and storage, with clear extension points
  • Correctness on money-handling edge cases and concurrency
  • How you test what you wrote

How to prepare

  • Build a ledger from an empty file in your interview language on a timer you set: accounts, deposit, withdraw, transfer, balance, history, idempotent replay.
  • Add one new transaction type (a fee or a reversal) to your finished build and note how many files you had to touch.
  • Write the edge-case test list for that build and implement the five most important tests, including a multi-threaded transfer test.

Where candidates lose points

Representing money as a float or double, or updating balances in place with no entries, in the ledger implementation.

Use integer minor units (or a decimal type) with a currency code, and make every operation write entries that net to zero before the balance changes. Say this out loud in the first minutes of the machine coding round, and add a test that checks balances equal the sum of entries.

A transfer that locks the two accounts in call order, or reads a balance and writes it back in separate steps.

Lock accounts in a fixed order (by id) or use a conditional atomic update that checks funds in the same statement. Practise explaining the deadlock that opposite-direction transfers create and the lost update that a read-then-write allows, since both reported design questions on concurrency lead there.

Treating optimistic locking and event sourcing as interchangeable, or choosing optimistic retries for the hottest account.

Explain that one is a concurrency check and the other a persistence model that usually relies on that same check, and name what each costs: retry storms under contention for version checks, lagging projections and event versioning for event sourcing. Then pick one for a stated scenario.

Describing a past project's architecture at box level and stalling when asked how a listed framework or database actually works.

For each technology on your CV, prepare the runtime and transaction internals you relied on and one production issue they caused. Remove technologies you cannot explain. Candidates report the resume deep-dive tests the limits of what you list.

Writing to the database and then publishing an event, with no idempotency on the transfer endpoint, in the data integrity design.

Default to a transactional outbox and idempotent consumers, put a unique idempotency key on every mutating request, and add reconciliation. Be ready to say what happens when any single step crashes halfway.

Questions & practice

What are the trade-offs between optimistic locking and event sourcing in a high-concurrency environment?

Medium · System Design & Architecture

Approach

  1. Point out that they sit at different layers. Optimistic locking is a concurrency control: read a version, write with UPDATE ... SET version = version + 1 WHERE id = ? AND version = ?, and treat zero affected rows as a conflict to retry or surface. Event sourcing is a persistence model: state is the fold of an append-only event stream, and it usually uses optimistic concurrency itself, appending only if the stream is still at the expected version.
  2. Optimistic locking: cheap with no lock held during think time and fine when conflicts are rare, but on a hot record conflicts and retries grow with concurrency and can turn into retry storms; it gives no history by itself.
  3. Event sourcing: a complete audit trail, replay and temporal queries come for free, which suits a ledger; the costs are read models built by projections that lag (often paired with CQRS), event schema versioning, snapshots for long streams, and harder ad hoc queries.
  4. Under high contention on a single aggregate both struggle the same way, because both reject concurrent writers to one version. Remedies are the same: shrink the aggregate, split the hot key, serialise writes per key through a queue or single writer, or use pessimistic locking with short transactions.
  5. Conclude with a choice for a stated scenario, for example a relational ledger with conditional updates for balances plus an append-only entries table, and event sourcing where full replay and audit justify the projection cost.

Follow-up

  • How do you rebuild a projection that has a bug, and what do readers see meanwhile?
  • Two commands append to the same stream at the same expected version. What happens to each?
  • How do you change the shape of an event that is already stored millions of times?

Implement a functional ledger system that supports standard transaction operations.

Medium · Machine Coding

Approach

  1. Agree the operations before coding: create account, deposit, withdraw, transfer, get balance, get history, and optionally reverse. Ask whether persistence, multi-currency and concurrency are in scope.
  2. Model: Account (id, currency, balance as a long in minor units), Transaction (id, idempotency key, type, status, timestamp) and Entry (transaction id, account id, signed amount). Never use floating point for money. Every operation writes entries that net to zero, using a system or external account for deposits and withdrawals.
  3. Put logic in a LedgerService behind a repository interface with an in-memory implementation. Validate: positive amount, account exists, same currency, sufficient funds, no transfer to self, and return the stored result on a repeated idempotency key.
  4. Thread safety: per-account locks acquired in ascending id order for transfers, or a single lock if you state that simplification. Operations are O(1) with hash maps; history is O(k) with a per-account entry list.
  5. Get a runnable main or test calling each operation early, then extend. Working code that covers the core path beats a large design that does not run.

Follow-up

  • Make this safe under concurrent transfers. What exactly do you lock?
  • Add multi-currency transfers with an exchange rate. Where does the rate live, and how do the entries balance?
  • How would you reverse a transfer, and what does history show afterwards?

How would you structure your code to ensure it is extensible for future business requirements?

Medium · Machine Coding

Approach

  1. Separate layers: domain objects with no framework code, a service that runs operations, repository interfaces for storage, and a thin API or CLI on top. Storage then swaps without touching business rules.
  2. Model each transaction type as a command or strategy implementing a common interface (validate, then produce entries), registered in a map, so a new type such as a fee, hold or refund is a new class rather than another branch in a growing switch.
  3. Keep validations as composable rules and rates or limits as injected configuration, and inject dependencies such as clock, id generator and repository through constructors so tests can control them.
  4. Name the limit: add an abstraction only where a second variant is plausible, and show one extension concretely, for example adding a fee type that touches one new class and one registration line.

Follow-up

  • Add a transaction fee charged on every transfer. Which files change?
  • Where would per-account limits live, and how would they be tested?
  • When does this level of abstraction cost more than it saves?

Describe how you would test the edge cases of the implementation you just wrote.

Medium · Machine Coding

Approach

  1. Unit edge cases per operation: zero and negative amounts, balance exactly equal to the withdrawal, one minor unit short, unknown account, currency mismatch, transfer to self, very large amounts near the integer limit, and a repeated idempotency key with the same payload (returns original) versus a different payload (rejected).
  2. Invariant tests: after any sequence of operations the sum of all entries is zero and each balance equals the sum of its entries. Property-based testing (Hypothesis, jqwik, fast-check) generates random operation sequences and checks these automatically.
  3. Concurrency tests: many threads transferring between two accounts in both directions, then assert the total is conserved, no balance went negative and the run finished without deadlock; repeat many times since races are intermittent.
  4. Failure tests: inject an exception between the debit and the credit and assert no partial entries or balance changes remain. Inject the clock and id generator so tests are deterministic.

Follow-up

  • How do you make a concurrency test fail reliably when the locking is wrong?
  • Which bug would property-based testing find here that example tests would miss?
  • What would you test differently once storage is a real database?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

  1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
  2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
  3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
  4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.

Follow-up

  • The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
  • Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

  1. Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
  2. For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
  3. Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
  4. Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.

Follow-up

  • 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
  • A dashboard runs this for 40 owners in one page load. What changes about the design?

Can you walk me through the architecture of your most recent project?

Medium · Resume & Experience Deep Dive

Approach

  1. Open with one sentence of context a listener needs: what the system does, who calls it, and the scale numbers you actually know (requests per second, rows, tenants). Do not invent numbers you cannot back up.
  2. Draw the components in the order a request meets them: client or upstream caller, entry point (gateway, load balancer), services, data stores, caches, queues and async workers, external dependencies. Name the concrete technologies.
  3. Trace one write path and one read path end to end, saying where the transaction boundary is, what is synchronous and what is asynchronous, and where data could be stale.
  4. Mark clearly which parts you personally designed or built versus inherited, then name one failure mode you handled (retries, idempotency, timeouts) and one limitation you would fix today.

Follow-up

  • Why that datastore for that data, and what would have pushed you to a different one?
  • What happens to an in-flight request if the queue or a downstream service is unavailable?
  • Which part of this did you build yourself, and what would you change in it now?

What were the most significant technical trade-offs you made in your previous role, and why?

Medium · Resume & Experience Deep Dive

Approach

  1. Pick two trade-offs that were real decisions with a losing option, for example synchronous call versus queue, relational versus document store, strong versus eventual consistency on one path, build versus buy, or a denormalised table for read speed.
  2. Structure each one as: the options on the table, the constraint that decided it (latency budget, team skill, deadline, consistency requirement), what you gave up, and the evidence afterwards that it was or was not the right call.
  3. Quantify where you can (latency, cost, incident count, delivery time) and say how you measured it.
  4. Close with what you would decide differently with today's information. A trade-off story with no cost paid reads as a feature list, which is the mistake that sinks this answer.

Follow-up

  • Would you make the same call today, and what would change your mind?
  • Who disagreed with the decision, and how was it resolved?
  • How did you know the cost you accepted stayed acceptable over time?

How would you have approached your last project differently if you had to scale it tenfold?

Medium · Resume & Experience Deep Dive

Approach

  1. First say which dimension grows tenfold, because the answer differs: request rate, data volume, number of tenants, or write rate on the same records.
  2. Find the first bottleneck with arithmetic: database connections and primary CPU, a single-writer primary, hot rows, N+1 queries, synchronous fan-out to slow dependencies, or a cache with a poor hit rate.
  3. Match fixes to the bottleneck: connection pooling and read replicas or caching for read load; partitioning by a key that keeps transactions local for write load; queues to absorb bursts; idempotency so retries at higher volume stay safe.
  4. Separate what you would have done from day one (idempotency keys, a partition-friendly key, stateless services) from what you would defer until measured (sharding, multi-region), and explain why premature sharding has a cost.

Follow-up

  • How would you move the existing data to the new layout without downtime?
  • Which query becomes expensive once you partition on that key?
  • What would you monitor to know you are approaching the next limit?

Can you explain the internal workings of the specific framework or language you listed on your resume?

Medium · Resume & Experience Deep Dive

Approach

  1. Answer for what is on your CV, at the level below the API. Java: heap versus stack, generational and G1 garbage collection, JIT, the thread model and java.util.concurrent, and in Spring, dependency injection through proxies and why a self-invoked @Transactional method does not start a transaction.
  2. Python: the GIL and what it means for CPU-bound versus IO-bound work, reference counting plus the cycle collector, and how the asyncio event loop schedules coroutines. Go: goroutines on the M:N scheduler, channels, and the concurrent garbage collector. Node: the single-threaded event loop and the libuv thread pool.
  3. For your database: MVCC, isolation levels (read committed, repeatable read, serializable), what SELECT FOR UPDATE locks, and how an index is used or skipped.
  4. Tie each internal to a production consequence you have seen: a GC pause in latency graphs, a blocked event loop, a lost update under read committed, a connection pool exhausted by long transactions.

Follow-up

  • What happens if a @Transactional method is called from another method in the same class?
  • Why would this async handler still block other requests?
  • Which isolation level would let two concurrent withdrawals both succeed, and how would you stop it?

How would you design a scalable ledger system that ensures strong consistency?

Medium · System Design & Architecture

Approach

  1. State the invariants first: every transaction's postings sum to zero, entries are immutable and append-only, each request applies at most once, and a balance never goes below its allowed limit. Store amounts as integers in minor units with a currency code.
  2. Schema: accounts (id, currency, balance, version), transactions (id, idempotency_key unique, status, created_at), entries (transaction_id, account_id, signed amount). A transfer is one database transaction that inserts the transaction row, inserts both entries and updates both balances, locking the two accounts in a fixed order (by id) to avoid deadlocks, or using a conditional update that only succeeds if the balance covers the debit.
  3. Idempotency: the unique constraint on the idempotency key makes a retry return the original result instead of applying twice. Corrections are new compensating entries, never updates.
  4. Scale: partition by account id so most transfers stay inside one partition; for cross-partition transfers use a saga through a pending or suspense account (debit and hold, then credit, compensate on failure) or two-phase commit, and say which you pick and why. Serve history from replicas, but do balance checks on the primary.
  5. Add a reconciliation job that recomputes balances from entries and checks that all entries sum to zero, and alert on drift.

Follow-up

  • How do you handle a hot account that receives a large share of all transfers?
  • A transfer spans two partitions and the credit side fails. What state is the money in, and how do you resolve it?
  • How would you reverse a transaction that was applied in error?

How do you ensure data integrity when designing distributed systems?

Medium · System Design & Architecture

Approach

  1. Start from where integrity breaks in distributed systems: duplicate delivery, partial failure between two writes, concurrent updates, and drift between systems of record.
  2. Duplicates: idempotency keys on every mutating API and idempotent consumers with a processed-message table, because at-least-once delivery is the realistic guarantee.
  3. Partial failure: avoid dual writes by using a transactional outbox (write the business row and the outgoing event in one local transaction, relay publishes afterwards) and sagas with compensating steps for multi-service workflows.
  4. Concurrency and validity: enforce invariants in the database with unique, foreign-key and check constraints, and use row locks or version checks where a read-then-write could race.
  5. Detection: reconciliation jobs that compare internal ledgers with each other and with external statements, plus checksums or counts on data pipelines, so drift is found instead of assumed away.

Follow-up

  • The outbox relay publishes an event and crashes before marking it sent. What happens next, and is that safe?
  • A consumer receives the same event twice, ten minutes apart. How does it know?
  • Reconciliation finds a mismatch. What is the process for correcting it?

How would you handle high-contention writes in a transactional system?

Medium · System Design & Architecture

Approach

  1. Confirm contention is on specific keys (a popular account, an inventory item), not overall volume, because the fixes differ.
  2. Single-statement atomic updates first: UPDATE accounts SET balance = balance - :amt WHERE id = :id AND balance >= :amt, checking affected rows. This avoids read-modify-write races without a separate lock round trip.
  3. When multiple rows must change together, use pessimistic locking (SELECT ... FOR UPDATE) with short transactions and a consistent lock order to prevent deadlocks. Avoid optimistic retry loops on the hottest keys, where conflict rates make them waste work.
  4. For a truly hot key, split it into N sub-rows or buckets and sum on read, or serialise writes per key through a partitioned queue or single-writer worker, and batch or net small writes. State the cost: bucketed balances make an exact overdraft check harder, and queued writes add latency.
  5. If you use SERIALIZABLE isolation, plan for serialization failures and retry with backoff.

Follow-up

  • With the balance split across buckets, how do you still prevent an overdraft?
  • How do you choose the number of buckets, and change it later?
  • Two transfers lock accounts A and B in opposite orders. What happens, and how do you prevent it?

Read latency spikes on a sixty-second sawtooth

Medium · cache stampede · single-flight · ttl

The cached listing read path serves about 14k reads/second at an 85% hit rate. p99 sits at 35 ms for 57 seconds, jumps to 900 ms for 3, and repeats. During each spike the primary shows several hundred identical listing queries starting within the same millisecond, all carrying one large tenant's id. Cache entries use a 60-second TTL. Give the mechanism, the ordered checks, the fix, and the correctness hazard your fix must not introduce.

Approach

  1. Match the period to a configured number before theorising about load. A spike every 60 seconds against a 60-second TTL is an entry expiring, and you confirm it by correlating spike timestamps with the entry's write time rather than with the traffic curve. If the period had matched a cron or a GC interval instead, this is a different investigation.
  2. Establish the concurrency of the miss. Several hundred identical queries in one millisecond means the miss path has no coalescing: every request that arrives between expiry and repopulation recomputes. The herd size is that key's arrival rate times its recompute time, so at 1.2k reads/second for the hot key and a 250 ms recompute you expect about 300 concurrent misses, which matches what is observed.
  3. Add single-flight on the miss path so one caller per key recomputes under a short-lived lock while the rest wait for its result. Prefer stale-while-revalidate where the read tolerates it: return the expired value immediately and refresh asynchronously, which removes the latency spike rather than serialising it into a queue of waiters.
  4. De-synchronise the keys. Write TTLs with jitter, for example 60 seconds plus or minus 10%, so a deploy or a mass invalidation does not align every key on the same second and turn a per-key herd into a fleet-wide one.

Follow-up

  • The same sawtooth appears on a key that is invalidated on write rather than expired. Is that the same bug?
  • How does your answer change if the recompute takes 4 seconds instead of 250 ms?

Suggested practice checklist

The plan follows the four reported rounds in order: a CV and screening pass, then the resume deep-dive questions, then ledger design and concurrency control, then a full ledger build with extension and tests, ending with a mock run of both technical rounds.

Day 1 — Screening pitch and CV audit

  • Write a short introduction that leads with your most transactional or concurrency-heavy project and names your main backend language and database.
  • List every technology on your CV, rate how deeply you can explain its internals, and delete or study each weak item.
  • Map the reported requirements (backend language, database design, distributed or transactional systems; event sourcing and CQRS as a plus) to evidence from your work and note the gaps.

Deliverable: A written introduction, a CV technology list with depth ratings and removals, and a requirement-to-evidence map with gaps marked.

Day 2 — Resume deep-dive answers

  • Redraw your most recent project's architecture from memory and trace one write and one read path through it out loud.
  • Write two trade-off stories with the rejected option, the deciding constraint, the cost and the outcome.
  • Pick the dimension that grows tenfold in that project, name the first three things that break, and write the fix for each.
  • Write one paragraph per listed framework or language on its concurrency, memory and transaction internals.

Deliverable: An architecture diagram with traced paths, two written trade-off stories, a 10x bottleneck list with fixes, and internals notes for each listed technology.

Day 3 — Ledger design with strong consistency

  • Design a double-entry ledger: schema, the transfer transaction, idempotency key handling, and balance reads on primary versus replicas.
  • Extend it to partitioning by account id and a cross-partition transfer through a pending account with compensation.
  • Write the integrity layer: transactional outbox, idempotent consumers, database constraints and a reconciliation job, and for each say what it protects against.

Deliverable: A ledger design document with schema, transfer flow, cross-partition flow, and an integrity section listing each mechanism and the failure it covers.

Day 4 — Concurrency control under contention

  • Write the SQL for a version-checked update, a SELECT ... FOR UPDATE transfer with ordered locks, and a conditional balance update, and note when each fails.
  • Build a table comparing optimistic locking, pessimistic locking, conditional updates and event-sourced streams on conflict behaviour, hot-key throughput, audit trail and read cost.
  • Outline a hot-account fix with bucketed sub-balances and a per-key write queue, including how the overdraft check still works.
  • Work the practice question on why an owner filter ignores the listing index, since database design is among the reported requirements.

Deliverable: Three annotated SQL snippets, a four-way comparison table, a hot-account outline, and a written answer to the index question.

Day 5 — Build the ledger

  • In your interview language, build a ledger from an empty file on a timer: accounts, deposit, withdraw, transfer, balance, history, idempotent replay.
  • Make transfers thread-safe with ordered per-account locks and keep money in integer minor units.
  • Add a fee transaction type through a strategy or command class and count the files you touched.

Deliverable: A runnable ledger project with thread-safe transfers and one added transaction type, plus a note of the time taken and the files changed for the extension.

Day 6 — Test the ledger

  • Write unit tests for the edge-case list: zero and negative amounts, exact-balance withdrawal, unknown account, currency mismatch, self-transfer, repeated idempotency key.
  • Add a property-based test checking that entries sum to zero and balances equal their entries after random operation sequences.
  • Add a multi-threaded opposite-direction transfer test and a fault-injection test that fails between debit and credit.

Deliverable: A test suite with edge-case unit tests, one property-based invariant test, one concurrency test and one fault-injection test, all passing.

Day 7 — Mock both technical rounds and rehearse pushback

  • Run a mock design session with a friend on concurrent financial transactions without data loss, and ask them to challenge every choice.
  • Rebuild the core ledger from scratch without notes, explaining aloud as you code.
  • Rehearse a story about arguing against a design and committing anyway, and one about turning a review disagreement into a decision.

Deliverable: Notes on the challenges raised in the mock design session and your answers, a second ledger build with time recorded, and two rehearsed stories.

Behavioural preparation

Candidates do not report a separate behavioural round for this role. The behavioural content sits inside the Resume Deep-Dive, where the reported questions ask about your trade-offs and what you would change, and candidates report that past-project limitations and defending design decisions under challenge come up. Prepare stories that show a decision, its cost and what you learned, in your own voice.

  1. What were the most significant technical trade-offs you made in your previous role, and why?
  2. How would you have approached your last project differently if you had to scale it tenfold?
  3. Describe a limitation in a system you built that you only recognised after it shipped, and what you did about it.
  4. Tell me about a time you argued against a design, lost, and committed to it anyway. What did you predict, and were you right?
  5. Describe a code review disagreement about correctness, such as a silently ignored update conflict, and how it was resolved.

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

  1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
  2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
  3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
  4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.

Follow-up

  • How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
  • Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

  1. Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
  2. Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
  3. Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
  4. Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.

Follow-up

  • Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
  • The author says clients cannot handle a 409. How do you check whether that is true?

Argue against a design, lose, and commit anyway

Medium · disagreement · service boundaries · decision records

Describe a design you argued against and lost. State the failure you predicted as a named mechanism, not a feeling about complexity: two services that would need one transaction, a projection with no rebuild path, a write path with no idempotency key. Say what evidence you brought, what the decision maker weighed instead, and what you did after the decision was made: what you instrumented, what you wrote down, and whether the prediction came true. Five minutes.

Approach

  1. State the prediction in falsifiable form up front: the mechanism, the condition that triggers it, and the observable outcome. A prediction that cannot be checked also cannot be credited to you later.
  2. Show the evidence you had at the time and label each piece honestly as measured, analogous, or intuition. Keeping the intuition is fine; disguising it as data is the thing that erodes your standing in the next argument.
  3. Represent the opposing case at full strength, including the constraint you did not control: a fixed date, a team boundary, or the fact that the decision was cheap to reverse and yours was not.
  4. Make disagree-and-commit concrete. Name the artefact you left behind so the prediction could be settled without you: the alert and its threshold, the counter on the dashboard, the decision note that recorded the trade-off and the condition that would revisit it.

Follow-up

  • What threshold on that alert would have proved you right, and did anyone ever look at it?
  • If the same proposal arrived tomorrow with the same deadline, would you argue it the same way?