BlockFi Software Engineer Interview Guide

The 60-second brief

BlockFi is described as a digital-asset finance company whose products include crypto-backed loans, interest-bearing accounts and trading systems. Candidates describe the Software Engineer role as backend work on the services behind those products: transactional systems, ledger databases and user-facing APIs, with ownership running from design through testing, deployment and monitoring. The reported stack leans on Elixir and the Phoenix framework for concurrent services, PostgreSQL for storage and AWS for infrastructure, and the listed requirements also name Go, Ruby, Java and C# as acceptable backend languages. Because the services move user assets, correctness of balances and an auditable record of every transaction come up repeatedly in what candidates report.

This guide is for backend-leaning Software Engineer candidates preparing for BlockFi. It covers the four reported rounds, the scenario questions on wallet balances, outages, traffic spikes and duplicate-safe processing, the locking, actor-model, parsing and SQL questions, and the resume-driven rounds where you defend your own past decisions. It does not assume prior cryptocurrency experience, and it explains how to prepare for Elixir-flavoured questions if your main language is something else.

Candidates report four rounds over roughly 3-5 weeks: a Recruiter Screen, a Technical Scenario Screen, a Chronological Experience Interview and a Panel & Product Discussion. Panel sessions are reported to sometimes be scheduled close together.

Focus areas

• Programming problem solving
• System design
• Elixir

Overview

The reported questions cluster around one theme: keeping money-like state correct while many things happen at once. Several scenarios are about concurrent updates to a balance, a counterparty or blockchain node failing halfway through a transaction, a surge of traffic during a volatile market, and a pipeline that must not apply the same transaction twice. The domain questions follow the same thread: optimistic versus pessimistic locking in a ledger, process-based concurrency compared with threads, a function that validates a stream of transaction payloads, and a slow SQL statement that blocks writers. If you can state the invariant, name the mechanism that protects it, and describe what happens when a step fails or is retried, you cover most of this set.

Candidates also describe a loop that leans on conversation more than on algorithm puzzles. Two of the four rounds are built around your own history and your work with product people: a resume walk-through and a panel with engineering leads, peers and product managers. The questions there are about the hardest system you designed, a production failure, a decision you regret, and balancing technical debt against a deadline. Preparation for these is different from preparation for the scenarios. You need specific projects, the alternatives you rejected, and numbers or outcomes you can defend.

On the stack, candidates report Elixir, the Erlang VM, Phoenix, PostgreSQL and AWS, with Kafka, RabbitMQ and Redis listed among the distributed-systems skills. Elixir experience is described as a plus rather than a requirement, and candidates are advised not to claim production Elixir they do not have. The practical route is to be strong in your own language, understand why processes with mailboxes and supervisors differ from shared-memory threads, and be fluent in PostgreSQL transactions and locking. The plan below gives each of those a dedicated day before the full run-through.

The BlockFi interview process

These are the stages candidates describe, in the order they report them. Treat the descriptions as a guide to the format, since individual loops vary.

Recruiter Screen

Candidates describe the Recruiter Screen as a short conversation about your background, your career goals, your compensation expectations and how well you fit the company's domain. No technical problem is reported for this stage. Use it to give a clear, concrete account of your work and to find out what the remaining rounds will cover.

What to demonstrate

• Your background and how it connects to transactional backend work
• Career goals and interest in the digital-asset domain
• Compensation expectations stated plainly

How to prepare

• Write a two-minute walk through your career that ends on why backend work on lending, interest-bearing accounts or trading systems appeals to you.
• Pick your closest experience to transactional systems (payments, ledgers, orders, inventory) so the domain question gets a concrete example rather than a general statement.
• Decide your compensation expectations and any constraints before the call, and ask which topics the later rounds will cover.

Technical Scenario Screen

The Technical Scenario Screen is reported as a deep dive into system design and programming concepts through real-world scenarios, including debugging hypothetical architectures. Candidates mention Elixir, concurrency and database management as the domain knowledge on show, and describe design themes such as database choice and modelling, locking, and resilience patterns like circuit breakers, retries with exponential backoff and dead-letter queues. The reported scenario and domain questions on wallet balances, outages, traffic spikes, idempotent pipelines, locking, actor-model concurrency, payload parsing and slow SQL are the material to practise for a stage like this. Answer each by stating the invariant, the mechanism and the failure mode.

What to demonstrate

• Reasoning about concurrency and safe state changes in a financial setting
• Design choices for databases, queues and failure handling
• Domain knowledge such as Elixir, concurrency models and database management

How to prepare

• Rehearse the four scenarios aloud: concurrent wallet updates, a gateway or node failing mid-transaction, a traffic spike, and a duplicate-safe processing pipeline. For each, name the invariant, the mechanism and what a retry does.
• Refresh PostgreSQL locking: row locks with SELECT ... FOR UPDATE, version-column optimistic checks, isolation levels, and consistent lock ordering to avoid deadlocks.
• Prepare a clear comparison of the actor model (isolated processes, mailboxes, supervisors) against shared-memory threads, even if your own language is different.
• Practise reading a slow query: EXPLAIN ANALYZE, the index it should use, and how to find sessions waiting on locks.

Chronological Experience Interview

The Chronological Experience Interview is reported as a peer-led technical discussion in which you walk through your resume and explain architectural challenges and project outcomes. Candidates describe topics such as the reasoning behind technical decisions and the alternatives rejected, scaling bottlenecks you resolved, post-mortems of outages or bugs, zero-downtime database migrations and splitting a monolith into services. The format is detailed storytelling about real work, so your preparation is a set of well-structured project accounts rather than new study.

What to demonstrate

• Depth of your own contribution to systems on your resume
• The rationale behind technical decisions and the trade-offs you accepted
• How you handled failures, bottlenecks and technical debt

How to prepare

• For each of your last two or three projects write one page: goal, architecture, your part, two decisions with the alternative you rejected, and one bottleneck with before-and-after results you can support.
• Prepare the production-failure story and the regretted-decision story in full, including what you changed afterwards.
• Prepare a zero-downtime migration or monolith-decoupling example. If you have none, outline how you would do an expand-then-contract schema change on a busy PostgreSQL table.
• Be ready to name the hardest system you designed from scratch and the trade-offs in it.

Panel & Product Discussion

Candidates describe the Panel & Product Discussion as a series of conversations with engineering leads, peer developers and product managers about cross-functional communication and behavioral fit. Reported topics include aligning technical design with user experience and business goals, negotiating time for refactoring with product managers, proceeding when requirements are vague, and responding to a product request that adds security or scalability risk. Sessions are reported to sometimes follow each other closely, so prepare short, reusable stories and a way to reset between them.

What to demonstrate

• Communication of technical constraints to non-engineers as business trade-offs
• Handling of technical debt and ambiguous requirements alongside product work
• Behavioral fit across several conversations

How to prepare

• Rehearse explaining a technical risk to a product manager as a trade-off of user impact, delivery date and cost, and offering a safer smaller scope.
• Prepare a technical-debt story where you negotiated time for refactoring while a feature deadline stood.
• Prepare a story where you found a user pain point and worked with product to design and build a fix.
• Write a one-page summary of your stories and questions for the team, and review it between back-to-back sessions.

Where candidates lose points

Answering the concurrent wallet-balance question with "use a database transaction" and nothing more

Name the mechanism. Either lock the row (SELECT ... FOR UPDATE, locking accounts in a fixed order to avoid deadlocks) or use a version column with a conditional update and retry, and say when each fits. Add an append-only ledger of entries so the balance can be rebuilt and audited.

Describing idempotency as "check whether the transaction already exists, then insert"

That check-then-insert has a race. Store a client-supplied idempotency key under a unique constraint, written in the same transaction as the effect, so a retry after a timeout hits the constraint and returns the original result. Then cover what happens if the gateway call succeeded but the commit did not.

Telling resume stories at headline level, with no alternatives, numbers or personal contribution

For each project, write down your own role, two decisions with the option you rejected, and a bottleneck with measurements you can back up. A chronological walk-through goes deep, so a story with only a summary gives you nothing to say on the second question.

Claiming Elixir or OTP expertise you cannot support

Candidates are advised not to fake it. State your real primary language, then explain the concepts that transfer: isolated processes with mailboxes, supervisors restarting failed workers, message passing instead of shared memory, and what the actor model does and does not make easier.

Answering a product-panel scenario purely technically, such as refusing a risky feature outright

Frame the risk as a trade-off the product manager can weigh: user impact, delivery date and cost. Offer a smaller, safer scope or a staged rollout, and say what you would measure to decide on the full version.

Questions & practice

How would you design a system to handle high-concurrency wallet balances without causing race conditions or da

Medium · Scenario-Based Systems & Architecture

How would you design a system to handle high-concurrency wallet balances without causing race conditions or database deadlocks?

Approach

1. Treat an append-only ledger of signed entries as the source of truth and keep a balance row per wallet as a cache updated in the same database transaction. Store amounts as integer minor units or NUMERIC, never floats, since crypto assets carry 8 or more decimal places.
2. Debit with one atomic conditional statement: UPDATE wallets SET balance = balance - $1 WHERE id = $2 AND balance >= $1, then check that exactly one row changed. In Postgres the row lock and the WHERE re-check on the newest row version close the read-then-write race without a separate SELECT.
3. Prevent deadlocks on transfers by locking both wallets in a fixed order (lower id first, e.g. SELECT ... FOR UPDATE ORDER BY id). Keep transactions short and never make a network call while holding a row lock.
4. For a hot wallet that takes most of the writes, give each wallet a single writer: partition commands by wallet id onto a queue, or one process per wallet on the BEAM. Name the cost, which is per-wallet throughput capped by one serial writer.
5. Edge cases: a retried request applying twice (put an idempotency key with a unique constraint on each ledger entry), overdraft attempts, and drift between the cached balance and the ledger sum (a reconciliation job that compares the two).

Follow-up

• How would you detect and repair drift between the cached balance and the sum of ledger entries?
• Would SERIALIZABLE isolation remove the need for explicit locks, and what must the application do when Postgres aborts a transaction with a serialization failure (SQLSTATE 40001)?
• One omnibus account receives most deposits. How do you stop it becoming the bottleneck?

Explain the difference between optimistic and pessimistic locking, and when you would use each in a financial

Medium · Practical Coding & Domain Knowledge

Explain the difference between optimistic and pessimistic locking, and when you would use each in a financial ledger system.

Approach

1. Pessimistic locking takes the lock before the read-for-update, e.g. SELECT ... FOR UPDATE in Postgres, which holds the row lock until commit so competing writers wait. It is predictable under heavy contention, but it costs lock waits and risks deadlocks unless locks are taken in a consistent order.
2. Optimistic locking reads without a lock and writes conditionally on a version column: UPDATE ... SET balance = $1, version = version + 1 WHERE id = $2 AND version = $3. Zero rows affected means a conflict, so re-read and retry. It is cheap when conflicts are rare, but under contention retries waste work and can starve a writer.
3. Map it onto a ledger: inserting entries rarely conflicts, and the contention point is the balance row of a busy account. Use pessimistic locking or a single atomic conditional UPDATE for debits on the same account, and optimistic locking for low-contention edits such as account settings.
4. Mention the database-level options: SERIALIZABLE in Postgres behaves optimistically and aborts with SQLSTATE 40001, so the application must retry the whole transaction; NOWAIT and SKIP LOCKED let a worker avoid blocking. In Elixir, Ecto offers optimistic_lock/3 on a changeset and a lock: "FOR UPDATE" query option.
5. Edge cases: retries must be idempotent, a lock must never be held across a network call, and a read-then-write balance check under plain READ COMMITTED without a lock is a race.

Follow-up

• Two withdrawals hit the same account at the same moment under optimistic locking. Walk through exactly what each one sees.
• How does Postgres detect a deadlock, and what does your application do when its transaction is the one aborted (SQLSTATE 40P01)?
• When would you choose SERIALIZABLE isolation over explicit row locks?

How does the actor model or process-based concurrency (such as in Elixir or Erlang) differ from thread-based c

Medium · Practical Coding & Domain Knowledge

How does the actor model or process-based concurrency (such as in Elixir or Erlang) differ from thread-based concurrency? What are the advantages?

Approach

1. On the BEAM a process is a lightweight unit with its own heap and no shared memory. Processes communicate only by asynchronous messages to a mailbox and handle one message at a time, so state owned by a process (for example a GenServer) is updated serially with no locks.
2. Threads share an address space and need mutexes or atomics for any shared state, which brings data races, lock ordering and deadlocks. OS threads also carry much larger stacks and costlier context switches, while BEAM processes are cheap enough to run hundreds of thousands at once.
3. Explain the runtime: one scheduler per core, preemption by reduction count so a busy process cannot starve others, and per-process garbage collection with no global stop-the-world pause. Fault tolerance comes from links, monitors and OTP supervisors that restart a crashed process from a known state.
4. Give the costs too. Messages are copied between heaps (large binaries over 64 bytes are reference-counted and shared). A single GenServer serialises every call and can become a bottleneck. Mailboxes are unbounded, so backpressure has to be designed in.
5. Actors do not remove every concurrency bug: two GenServers calling each other synchronously can deadlock until GenServer.call's default 5-second timeout, and message order is guaranteed only between one pair of processes, so races across actors remain possible.

Follow-up

• How would you model each wallet as a process, and what happens to an in-flight update if that process crashes?
• One GenServer has become a hotspot. How would you partition its work across processes?
• How do you apply backpressure when a consumer's mailbox keeps growing?

Write a clean, well-tested function to parse and validate a stream of incoming transaction payloads, ensuring

Medium · Practical Coding & Domain Knowledge

Write a clean, well-tested function to parse and validate a stream of incoming transaction payloads, ensuring robust error handling for malformed data.

Approach

1. Fix the contract first: the format (say newline-delimited JSON), the required fields (id, account, amount, currency, timestamp) and their types, and a per-record result such as {:ok, txn} or {:error, reason, line_number}. A malformed record produces an error value; it does not crash or end the stream.
2. Process lazily (Elixir Stream or a Python generator) so memory stays constant per record and time is O(n). Validate in stages: decode, then shape (fields present, right types), then meaning (amount positive, precision within the currency's limit, known currency, parseable timestamp).
3. Never parse money as a float: Python's json.loads(s, parse_float=decimal.Decimal) or Jason.decode(s, floats: :decimals) keeps exact values, or require amounts as strings. Decide whether duplicate transaction ids are rejected, and bound the memory used to track seen ids.
4. Collect errors with the line number and reason and route them to a dead-letter sink, so valid records keep flowing and bad ones can be inspected and replayed.
5. Write table-driven tests for: a valid record, a missing field, a wrong type, zero or negative amount, too many decimal places, invalid JSON, an empty line, a duplicate id, a huge number and a truncated final line. Add a property-based test (StreamData or Hypothesis) asserting the parser never raises on arbitrary input.

Follow-up

• The input arrives in chunks and a record is split across two of them. How does the parser handle that?
• Should one malformed record stop the whole batch? Argue both sides for a financial feed.
• A producer adds a new optional field. How do you accept it without breaking older producers or consumers?

How would you optimize a slow-running SQL query that is blocking critical database transactions in a high-writ

Medium · Practical Coding & Domain Knowledge

How would you optimize a slow-running SQL query that is blocking critical database transactions in a high-write environment?

Approach

1. Diagnose before changing anything: pg_stat_activity (wait_event_type = 'Lock', query_start, state 'idle in transaction'), pg_blocking_pids(pid) to see who blocks whom, and pg_stat_statements for total and mean time. Run EXPLAIN (ANALYZE, BUFFERS) on a replica or copy, since ANALYZE actually executes the statement.
2. Know what really blocks under MVCC: a plain SELECT does not block an UPDATE in Postgres. The usual causes are a long transaction holding row locks (UPDATE or SELECT ... FOR UPDATE), DDL waiting on an ACCESS EXCLUSIVE lock that then queues everything behind it, or a session left idle in transaction. Long transactions also hold back vacuum and bloat the table.
3. Fix the query: add an index that matches the predicate (equality columns before range columns in a composite index), avoid wrapping indexed columns in functions, and build it with CREATE INDEX CONCURRENTLY so writes continue (it cannot run inside a transaction block and can leave an INVALID index if it fails).
4. Shrink the lock window: break large updates into batches keyed by primary key, each in its own short transaction, and move reporting reads to a replica. Add statement_timeout, lock_timeout and idle_in_transaction_session_timeout as guardrails.
5. Weigh write cost, since every extra index slows inserts on a write-heavy table. Then verify by comparing plans before and after and watching p99 latency and lock-wait counts.

Follow-up

• What are the risks of CREATE INDEX CONCURRENTLY, and how do you clean up if it fails?
• A nightly job must update millions of rows in this table. How do you run it without blocking live transactions?
• How do you add a column with a default to a large, busy table without a long lock?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

1. Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
2. For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
3. Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
4. Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.

Follow-up

• 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
• A dashboard runs this for 40 owners in one page load. What changes about the design?

What would you do if a third-party payment gateway or blockchain node goes down mid-transaction? How do you en

Medium · Scenario-Based Systems & Architecture

What would you do if a third-party payment gateway or blockchain node goes down mid-transaction? How do you ensure eventual consistency?

Approach

1. Record the intent before calling out: write a transaction row in a PENDING state with an idempotency key and place a hold on the user's funds, then call the gateway or node. Use an outbox table so the state change and the outbound message commit together and no dual-write gap opens.
2. Treat a timeout as an unknown outcome, never as a failure. Retry with the same idempotency key if the gateway supports one, or look the transaction up by its reference. On-chain, rebroadcasting the same signed transaction is safe because it carries the same hash and nonce (or spends the same inputs), so it cannot land twice.
3. Drive each transaction through an explicit state machine (created, submitted, confirmed, failed, reversed) and run a reconciliation job that moves stale PENDING rows to a terminal state from the provider's records or the chain. For chains, count a deposit or withdrawal as final only after a chosen number of confirmations, so a reorg cannot undo it.
4. Contain the outage: retries with exponential backoff and jitter, a circuit breaker so callers fail fast while the dependency is down, a dead-letter queue plus an alert once retries run out, and compensating actions (release the hold, reverse the ledger entry) in place of distributed locks.
5. Show users the true state (pending, not completed) and make every consumer of the resulting events idempotent, since recovery paths can deliver the same event again.

Follow-up

• The gateway charged the card but your service crashed before saving the response. How do you find and fix that record?
• A transaction you marked confirmed disappears in a chain reorganisation. What happens to the user's balance?
• How long do you keep funds on hold for an unresolved transaction, and what decides that limit?

How would you handle a massive spike in user traffic during a high-volatility market event? What rate-limiting

Medium · Scenario-Based Systems & Architecture

How would you handle a massive spike in user traffic during a high-volatility market event? What rate-limiting or queuing strategies would you employ?

Approach

1. Split the traffic by class first. Market data and balance reads can come from a cache or a pub/sub fan-out (websockets fed by one upstream subscription), so they never hit the primary database per request. Orders and withdrawals are writes that must not be lost and must keep per-account order.
2. Rate-limit at the edge with a token bucket per user or API key. It allows bursts up to the bucket size and sustains the refill rate, which avoids the double burst a fixed window allows at the window boundary. Back it with an atomic Redis script and return 429 with Retry-After.
3. Put writes on a durable queue (Kafka or RabbitMQ), acknowledge with an id and a pending status, and partition by account id so one account's commands stay ordered while consumers scale out. Keep the queue bounded and reject new work once depth means it would execute against a stale price.
4. Shed load by priority: protect risk-reducing actions and core trading, and switch off non-essential features with flags. Isolate pools (bulkheads) so a slow dependency cannot drain every thread or connection.
5. The database is usually the real ceiling. Cap connections with a pooler such as PgBouncer, pre-scale before scheduled events (autoscaling reacts too slowly to a sudden spike), and load-test the peak before relying on any of this.

Follow-up

• The rate limiter's Redis goes down. Do you fail open or fail closed, and does the answer differ for reads and for withdrawals?
• A queued market order executes after the price has moved sharply. How should the system and the user-facing status handle that?
• How would you choose the bucket size and refill rate for a retail user versus an API client?

Describe how you would implement an idempotent transaction processing pipeline to prevent double-spending or d

Medium · Scenario-Based Systems & Architecture

Describe how you would implement an idempotent transaction processing pipeline to prevent double-spending or duplicate transactions.

Approach

1. Have the client send an idempotency key per logical operation. Store it in a table with a unique constraint on (account_id, key), together with a hash of the request and the stored response, and insert it in the same database transaction as the ledger entries so both commit or neither does.
2. Define the duplicate cases: same key and same payload returns the stored result; same key with a different payload is rejected as a client error; a key whose first attempt is still running gets a conflict response or a wait, never a second execution.
3. Assume queues deliver at least once. Each consumer records processed message ids in the same transaction as its side effect, or writes with INSERT ... ON CONFLICT DO NOTHING, so a redelivery has no effect. Exactly-once effects come from at-least-once delivery plus idempotent processing.
4. Idempotency only stops the same request applying twice; it does not stop two different requests spending the same funds. Pair it with an atomic balance check (a conditional UPDATE or a row lock) so concurrent debits cannot overdraw.
5. State the trade-offs: how long keys are retained (storage against the client's retry window), and publishing downstream events through an outbox after commit so a crash between commit and publish neither drops nor duplicates the event.

Follow-up

• The process crashes after calling an external payment provider but before saving the response. What happens on the client's retry?
• How long do you keep idempotency keys, and what goes wrong if a client retries after they expire?
• How do you make a call to a downstream service that does not accept idempotency keys safe to retry?

Walk me through the most technically challenging system you have designed from scratch. What trade-offs did yo

Medium · Chronological Experience & Deep Dive

Walk me through the most technically challenging system you have designed from scratch. What trade-offs did you make?

Approach

1. Choose a system you designed yourself and can defend at every layer. Open with the problem and its constraints in numbers you can stand behind: load, data volume, consistency needs, team size and deadline.
2. Sketch the architecture in order: entry points, services, data stores, queues, and the path of one request. Say plainly which parts you owned and which belonged to others, speaking as I, not as a team.
3. Name two or three real trade-offs, each with the alternative you rejected and why, for example a relational store with transactions over a document store because multi-row updates had to be atomic, or eventual consistency for reporting in exchange for write throughput.
4. Close with measured results, what broke or strained later, and what you would change now. A design with no weaknesses you can name sounds rehearsed.
5. Prepare a second level of detail for any box you draw: schema, failure handling, deployment and tests. Expect to be pulled into any of them.

Follow-up

• Which component fails first at ten times the load, and how would you know?
• You rejected one alternative. With what you know now, would you still reject it?
• How did you test the system's failure modes before it went live?

What is an architectural decision you made in a past project that you now regret, and what would you do differ

Medium · Chronological Experience & Deep Dive

What is an architectural decision you made in a past project that you now regret, and what would you do differently today?

Approach

1. Pick a decision you made or drove yourself, not one imposed on you, and give the context that made it reasonable at the time: the deadline, the scale, the information you had.
2. Describe the cost concretely: the symptom that surfaced and when. Examples: a database shared by several services blocking independent deploys, synchronous calls chaining failures across services, or a schema that made a later change expensive.
3. Explain what you did about it. If you migrated, describe the zero-downtime path: expand and contract (add the new structure, dual-write, backfill, switch reads, remove the old) with a rollback point at each step.
4. State what you would do differently and the earlier signal you now watch for. That shows a lesson you can reuse, not only an apology.
5. Avoid the answers that sink this question: a trivial regret, a disguised boast, or blaming whoever approved the decision.

Follow-up

• What signal could have told you earlier that the decision was wrong?
• With only the information you had then, would you still make the same call?
• How did you move off it without downtime or data loss?

How have you handled technical debt in past roles while still delivering product features on a tight deadline?

Medium · Chronological Experience & Deep Dive

Approach

1. Tell one concrete story: the debt, its measurable cost (incidents, on-call pages, slower changes in that area) and the deadline it collided with.
2. Show how you triaged. Fix now any debt that blocks the feature or threatens correctness of money or data; defer cosmetic debt; limit any refactor to the code the feature touches.
3. Explain how you made the case to product in their terms (delivery risk, incident risk, time saved on the next feature) and what you agreed: a ticket with an owner, capacity in the next cycle, or a narrower first release.
4. Name the guardrails you used to ship safely on top of the debt: characterisation tests around the risky area before changing it, feature flags, and extra monitoring after release.
5. Finish with follow-through: whether the deferred debt was actually paid, and what happened if it was not.

Follow-up

• Tell me about debt you deliberately left unpaid. What did it cost later?
• How did you convince a product manager to give up feature time for the refactor?
• How do you decide whether a piece of debt is worth paying at all?

One log partition stops advancing while the others drain

Easy · poison message · head-of-line blocking · consumer lag

Search results for a subset of tenants are hours stale; the rest are current. The projection consumer reports lag of zero on 15 of 16 partitions and 400,000 on one. Its error rate is flat and its CPU is idle. outbox_event has no pending rows older than a second, so the relay has published everything it holds. Identify the mechanism, give the ordered checks, and state what you do in the first ten minutes versus what you change permanently.

Approach

1. Read the lag distribution first. A slow consumer lags everywhere; zero on fifteen partitions and 400,000 on one is not throughput. Idle CPU on the stuck partition means the consumer is not advancing its offset at all, which points at one message it cannot get past rather than at a rate problem.
2. Exonerate the producer before touching the consumer. No pending outbox rows older than a second means the relay published, so the event exists in the log. This separates never sent from sent and never applied, which are different code paths and usually different owners.
3. Read the message at the stuck offset and the handler's log lines for its event_id. A flat error rate with no progress has two explanations and you must distinguish them: the handler is throwing and the retry loop is swallowing it, or the handler is blocking on something and never returning. Idle CPU with no error lines favours the second.
4. Mitigate before diagnosing further. Move the offending event to a dead-letter store and commit the offset past it. Adding consumers does nothing here, because a partition is consumed by exactly one member of the group, and the blast radius is every aggregate hashed to that partition, not only the aggregate that produced the bad event.

Follow-up

• The dead-lettered event carried aggregate_version 7 and the projection had applied 6. What must the replay do differently if 8 and 9 landed in the meantime?
• How do you show staleness to the user while the partition is behind, given the API already returns the projection's watermark?

Suggested practice checklist

Seven days built around the four reported rounds: a loop and project inventory first, then ledger concurrency, failure and idempotency scenarios, Elixir-style concurrency with a coding piece, the resume stories, and finally the incident story and a full run-through.

Day 1 — Map the loop and inventory your projects

• Write out the four reported rounds and, beside each, the question types it covers and the story or topic you are least ready for.
• List your last two or three projects with your role, the main architecture, and one decision and one failure from each.
• Write your two-minute career walk and a sentence on why digital-asset finance and transactional backend work interest you.

Deliverable: A one-page round map, a project inventory and a written two-minute career walk.

Day 2 — Ledger concurrency and locking

• Answer the concurrent wallet-balance question aloud, covering row locks, lock ordering, a version-column alternative and an append-only entries table.
• Write the optimistic-versus-pessimistic locking comparison for a ledger: contention level, cost of a retry, and which one you would choose for hot accounts.
• Take a slow, write-blocking query on a local PostgreSQL table, run EXPLAIN ANALYZE, add or change an index, and write down what changed.

Deliverable: Spoken answers to the wallet-balance, locking and slow-query questions, plus the before-and-after plan from your local experiment.

Day 3 — Failure handling, spikes and idempotency

• Design the idempotent transaction pipeline on paper: idempotency key, unique constraint, state machine (accepted, pending, failed, confirmed) and the retry path after a timeout.
• Answer the outage-mid-transaction question: timeouts, retries with exponential backoff and jitter, a circuit breaker, a dead-letter queue and reconciliation to reach consistency.
• Answer the traffic-spike question: rate limiting at the edge, queueing writes, shedding or degrading reads, and protecting the ledger database.

Deliverable: Three design sketches (pipeline, outage handling, spike handling) and a spoken answer for each, plus notes from the partition-debugging practice.

Day 4 — Actor model, Elixir basics and the coding question

• Write a short comparison of process-based concurrency and thread-based concurrency: isolation, mailboxes, supervisors, failure containment and where shared state still appears.
• Write a function in your strongest language that parses and validates a stream of transaction payloads, with explicit handling for malformed input and a clear error type.
• Write unit tests for it covering missing fields, wrong types, duplicate identifiers and a truncated final record.
• Read one small Elixir GenServer example and explain each callback in your own words.

Deliverable: A written actor-versus-thread comparison, a validated parser with its test file, and a plain-language explanation of one GenServer.

Day 5 — Chronological interview stories

• Write the answer to the hardest-system-from-scratch question: requirements, the architecture, two trade-offs and what you would change.
• Write the regretted-architectural-decision story, including what you would do today and what the decision cost.
• Write the technical-debt-versus-deadline story with the negotiation, the scope you cut and what you protected.
• Practise one of them aloud with a friend or a recording and cut anything that does not explain a decision.

Deliverable: Three written story outlines, one recorded spoken run of the strongest.

Day 6 — The production-failure story and behavioral practice

• Write the production-failure story using the order: impact, containment, diagnosis, root cause, data repair, prevention.
• List two follow-up questions for each of your stories and write the answers.

Deliverable: A finished incident story with follow-up answers and notes from two practice behavioral prompts.

Day 7 — Panel run-through and full mock

• Rehearse a product conversation: a manager asks for a feature that adds a scalability or security risk. Explain the risk as a trade-off and propose a safer scope.
• Run a mixed set aloud: the idempotent pipeline, the technical-debt story and the locking comparison, one after another, without notes.
• Write your questions for the engineering team about technical debt and scaling challenges, and the one-page story summary to review between sessions.

Deliverable: A recorded mixed run-through, a product-risk answer and a list of questions for the team.

Stories for the experience and panel rounds

Candidates describe the later rounds as centred on your own history and your work with product and QA: the systems you built, a production failure, a regretted decision, technical debt against deadlines, vague requirements, and requests that carry risk. Reported prompts come first below. For each, prepare a story where you made the call, show the trade-off and the result, and name what you changed afterwards.

1. What is an architectural decision you made in a past project that you now regret, and what would you do differently today?
2. How have you handled technical debt in past roles while still delivering product features on a tight deadline?
3. Walk me through the most technically challenging system you have designed from scratch. What trade-offs did you make?
4. A product manager asks for a feature that adds a significant security or scalability risk. How did you handle a situation like this?
5. Describe a time you had to proceed with requirements that were vague or changing, and how you reduced the uncertainty.

Describe a time when a system you were responsible for failed in production. How did you diagnose the issue, m

Medium · Chronological Experience & Deep Dive

Describe a time when a system you were responsible for failed in production. How did you diagnose the issue, mitigate the fallout, and prevent it from happening again?

Approach

1. Choose an incident where you made decisions during the response, not one you observed. Give the setup in two sentences: what failed, who or what was affected, and how you first learned of it. In a transactional system the impact should be concrete, such as duplicated writes, stale balances or a stuck queue.
2. Narrate in order and keep two phases apart: stopping the damage first (rollback, feature flag, pausing a consumer, failing over), then diagnosing with evidence (logs, metrics, traces, query plans, a reproduction). Say what you looked at first and why, and which hypothesis you ruled out.
3. State the root cause precisely, including contributing factors such as a missing alert, an untested retry path or a lock held by a long migration. Describe the cause in terms of the system and the process, not a person.
4. Cover the aftermath: how you checked and repaired affected data (for example reconciling records against the source of truth), the blameless review, and the concrete changes you shipped, such as a regression test, an alert, a timeout, an idempotency key or a runbook. Say how you later confirmed the change worked.
5. The answer fails when it stays vague ("we fixed it"), blames others, omits data repair, or ends with no prevention. Close by saying what you would do differently in the first ten minutes.

Follow-up

• How did you know that was the root cause and not just a symptom?
• What would have caught this before it reached production?
• What did you do about the records or users already affected?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

1. Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
2. Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
3. Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
4. Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.

Follow-up

• How would you detect a consumer that reads the field only during a monthly export?
• One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.

Follow-up

• How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
• Their handler calls an external endpoint before recording that it did. What do you tell them to change first?