deCircle Software Engineer Interview Questions & Guide 2026

The 60-second brief

As a Software Engineer at deCircle, you are at the forefront of the decentralized finance and blockchain infrastructure revolution. This role is fundamental to the company’s mission: you will build, maintain, and scale the high-performance systems that power deCircle's core products. Whether you are working on protocol-level development in Rust, architecting backend systems, or securing complex smart contracts, your contributions affect the security and efficiency of deCircle's decentralized ecosystem.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

• Build at-least-once pipelines with explicit deduplication horizons
• Make every write idempotent under client retries
• Keep money in integer minor units

Overview

The environment at deCircle is characterized by high technical rigor and a focus on innovation within the blockchain space. You will work alongside world-class engineers to solve non-trivial problems in distributed systems, cryptography, and protocol design. This position suits engineers who thrive in fast-paced, mission-driven environments where code quality, security, and scalability matter to deCircle's users and the broader network.

The deCircle interview process

deCircle candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Technical Screen

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

• Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
• Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
• Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
• Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

• From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
• Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
• Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Deep-Dive Coding Session

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Discussions

This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.

What to demonstrate

• Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
• Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
• Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage

How to prepare

• For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
• Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
• Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend

Where candidates lose points

Holding money in a floating-point type, or rounding it more than once

Binary floating point cannot represent 0.01 or 0.1 exactly, so sums drift and two code paths that should agree disagree by cents nobody can trace back. The fix is integer minor units or an exact decimal type end to end, with sub-cent rates expressed as scaled integers such as micro-units, because a per-request price genuinely is smaller than a cent. The second half of the trap is rounding position: rounding each line and then summing gives a different total from summing and rounding once, and half-up and half-even diverge systematically across many lines, so rounding must happen at one named place and every downstream reader must carry the rounded value rather than recompute it from quantity and rate.

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Starting work without saying what you are about to spend time on

State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.

Quoting amortised or average cost as if it were a worst-case guarantee

Appending to a dynamic array is amortised O(1), but the append that triggers a resize copies every element, and hash lookup is constant only while the hash spreads the actual keys. Say which guarantee you are offering when the caller cares about the latency of one call rather than the total over many.

Questions & practice

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

1. Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
2. The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
3. Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
4. Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
5. Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
6. State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

• A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
• The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
• What makes a re-run over the same day produce byte-identical rollups?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

1. Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
2. Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
3. Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
4. Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
5. Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
6. Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Follow-up

• The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
• How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
• Ingest is still running while you investigate. What makes your two numbers comparable at all?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

1. Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
2. Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
3. Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
4. Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
5. Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
6. Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Worked solution (15 min)

1. Write the grammar on one line before coding: header := field (',' field)*, field := key '=' value, split on the first = only.
2. Implement the parser to return {t: int, v1: [hex, ...]}, rejecting a missing t, a duplicate t, any v1 that is not 64 hex characters, and a header over 1 KiB, all before any cryptography runs.
3. Implement the verifier: for each active secret compute HMAC-SHA256(secret, f'{t}.'.encode() + raw_body), compare it in constant time against each parsed v1, and OR the results with no early exit.
4. Test with a valid signature; the same body with t moved 400 seconds into the past; the same body with t 400 seconds into the future; a header carrying an unknown v2= alongside a valid v1; and a body re-serialised with different JSON key order.

Expected result: The valid case accepts. Both out-of-tolerance cases reject, including the future one. The unknown v2 field is ignored and the v1 still verifies. The re-serialised body fails, which is correct and is exactly why the raw bytes must be retained.

Verify it yourself

• Timing the comparison over 10,000 mismatching signatures that differ at the first character versus the last shows no separation you can measure.
• Removing t from the MAC input makes an old captured body with its original signature pass. Write that test, watch it pass, then put t back.
• Every rejection except the signature mismatch itself returns before an HMAC is computed.

Follow-up

• The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
• A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
• How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

1. Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
2. Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
3. Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
4. Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
5. Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Worked solution (20 min)

1. Load about 50k deliveries for one tenant, then walk them with the OFFSET query while a writer inserts 10 rows/second, collecting every returned delivery_id.
2. Compare the distinct ids collected against the set of ids that existed when the walk started, and record the shortfall.
3. Repeat the walk with the keyset query and confirm every pre-existing id is returned exactly once.
4. Run explain (analyze, buffers) on page 1 and page 500 of each form and compare shared buffer hits.

Expected result: A keyset query using the row comparison (created_at, delivery_id) < ($2, $3) with order by created_at desc, delivery_id desc limit 100, backed by (tenant_id, created_at, delivery_id) scanned backwards; plus the statement that OFFSET drops rows under concurrent insert and costs time proportional to the offset, and that updated_at is disqualified because it mutates on retry.

Verify it yourself

• Buffer counts for page 500 are the same order of magnitude as page 1.
• Every id present before the walk is returned exactly once when only inserts run concurrently.
• The query plan shows no Sort node above the index scan.
• Two rows sharing a created_at value to the microsecond are both returned across the page boundary.

Follow-up

• The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
• A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
• The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

1. Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
2. Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
3. Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
4. Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
5. Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
6. Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Follow-up

• Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
• A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
• Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

How would you design a system to handle high-throughput transactions while maintaining low latency?

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. Choose a partition key and say what query it makes expensive.
3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

Describe the architecture of a recent project you led and the rationale behind your technical choices.

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. State the consistency you need, and where you are willing to be stale.
3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

What strategies do you employ to ensure data consistency in a decentralized environment?

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• What would you drop to keep the system up under load?
• How does this behave when that dependency is down for an hour?

How do you approach testing and auditing smart contracts to ensure protocol security?

Medium

Approach

1. Say what you would check first and why it is the highest-information step.
2. Work from the requirement backwards to the design.
3. State your assumptions explicitly before working the problem.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

How do you manage memory safety and concurrency in Rust?

Medium

Approach

1. Work from the requirement backwards to the design.
2. State your assumptions explicitly before working the problem.
3. Clarify what is being asked and what a complete answer contains.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Ship three breaking-looking changes without breaking pinned SDKs

Hard · versioning · backward compatibility · deprecation

GET /v1/runs/{id} returns status from the set queued, leased, running, succeeded, failed, timed_out, cancelled. You must add the terminal state lost, add an optional billable_seconds field, and start rejecting name values over 64 characters that were previously accepted. Clients are generated SDKs pinned inside customer build pipelines you cannot redeploy or reliably contact, some with strict deserialisers. Classify each change as compatible or breaking and justify it mechanically, then give the rollout: the versioning scheme, the transform old clients receive, the telemetry, and the sunset process.

Approach

1. Classify against a deserialiser rather than against intuition. A new enum value breaks a generated client that validates the enum or switches exhaustively; a new optional field breaks one generated with additionalProperties false; tightening validation breaks every caller already sending an 80-character name even though no field moved. All three are breaking for some real client, and that is the answer, not a technicality.
2. Choose the versioning axis on maintenance cost. A path version forks the handler, and each subsequent change multiplies the fork. A date-pinned version header, defaulted per credential at first use and pinned thereafter, keeps one internal model and a chain of ordered response transforms, so the handler stays single and the transforms compose.
3. Write the downgrade honestly. Older versions have no lost, so the transform maps it to the least-wrong existing value and the changelog says which; mapping it to failed asserts an outcome nobody observed, so pair it with a reason field the old version already exposes rather than inventing a field, and keep the mapping in one documented table instead of a conditional inside the serialiser.
4. Sequence the validation tightening separately from the payload changes: first measure how many tenants send names over 64 characters, then warn for a stated window with a Deprecation header and a non-fatal warning code, then reject only for versions dated after the change, never retroactively for a pinned client.
5. Deprecate from evidence, not from announcement: per-version, per-tenant, per-credential request counts from the gateway, a Sunset header carrying the date, direct contact for the handful of accounts that make up most of the tail, and short scheduled brownouts before the cutoff so the clients nobody can reach fail on your calendar rather than during their release.

Worked solution (40 min)

1. Write the three-row classification table with the client-side mechanism that breaks in each case, not just a verdict.
2. Pick the versioning scheme and write the header, the default-and-pin rule for a new credential, and where the transform chain sits relative to the handler.
3. Write the lost mapping for pre-change versions as an explicit table entry plus the changelog sentence a customer reads.
4. Write the validation timeline with four dated steps, and mark which one is visible to clients that never change their version.
5. Write the deprecation plan: the telemetry query by version and tenant, the headers, the brownout schedule, and the fallback for the largest holdout.

Expected result: All three changes classified breaking with the mechanism named, a date-pinned version header with one internal model and ordered downgrade transforms, a documented lost mapping, validation tightened only for versions after the change, and a sunset driven by per-version traffic with brownouts.

Verify it yourself

• No change is called additive without naming the client behaviour that makes it safe.
• A client that never changes its version header sees no behaviour change from any of the three.
• The plan names a measurement that tells you when sunset is safe.
• The lost mapping is stated once, in a table, not implied by serialiser branches.

Follow-up

• Which step of your rollout cannot be rolled back cleanly once a client has depended on it?
• A tenant pinned to a sunset version breaks their build the morning after the cutoff. What in your plan already lets you resolve it within an hour?
• Your transform chain is now six versions deep. What stops it becoming the fork you were avoiding?

One tenant's counter writes stall the whole connection pool

Medium · lock contention · connection pool · multi-tenancy

A change that made a per-tenant usage counter correct now produces site-wide latency whenever one large tenant writes: unrelated endpoints time out waiting for a connection while database CPU stays low and no statement is slow. The change wraps the counter update in a transaction that takes SELECT ... FOR UPDATE on one row, calls an external pricing service, then updates and commits. Give an ordered checklist, the arithmetic that bounds that tenant's write rate, and three repairs with the cost each one accepts.

Approach

1. Separate waiting from working. Low database CPU alongside high application latency points at a queue, so instrument connection-acquisition wait separately from query execution time; that queue forms in the application and is invisible in database metrics, which is why the database looks healthy throughout.
2. Confirm the lock rather than assuming it: sample waiting sessions and group by wait event, relation and tuple. Contention concentrated on one tuple belonging to one tenant is the signature; a deadlock would instead show the database aborting transactions after its detection timeout, which is not happening here.
3. Do the arithmetic out loud. Throughput on a serialised row is one divided by the lock hold time, and the hold spans the external call, so a 20 ms pricing call caps that tenant near 50 writes per second no matter how many pods run. Every waiter also holds a pooled connection while it queues, so the shared pool drains and unrelated tenants fail at acquisition.
4. Repair one: shrink the critical section to a single statement with the price resolved before the transaction opens. Cost is a stale price for the duration of one request and a second round trip; benefit is a hold time measured in the database's own execution time.
5. Repairs two and three change where the contention lives rather than how long it is held. Sharding the counter into per-(tenant, bucket) rows and summing on read multiplies write throughput by the shard count, at the cost of an aggregate on every read and a shard count you must size against the largest tenant rather than the median. Accumulating in memory and flushing periodically removes the per-write round trip entirely, paid for with a bounded loss window on crash, which is acceptable for a rate limiter and not for a billing counter.
6. Contain independently of which repair wins: a separate pool or per-tenant concurrency cap for this write class, a statement timeout low enough that a pathological query dies before it accumulates waiters, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks.

Follow-up

• What would a genuine deadlock look like here, which two code paths would produce one, and how does the database's response differ from what you observed?
• If a transaction-pooling proxy sits in front of the database, which of your three repairs changes behaviour, and what stops working that would have worked on a direct connection?
• The counter also enforces a quota. Why is SELECT the count and then INSERT still wrong after you have fixed the contention?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

• Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
• Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
• Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

• Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
• Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
• Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

• Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
• Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
• Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

• Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
• Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
• Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

• Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
• Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
• Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

• Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
• Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
• Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

• Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
• Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
• Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

How does your experience with Python translate to building scalable blockchain infrastructure?

Medium · behavioural and engineering judgement

Approach

1. State the situation in two sentences and spend the rest on the reasoning.
2. Pick a story where you made the decision, not one where you watched it.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• What did you decide not to do, and why?
• How did you know your change caused the improvement?

How do you handle failure modes in a distributed network?

Medium · behavioural and engineering judgement

Approach

1. Close with what you would do differently, concretely.
2. Name the disagreement and how you resolved it with evidence.
3. Pick a story where you made the decision, not one where you watched it.

Follow-up

• How did you know your change caused the improvement?
• What would you do differently if you ran that again?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

1. Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
2. Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
3. Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
4. Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
5. State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
6. Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

• Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
• The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
• You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?