Interview guides · Aerospike

Aerospike Software Engineer Interview Questions & Guide 2026

Prepare for Aerospike engineering interviews with a tested log-backed key-value engine, version checks, index budgets, restart recovery and timeout reasoning.

The 60-second brief

Aerospike builds a real-time database platform. For an engine seat, prepare to explain record indexing, storage and concurrent updates; verify product guarantees in current official documentation.

Use the new original log-backed store first. Match client, cloud or engine depth to the actual requisition. Existing billing, SQL, interval and delivery exercises remain secondary transferable practice.

Focus areas

Locate records through index offsets and explicit versions
Reject stale mutation and full-index admission without data loss
Explain recovery, unknown outcomes and durability boundaries

Overview

Aerospike software engineer interview preparation should start with the work the role owns. For a database-engine seat, a useful answer goes below a generic distributed-system diagram: how an index locates a record, what a concurrent update can overwrite, when a write becomes visible and what restart recovery actually reconstructs. This guide develops an original log-backed key-value engine and a checkable failure trace. It retains the existing practice prompts and worked SQL, interval and delivery exercises as transferable material. The new code is a small Python teaching model, not Aerospike’s implementation or a private assessment.

For transferable storage practice, use Design a key-value store. This verified Databricks-tagged record has a broader POSIX and power-loss contract than the new local Windows teaching engine. Its use here is a preparation recommendation, not an Aerospike interview claim.

Aerospike preparation checkpoints: confirm your actual loop

The retained stages are editorial preparation checkpoints, not a verified universal Aerospike loop. Confirm actual sequence, duration, assessment language and permitted tools with recruiting.

Recruiter Screen

Identify whether you would work on engine internals, clients, cloud operations or integrations. Ask what the team owns, how it handles on-call and which assessment environment you will use. Bring one truthful example that fits the actual seat.

What to demonstrate

Match a past contribution to the role rather than claiming every requirement.
State unfamiliar areas and the closest relevant work.

How to prepare

Confirm language, stages, tool rules and any existing-repository work.
Write one concrete question about the team’s storage or operational scope.

Technical Discussions

Explain how an index maps a key to data, what versions protect against and which allocations a memory estimate includes. Use the original engine’s logical budget as a small example, then distinguish it from Aerospike’s configured primary-index representation.

What to demonstrate

Name index fields and read/write boundaries.
Separate bytes accounted from actual process memory.

How to prepare

Trace a lookup and conditional mutation, including deleted state.
Read the current primary-index and transaction documentation; do not infer product semantics from the toy code.

Coding Assessments

Implement a bounded request contract and make the expected-version check part of the same critical section as persistence and index publication. Demonstrate a stale writer, a full index and an incomplete final frame. Explain what happens when synchronization fails.

What to demonstrate

Finish executable operations and test the failing boundary.
Describe serialization and its throughput cost.

How to prepare

Run the new log-backed-store exercise and four-thread increment test.
Explain why an index entry cannot be treated as an evictable cache value.

System Design Challenges

Extend the single-node engine only after agreeing on workload, consistency and failure scope. Draw the ordered persistence path, compaction installation and reader lifetime. For a distributed design, add partition ownership, replication and migration without claiming the toy’s lock proves cluster behavior.

What to demonstrate

Keep visibility, persistence and acknowledgement separate.
State which requirements are modeled and which need another mechanism.

How to prepare

Write a crash-point table with complete frames, partial tail and missing reply.
Defend a tombstone-retention rule and durable file-set installation before deleting old segments.

Behavioral Interviews

Rehearse a real memory, performance or reliability incident. Explain what you observed, what discriminating probe you ran, what you personally changed and what remained uncertain. Qualify latency percentiles by hop, load and measurement window.

What to demonstrate

Use evidence and truthful ownership rather than a synthetic story as experience.
Explain detection, communication, mitigation and follow-up.

How to prepare

Recover real measurements or mark estimates clearly.
Explain one rejected alternative and the evidence that would have changed your decision.

Final Round Discussions

Prepare for follow-ups that change the read/write ratio, durability contract or role scope. Revisit the assumptions rather than repeating the same architecture. For client work, discuss uncertain outcomes and operation identity; for engine work, discuss storage and version publication.

What to demonstrate

Adapt the mechanism when the guarantee changes.
Finish with an observable test or decision.

How to prepare

Run one changed-requirement mock.
Ask the recruiter which final conversations apply to your actual process.

Where candidates lose points

Treating the primary index as an evictable cache

An index entry is the path to a stored value. Dropping it to admit a new key makes a live record unreachable in this model. Reject new-key overflow before append; retain deleted-key versions and distinguish logical charges from actual process memory.

Treating a missing write response as rollback

The four process-exit tests recover put/delete after synchronization without any successful reply. Reconcile authoritative state and use operation identity where required. A version check prevents stale overwrite, but it does not identify your operation after later writers have changed the key.

Calling a process-restart test proof of power-loss durability

Incomplete tails, full-frame corruption, directory entries, compaction installation and storage-device persistence are different boundaries. State which ones your implementation and tests cover. This fixture does not establish production power-loss durability.

Reporting a performance number without its measurement boundary

Specify workload, concurrency, key/value sizes, read/write mix, percentile and measured hop. Separate index traversal, lock wait and storage time before choosing a repair. The code’s correctness tests provide no throughput or latency benchmark.

Questions & practice

Build a log-backed key-value engine with version-checked writes

Hard · primary-index · conditional-update · append-log · crash-recovery

Original practice: one owning instance, multiple threads, 1–64-byte UTF-8 string keys and 0–1024-byte immutable values. Track file offsets, global mutation versions and tombstones. Use a 130-byte logical index budget for keys a/b. Reject stale expected versions and new-key overflow before append; reconstruct the index on restart. This is a local Python process-recovery model, not Aerospike or a power-loss-safe engine.

Approach

Write the API contract and distinguish primary-index capacity from cache eviction.
Serialize expected-version check, append, file sync and index publication under one lock.
Rebuild complete sequential frames; truncate only incomplete tail and reject complete checksum damage.
Recover after unknown outcomes and explain missing compaction, directory durability, file ownership and replication.

Worked solution (40 min)

Match preparation to the engineering seat Ask whether the role concerns the database engine, a language client, cloud operations, integrations or another product surface. Confirm coding language, interview stages, repository work and tool rules with recruiting. The reviewed careers page does not establish a universal stage count or timeline. For engine work, practice allocation, indexing, concurrent access, persistence and diagnostics. A client role also needs retries, deadlines and API compatibility. Cloud or operations work needs rollout, capacity, failure isolation and recovery. Those are editorial study routes; the actual requisition controls the balance. Aerospike’s current primary-index documentation describes record metadata and data-location information. The default index is in memory, with other options depending on edition and configuration; do not state that every Aerospike deployment keeps all data or indexes in RAM. The documentation also describes partitioned index structures and independently lockable sprigs. Our single dictionary and lock illustrate much smaller design boundaries.
Storage API and limits One owning instance serves multiple threads. Keys are strings occupying 1–64 UTF-8 bytes; values are immutable bytes of length 0–1,024. get returns (value, version), with (None,0) for a never-seen key. mutate(key,None) records a deletion. Versions are a store-wide monotonically increasing sequence, not Aerospike’s per-record generation representation. An optional expected version makes a write conditional. Zero creates a never-seen key; a mismatching version returns False without appending. Each successful mutation receives a new sequence. Deleted keys retain their latest version so stale readers do not confuse deletion and recreation with a new, never-seen key. The logical primary-index budget charges 64 bytes plus the UTF-8 key length for every known key, including tombstones. This is a teaching accounting rule, not a measurement of Python or an Aerospike capacity estimate. Python dictionary objects, the file frame, decoded values and other allocations remain outside that charge. The engine has no hard process-RAM cap.
Separate the index from stored values The in-memory dictionary stores the latest record’s file offset, sequence and deletion flag. It does not retain every value. A read locates the frame, checks its integrity and returns the value. Point lookup is expected constant time in the index, plus bounded record decoding and disk access; it is not a claim of constant end-to-end latency. Each append contains a four-byte big-endian body length, a compact JSON body and a four-byte CRC32. The body records sequence, operation, key and base64-encoded value. CRC detects accidental damage in the modeled frame; it is not cryptographic authentication. Bound the length before allocating or parsing a frame. The primary index is authoritative for finding every known key. Silently evicting a live index entry to admit another would make a stored value unreachable until a rebuild. This engine rejects a new key when the logical budget is full. Overwrites and deletes of known keys do not need another slot; a deleted slot remains charged.
Runnable log-backed key-value engine Save the following as tiny_log_kv.py. It uses Python’s standard library. Create a store with TinyLogKV('practice.log'), call its operations and close it before opening another owner of the same path. No process-level file lock is implemented. The complete tested code is provided in this worked exercise’s code panel. The single lock covers the version check, append, file synchronization and index update. Reads take that lock too. That creates a simple serial execution order among healthy calls on this one instance, including a read immediately after a successful write. It also makes a slow disk synchronization block unrelated reads; concurrency correctness does not imply high throughput. The implementation handles partial writes by continuing until the frame is fully written. It synchronizes the file before applying the new index entry and returning success. If writing, synchronization or the teaching hook fails, it marks the instance unusable. Close and reopen; do not continue serving from an index that may disagree with the log.
Work the index and version trace Each one-byte key costs 65 logical bytes. Put a=AA at version 1, b=BB at 2, then a=AAA with expected 1 at 3. A stale expected 1 returns False without append. Delete b with expected 2 at 4; its tombstone remains charged. New c is rejected at the 130-byte index limit. Restart recovers a=AAA/version 3 and b=None/version 4; recreate b=B with expected 4 at 5, still 130 logical bytes.
Recover complete records without hiding corruption On startup, scan frames in order and rebuild the latest offset/version map. Require consecutive sequences and validate record fields, operation, key/value limits and checksum. Work done during recovery is proportional to the retained log, including old versions; it does not become proportional only to the live key count. An incomplete final header or body is a truncated tail. This fixture truncates that tail to the preceding complete frame. A complete frame with a bad checksum or invalid fields fails startup instead of silently treating corruption as a missing final write. The distinction matters when a damaged record may have been acknowledged earlier. After a file-sync error, the write’s outcome is unknown. A complete record can survive even when the caller saw an exception. The test injects such an error and observes the new record on reopen; that local observation is not a guarantee of what survives a machine crash. The process-exit tests stop after synchronization and immediately before reply, for both put and delete. All four recover the new operation despite there being no successful response. A lost reply is not evidence of rollback. Version checks prevent stale overwrite, but they do not replace request-ID deduplication when a client needs to identify its exact operation among later writes. The code does not synchronize a new file’s parent directory, perform atomic manifest installs, compact data or test storage-device failure. It must not be presented as proof that every acknowledged write survives power loss. A production durability argument needs filesystem, directory-entry, device and recovery semantics beyond the tested process boundary.
Test the boundaries that change the guarantee The local review passed 3,240 checks. Ten seeded 100-operation histories were compared with an independent dictionary/version oracle and reopened periodically. Four concurrent workers performed 100 version-checked increments each; the final value was 400 with version 401 and 400 distinct successful increment versions. The review also tried 52 cuts within a real final frame, four actual child-process exits, complete-frame checksum corruption and an injected synchronization failure. Boundary cases covered 64-byte UTF-8 keys, 1,024-byte values, oversized requests, empty keys, invalid expected versions and rejecting new keys before modifying a full index’s log. These are local correctness checks, not Aerospike server tests or a benchmark. No network partition, replica promotion, multi-process writer, power failure, compaction or hard process-memory limit was exercised. State those boundaries before using the result to defend a larger design.
Extend the design without changing semantics silently For higher concurrency, consider per-partition synchronization and an ordered persistence path. A key check and its publication must still refer to the same version. Group commit can amortize synchronization, but acknowledge each operation only after the batch containing it meets the stated persistence contract; a periodic flush changes the possible loss window. Compaction introduces another recovery problem. A replacement segment must retain the correct latest values and deletion state, and a durable metadata switch must select a complete consistent file set. Removing a tombstone while an older value remains reachable can resurrect a deleted key. Define safe reclamation for readers of old files. None of those mechanisms is in the code above. For a distributed follow-up, choose the consistency contract before naming replication. Explain ownership during migration, how conditional updates are ordered, what happens when quorum cannot be obtained and which timeout outcomes remain uncertain. Aerospike’s transaction documentation includes in-doubt commit handling; SDK and transaction-specific rules must come from those docs, not from this teaching engine.
Keep the existing drills in their proper place The guide’s original 14 prompts and three worked exercises remain available. Prioritize the key-value-store prompt, consistency reasoning and memory/performance debugging for an engine role. Use the credential-history SQL exercise for schema and index reasoning, the sandbox interval exercise for precise resource accounting, and delivery/replay design for retry guarantees. Billing reconciliation, invoice joins and webhook signatures are transferable correctness drills. They do not establish that an Aerospike database-engine interview asks about billing or SaaS control planes. Choose them when your actual role or a weakness makes them relevant rather than spending the entire week there. For debugging, form a discriminating experiment. If p99 rises while mean latency stays steady, separate lock wait, storage latency, allocation pressure and workload mix before changing code. An overwritten value needs a trace of observed version, expected version and committed sequence. More logging can alter a race, so include deterministic synchronization or invariant checks in a reduced reproduction.
A week of artifacts you can defend Start by confirming the role and writing the storage contract. Next implement the trace and stale-version rejection. Use the third day for logical memory limits and the fourth for restart recovery and crash points. On day five, discuss compaction and the unimplemented production durability boundaries. Use day six for a role-relevant original drill and a truthful Linux/performance project story. Finish with a mock that changes the read/write ratio or introduces a replica failure. In a project story, identify your own code and decisions, the measurement window, the failure evidence and what remained unresolved. If you have not written a database engine, explain the closest relevant system honestly and use this exercise as preparation, not as claimed employment experience.

import base64, json, os, struct, threading, zlib


class TinyLogKV:
    """One owning instance; logical index budget, not a process RAM cap."""
    MAX_FRAME = 8192

    def __init__(self, path, index_budget=130, hook=lambda phase: None):
        self.lock = threading.RLock()
        self.index, self.used, self.seq = {}, 0, 0
        self.budget, self.hook, self.poisoned = index_budget, hook, False
        self.file = open(path, 'r+b' if os.path.exists(path) else 'w+b', buffering=0)
        try:
            while True:
                offset = self.file.tell()
                record = self._read()
                if record is None:
                    break
                if record['seq'] != self.seq + 1:
                    raise ValueError('nonconsecutive sequence')
                self._apply(record, offset)
        except EOFError:
            try:
                self.file.truncate(offset)  # Only an incomplete final frame.
                self.file.flush(); os.fsync(self.file.fileno())
            except BaseException:
                self.file.close()
                raise
        except BaseException:
            self.file.close()
            raise
        self.file.seek(0, 2)

    def _read(self):
        header = self.file.read(4)
        if not header:
            return None
        if len(header) != 4:
            raise EOFError('incomplete header')
        size = struct.unpack('>I', header)[0]
        if not 1 <= size <= self.MAX_FRAME:
            raise ValueError('invalid frame size')
        body, crc = self.file.read(size), self.file.read(4)
        if len(body) != size or len(crc) != 4:
            raise EOFError('incomplete final frame')
        if zlib.crc32(body) != struct.unpack('>I', crc)[0]:
            raise ValueError('checksum mismatch')
        record = json.loads(body)
        if set(record) != {'seq', 'op', 'key', 'value'}:
            raise ValueError('invalid fields')
        if type(record['seq']) is not int or record['seq'] < 1:
            raise ValueError('invalid sequence')
        self._key(record['key'])
        if record['op'] not in ('put', 'delete'):
            raise ValueError('invalid operation')
        value = base64.b64decode(record['value'], validate=True)
        if len(value) > 1024 or (record['op'] == 'delete' and value):
            raise ValueError('invalid value')
        return record

    def _key(self, key):
        if not isinstance(key, str) or not 1 <= len(key.encode('utf8')) <= 64:
            raise ValueError('key must occupy 1..64 UTF-8 bytes')

    def _apply(self, record, offset):
        key = record['key']
        if key not in self.index:
            charge = 64 + len(key.encode('utf8'))
            if self.used + charge > self.budget:
                raise MemoryError('logical index budget exceeded')
            self.used += charge
        self.index[key] = (offset, record['seq'], record['op'] == 'delete')
        self.seq = record['seq']

    def _usable(self):
        if self.poisoned or self.file.closed:
            raise RuntimeError('close and reopen after an uncertain write')

    def get(self, key):
        self._key(key)
        with self.lock:
            self._usable()
            offset, version, deleted = self.index.get(key, (0, 0, True))
            if deleted:
                return None, version
            self.file.seek(offset)
            record = self._read()
            if record is None or record['key'] != key or record['seq'] != version:
                raise ValueError('index and log disagree')
            return base64.b64decode(record['value']), version

    def mutate(self, key, value, expected=None):
        self._key(key)
        if value is not None and (type(value) is not bytes or len(value) > 1024):
            raise ValueError('value must be bytes of length at most 1024, or None')
        if expected is not None and (type(expected) is not int or expected < 0):
            raise ValueError('expected version must be a nonnegative integer')
        with self.lock:
            self._usable()
            version = self.index.get(key, (0, 0, True))[1]
            if expected is not None and expected != version:
                return False
            charge = 0 if key in self.index else 64 + len(key.encode('utf8'))
            if self.used + charge > self.budget:
                raise MemoryError('logical index budget exceeded')
            record = {'seq': self.seq + 1, 'op': 'delete' if value is None else 'put',
                      'key': key, 'value': base64.b64encode(value or b'').decode()}
            body = json.dumps(record, separators=(',', ':')).encode()
            frame = struct.pack('>I', len(body)) + body + struct.pack('>I', zlib.crc32(body))
            offset = self.file.seek(0, 2)
            try:
                view = memoryview(frame)
                while view:
                    written = self.file.write(view)
                    if not written:
                        raise OSError('no write progress')
                    view = view[written:]
                self.file.flush(); os.fsync(self.file.fileno())
                self.hook('after_sync')
                self._apply(record, offset)
                self.hook('before_reply')
            except BaseException:
                self.poisoned = True
                raise
            return self.seq

    def close(self):
        with self.lock:
            self.file.close()

Expected result: Trace: a=AAA/version 3, b=None/version 4 after deletion and restart; b=B/version5 after recreation. Logical known-key charge stays 130 bytes. Stale expected-version write and new-key admission beyond capacity make no append. Complete persisted records can survive without a reply; reopen after uncertain write. Single lock serializes healthy operations and blocks reads during file sync.

Verify it yourself

3,240 actual local checks passed with 10 seeded 100-operation oracle histories and periodic restart.
Four threads × 100 successful conditional increments: value 400, version 401; no lost update.
52 incomplete-tail cuts recover the preceding value; complete CRC damage rejects startup without silent truncation.
Four process exits after sync/before reply recover put/delete even though no success response arrived.
Key/value/version boundaries and new-key capacity rejection leave the log unchanged.
Scope excludes power loss, directory durability, compaction, multi-process ownership, replication and benchmarks.

Follow-up

Can a failed synchronization still leave a complete record on reopen?
How can a group-commit design keep visibility and acknowledgement semantics?
How do you prevent resurrection when compacting old values and tombstones?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Follow-up

The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
Ingest is still running while you investigate. What makes your two numbers comparable at all?

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Worked solution (20 min)

Write the null policy as three lines of prose first, one per case, and keep them beside the output.
Emit 2n endpoint tuples (timestamp, delta, run_id) and sort on the key (timestamp, delta) so -1 precedes +1.
Sweep, tracking cur, peak, peak_at updated only on a strict increase, and the first run_id whose +1 takes cur to C+1.
Build a fixture with two runs where one ends exactly when the next starts, three genuinely overlapping runs, one run with a null finished_at and status running, and one with status lost and a 300-second wall_clock_limit_seconds.
Re-run with every timestamp shifted by a constant and confirm the peak is unchanged while peak_at shifts by the same constant.

Expected result: Peak is 3, from the overlapping trio. The exact-handoff pair yields a peak of 1, not 2. peak_at is the start instant of the third overlapping run. The lost run occupies exactly [started_at, started_at + 300s) under the stated policy.

Verify it yourself

Setting peak_at on >= rather than > moves the reported instant to the last time the peak was attained. Flip it and watch the fixture change.
Sorting by timestamp alone, without the delta tie-break, raises the reported peak on the handoff fixture.
The deltas over the whole sweep sum to zero once every open-ended run has been clipped.

Follow-up

Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Model credential revocation so history survives the delete

Easy · soft delete · partial indexes · check constraints · audit history

tenant_api_key stores key_id, tenant_id, workspace_id, name, key_prefix, secret_hash, scopes text[], status (active, revoked, expired, compromised), auth_version, created_at, expires_at, last_used_at, revoked_at, revoked_reason. Rotation inserts a new row and revocation never deletes, because an incident review asks which credential served a request last quarter. Write the constraints that enforce: a label is unique only among a tenant's live keys, revoked_at and status can never disagree, and scopes is never empty. Then write the authentication lookup predicate, and name one column in this table that must stay out of it.

Approach

Reach for a partial unique index rather than a plain UNIQUE: create unique index on tenant_api_key (tenant_id, name) where revoked_at is null. Any number of revoked rows may share a label, the live namespace stays unique per tenant, and the revoked majority is not in the index at all, so it stays small on a table that only grows.
Tie the nullable timestamp to the enum so the two cannot drift: check ((revoked_at is not null) = (status in ('revoked','compromised'))) and check ((revoked_at is null) = (revoked_reason is null)). A revocation that records no reason is the one an incident review cannot use.
Write the emptiness check as check (cardinality(scopes) > 0), not array_length(scopes, 1) > 0. array_length returns NULL for an empty array, a CHECK constraint passes when its expression is NULL, so the array_length version accepts exactly the value it was written to reject.
Make the lookup a single index probe with every liveness condition inside it: where secret_hash = $1 and revoked_at is null and (expires_at is null or expires_at > now()) and auth_version = $2, backed by a unique index on secret_hash. Nothing is filtered in application code, so there is no path that forgets a clause.
Keep last_used_at out of that predicate. It is written asynchronously and is allowed to lag by a minute, so it is a usage signal; feeding it into an authorisation decision makes the decision depend on a write that may be late, batched away or lost.
Flag the modelling smell while you are here: expired is derivable from expires_at < now(), so storing it as a status obliges a job to keep it true and guarantees the column is wrong between the expiry instant and that job's next run. Derive it in the predicate; keep the stored status for states that are decisions rather than clock readings.

Worked solution (20 min)

Create the table with all three constraints on a scratch database and insert two revoked rows sharing (tenant_id, name); the partial index should accept both.
Insert a second live row with that same name and confirm the violation names the partial index.
Run update tenant_api_key set revoked_at = now() leaving status = 'active' and confirm the CHECK rejects it; then try insert ... scopes = '{}' against both the cardinality and the array_length forms and note that only one rejects it.
Run explain (analyze, buffers) on the lookup predicate for a live key and confirm an index scan on secret_hash with rows removed by filter equal to zero.

Expected result: A partial unique index on (tenant_id, name) where revoked_at is null; check ((revoked_at is not null) = (status in ('revoked','compromised'))); check ((revoked_at is null) = (revoked_reason is null)); check (cardinality(scopes) > 0); a single-probe lookup on secret_hash plus revoked_at, expires_at and auth_version; last_used_at excluded, with a note that 'expired' should be derived rather than stored.

Verify it yourself

Two revoked keys with the same name insert cleanly; a second live one does not.
The array_length form of the emptiness check accepts an empty array and the cardinality form rejects it.
No statement anywhere in the design deletes or updates a revoked row.
Revoking a key touches exactly one row and takes effect on the next lookup with no application-side filtering.

Follow-up

Rotation issues a replacement while the old key stays live for a 30-day overlap. What does the uniqueness rule become, and what does the UI show to tell two same-named keys apart?
A password reset bumps the principal's auth_version. No row in this table changed. How does the next request fail, and what query counts how many keys that bump just killed?
A key turns up in a public repository. Which columns let you find it, and what do you write to the row?

Find the join that inflates every invoice total

Medium · join fan-out · aggregation grain · sql correctness

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

Approach

Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Follow-up

Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
The same report is needed per tenant per month. What index makes the line CTE cheap?

Explain how you would ensure data consistency in a distributed system.

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Choose a partition key and say what query it makes expensive.
Name the failure you are designing for, then the recovery path.

Follow-up

How does this behave when that dependency is down for an hour?
What breaks first when traffic grows ten times?

Design an in-memory key-value data store from scratch.

Medium · system design and architecture

Approach

Define a one-owner contract, index entry, value bounds and conditional mutation before adding distribution.
Separate logical index accounting from process memory; never silently evict a primary-index entry.
Use the new worked exercise for file offsets, sequence checks, tombstones and restart recovery; name compaction and power-loss boundaries.

Follow-up

What happens when a deletion leaves a charged tombstone?
Which directory and metadata operations are required for a production power-loss argument?

How would you structure an API to handle high-frequency requests?

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Name the failure you are designing for, then the recovery path.
Choose a partition key and say what query it makes expensive.

Follow-up

How does this behave when that dependency is down for an hour?
What would you drop to keep the system up under load?

How do you approach debugging complex memory or performance issues?

Medium

Approach

Separate allocation, lock wait, storage latency and workload mix using a reduced trace and measured windows.
Use generation/sequence evidence to investigate overwritten values; adding logs can change the race.
State the observed system boundary and reproduce a violated invariant before changing synchronization.

Follow-up

How would you inspect lock-wait time separately from storage time?
What would falsify your current explanation?

Write the delivery guarantee and replay API for outbound events

Medium · webhooks · at-least-once · ordering

The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.

Approach

Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.

Worked solution (30 min)

Write the guarantee sentence and the one-sentence reason exactly-once is not available over HTTP.
List the headers a customer needs to build their own dedup table: event id, delivery id, attempt number, subscription id, signature and timestamp.
Write the response-code table with four classes and the action for each, including the 410 case.
Write the retry schedule with concrete numbers, the terminal condition, and the breaker's threshold and its effect on the delivery row's status.
Specify the redelivery request and response, and say which columns change and which do not.

Expected result: At-least-once with a stable event id, ordering offered only as an opt-in single-in-flight mode with head-of-line blocking stated, a four-class response table, a jittered capped schedule with a terminal state, and redelivery implemented as new rows at an incremented sequence carrying identical bytes.

Verify it yourself

The customer can deduplicate using headers alone, without parsing the payload.
Nothing in the contract requires the sender to know whether the customer committed.
A day-long outage has a defined end state rather than unbounded retries.
Redelivery leaves the original delivery rows readable as history.

Follow-up

A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?

A rare job-run overwrite that logging makes disappear

Hard · fencing tokens · leases · concurrency

About one job run in fifty thousand records billable_seconds matching no observed sandbox lifetime, and a few rows show worker_id changing after finished_at was already set. It does not reproduce: debug logging around the terminal write made it vanish for two weeks before it returned. Runs last from 200 ms to 30 minutes, the lease is 60 seconds and is renewed while a run executes. Give an ordered checklist, the mechanism, and a fix that makes the illegal write impossible rather than merely rarer.

Approach

Mine the evidence instead of chasing a repro: select rows where updated_at is later than finished_at, or where a terminal status was written twice, and join them to attempt history to recover both writer identities. The defect has already happened tens of times and the rows are the recording.
State the signature before measuring it. If the cause is a lease that expired while the original worker was stalled, affected runs should cluster where the gap between the last renewal and the terminal write exceeds the lease, and should correlate with worker pause metrics rather than with workload shape.
Read the disappearance honestly. Logging inside the window changed the timing and lowered the probability; it is evidence about how narrow the window is, not a fix. Reproduce by widening the window on purpose, shortening the lease and injecting a pause between sandbox exit and the terminal write, rather than by adding more instrumentation.
Name the mechanism precisely: the lease expires during a stall such as a long garbage-collection pause or a brief partition, the run is re-dispatched, and the original worker then wakes and writes its terminal state over the new attempt's row. A lease alone cannot stop this, because the check and the write are separated by the stall.
Fix by fencing the write itself: UPDATE job_run SET status = $2, finished_at = $3, billable_seconds = $4 WHERE run_id = $1 AND status = 'running' AND lease_token = $5, with zero rows affected interpreted as having been fenced rather than as success. The token lives on the row so the store arbitrates, not the worker's memory.
Keep the state machine honest: a retry inserts a new row pointing at parent_run_id rather than resetting the old one, and a run whose worker vanished terminates as lost with billable_seconds null, because recording failure asserts an outcome nobody observed and then bills and retries on that assertion.

Follow-up

The supervisor also emits a usage event on completion. What does the fenced worker do about the event it already emitted, and how does metering absorb it?
Lease renewal is itself a network call. What happens when a renewal times out, and how does the worker decide whether it still holds the lease?
Why is lengthening the lease past the longest legitimate run the wrong lever, and what breaks if you do it anyway?

Suggested practice checklist

Start with the log-backed engine; select client, cloud or engine follow-ups from the confirmed requisition. Existing SaaS/SQL drills are transferable secondary exercises.

Day 1 — Define role and storage scope

Confirm the actual engineering seat, language and assessment environment.
Define ownership, memory accounting, conditional write and acknowledgement contracts.

Deliverable: A role-specific storage contract.

Day 2 — Implement offsets and versions

Run the a/b update, stale write, delete and recreation trace.
Explain primary-index reachability and the difference from a value cache.

Deliverable: Checked version and index trace.

Day 3 — Bound requests and index admission

Test UTF-8 key size, value size and expected-version types.
Reject new keys before append when index capacity is exhausted.

Deliverable: Boundary checks and explicit memory limits.

Day 4 — Recover and inject failure

Rebuild the latest index from complete frames.
Compare incomplete tail, complete corruption and missing replies.

Deliverable: Crash-point table with untested power-loss scope.

Day 5 — Discuss production storage depth

Define compaction installation and tombstone retention.
Explain batching, sharding, read visibility and timeout outcomes.

Deliverable: A design with precise persistence and consistency boundaries.

Day 6 — Use a relevant original drill

Choose original SQL, intervals or delivery practice for your actual role.
Rehearse a truthful Linux/performance project story and measured evidence.

Deliverable: One retained drill plus an honest project explanation.

Day 7 — Defend a changed requirement

Change read/write ratio or introduce migration/replica failure.
Name which guarantees no longer follow from the local engine.

Deliverable: A revised design and a clear next experiment.

Explain one real storage, memory or reliability decision

Use a truthful project: the observed failure, your own contribution, the probe that changed the diagnosis and the measured result. State what teammates handled and what remained unresolved. A synthetic fixture is preparation material, not something to present as production experience.

What is your experience with Linux/Unix environments?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

Describe your approach to resolving a complex technical support case or production bug.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Walk me through a challenging project listed on your resume.

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?