Interview guides · Audible
Audible Software Engineer Interview Questions & Guide 2026
Concurrency drills for Audible software engineer prep: races, lock ordering, and what a check-then-act sequence assumes.
The 60-second brief
As a Software Engineer at Audible, you are at the intersection of high-scale distributed systems and world-class media consumption. You are not just writing code; you are building the infrastructure that delivers spoken-word entertainment to millions of listeners globally. Your work directly impacts how users discover, purchase, and experience audiobooks, podcasts, and original content.
Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.
Focus areas
- Enforce entitlement at the licence boundary
- Expire rights across cached and offline copies
- Key CDN caches without sharding on tokens
Overview
This role is both technically demanding and strategically significant. You will tackle complex challenges related to high availability, low-latency streaming, recommendation engines, and massive data processing. Whether you are optimizing a search algorithm for a vast library or designing a system to handle peak traffic during new releases, your contributions are core to the Audible mission. You will work within cross-functional teams, collaborating with product managers and other engineers to solve real-world problems that define the future of audio entertainment.
The Audible interview process
Audible candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Initial Screening
Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.
What to demonstrate
- Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
- Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
- Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled
How to prepare
- Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
- Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
- Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market
Technical Deep-Dives
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Cultural Fit Assessment
Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.
What to demonstrate
- Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
- Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
- Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
- Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply
How to prepare
- For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
- For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
- Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last
Final Decision
A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.
What to demonstrate
- Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
- Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
- Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
- Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing
How to prepare
- Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
- Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
- Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one
Where candidates lose points
Assuming one encrypted copy of the ladder serves every DRM system.
Common encryption (ISO/IEC 23001-7) defines several protection schemes, and two matter in practice: AES-CTR full-sample ('cenc') and AES-CBC pattern encryption ('cbcs'). One of the three widely deployed DRM systems accepts only cbcs, and older client versions of the other two accept only cenc, so a single-copy strategy is correct only if every device class you still support has reached cbcs. Discovering late that one legacy class has not means a second packaging run, a second set of objects in every cache, and roughly double the storage for the affected catalogue.
Letting the live playlist grow with the DVR window.
A sliding-window playlist carries one entry per segment, so a four-hour DVR at two-second segments is 7,200 entries - at roughly 100 bytes per entry, about 720,000 bytes - and every viewer refetches the document about once per target duration. At 100,000 concurrent viewers that is 720 kB x 100,000 every two seconds, roughly 288 Gbit/s of playlist text that is nearly identical between fetches. Delta playlist updates and byte-range addressing solve it, but they have to be designed in, not discovered once the playlist has become the largest object being served.
Quoting amortised or average cost as if it were a worst-case guarantee
Appending to a dynamic array is amortised O(1), but the append that triggers a resize copies every element, and hash lookup is constant only while the hash spreads the actual keys. Say which guarantee you are offering when the caller cares about the latency of one call rather than the total over many.
Retrying a write that is not safe to repeat
A timeout tells you nothing about whether the server applied the write, so a blind retry of a create or a charge can duplicate it. Either make the operation idempotent, with a caller-supplied key the server deduplicates on or a conditional update, or do not retry it; and use exponential backoff with jitter so the retries of many clients do not synchronise into a second outage.
Questions & practice
Implement a filesystem traversal using DFS.
Medium · data structures and algorithms
Approach
- Name the brute-force solution and its complexity before improving on it.
- Walk one small example through your approach before writing the whole thing.
- Choose the data structure from the access pattern, not from familiarity.
Follow-up
- How does this change if the input no longer fits in memory?
- What is the worst case, and how likely is it on real data?
Given a set of audiobooks with distinct lengths, find the best combination for a playlist.
Medium · data structures and algorithms
Approach
- Name the brute-force solution and its complexity before improving on it.
- State the target complexity and say which constraint rules the naive version out.
- Walk one small example through your approach before writing the whole thing.
Follow-up
- Which test case would catch an off-by-one here?
- What is the worst case, and how likely is it on real data?
Find all combinations of two audiobooks that sum to a given target value (and the follow-up for four books).
Medium · data structures and algorithms
Approach
- Walk one small example through your approach before writing the whole thing.
- State the target complexity and say which constraint rules the naive version out.
- Choose the data structure from the access pattern, not from familiarity.
Follow-up
- What is the worst case, and how likely is it on real data?
- How does this change if the input no longer fits in memory?
Number of Islands: Given a grid, count the number of islands.
Medium · data structures and algorithms
Approach
- Choose the data structure from the access pattern, not from familiarity.
- Walk one small example through your approach before writing the whole thing.
- State the target complexity and say which constraint rules the naive version out.
Follow-up
- How does this change if the input no longer fits in memory?
- Which test case would catch an off-by-one here?
Alert on a rolling rebuffer ratio over a time-based window
Medium · sliding window · two pointers · alerting
A session's QoE samples arrive in playback order as (sequence, wall_clock_ms, wall_delta_ms, rebuffer_ms). Find the earliest sample at which the rebuffer ratio over the trailing 120,000 ms of wall time exceeds 5%, ignoring any window covering under 30,000 ms of wall time. Sample spacing varies between 2 and 30 seconds depending on device and player version, and a long session can carry 10^5 samples. Return the sample index or none. Target O(n) time and O(1) additional space.
Approach
- Two indices over the time-ordered array with running integer sums of wall_delta_ms and rebuffer_ms. Advance the right index one sample at a time, then pull the left index forward while the window's wall span exceeds 120,000 ms, subtracting as you go. Each index only moves forward, so total work is O(n) with two accumulators and no auxiliary structure.
- Define the window by accumulated wall time, not by sample count. Spacing varies by more than 10x, so a fixed count of samples is a window somewhere between four minutes and an hour depending on which device reported it.
- Gate the predicate on a minimum covered span of 30,000 ms. Without it, the first two samples of any session containing one stall produce a window at or near 100% and every session with a startup stall pages someone.
- Name the naive alternative and its cost: recomputing the sums across the window at each step is O(n * w), roughly 6 x 10^6 operations at 10^5 samples and a 60-sample window. Acceptable for one session on demand, not acceptable when the same evaluation runs against 10^6 concurrent sessions.
- Keep both accumulators as int64 milliseconds so add-then-subtract is exact; a float accumulator drifts over 10^5 add/subtract pairs and the drift is largest on exactly the longest sessions.
Worked solution (20 min)
- Write the two-pointer loop with both accumulators and the eviction condition on wall span.
- Add the minimum-coverage gate and show which sessions it suppresses.
- Argue the O(n) bound from the fact that each index advances monotonically.
- Trace a session with one 5-second stall followed by 10 minutes of clean playback and confirm the alert clears once the stall leaves the window.
Expected result: The index of the first sample whose trailing 120,000 ms window covers at least 30,000 ms and exceeds a 5% rebuffer ratio, found in a single forward pass with two int64 accumulators and O(1) extra space.
Verify it yourself
- Doubling every wall_delta_ms without changing rebuffer_ms halves the reported ratio.
- A session with one early stall and ten clean minutes does not alert after the stall ages out.
- On a 200-sample input the answer matches a brute-force recomputation of every window.
Follow-up
- Evaluate it incrementally as samples stream in with no array retained. What state do you keep per session, and what is its memory bound at 10^6 concurrent sessions?
- A late sample arrives out of order after the window has already moved past its timestamp. Do you re-evaluate, and what does that do to the idempotency of the alert?
Make a monthly payout scan use an index at two billion rows
Medium · sargability · covering index · partitioning · query planning
qualified_play(id BIGSERIAL PK, session_id, content_version_id, rights_holder_id, account_id, territory_code, qualified_at TIMESTAMPTZ, status ENUM('pending','counted','reversed'), source ENUM('online','offline_sync'), dedupe_key UNIQUE, counted_in_period) holds about two billion rows, roughly 150 million per month, and is physically ordered by insert time. The monthly close runs SELECT rights_holder_id, count(*) FROM qualified_play WHERE date_trunc('month', qualified_at) = DATE '2026-08-01' AND status = 'counted' GROUP BY 1 and scans the entire table. Give three separate reasons it does, and the change you would make for each.
Approach
- Reason one is the expression. Wrapping the column in
date_truncmakes the predicate non-sargable, so no b-tree onqualified_atis usable and the planner also falls back to a default selectivity guess instead of the column's histogram. Rewrite to the half-open rangequalified_at >= DATE '2026-08-01' AND qualified_at < DATE '2026-09-01', which selects the same rows. An expression index is the alternative, but in PostgreSQLdate_trunc(text, timestamptz)is STABLE rather than IMMUTABLE because it depends on the session TimeZone, so it cannot be indexed directly — you would indexdate_trunc('month', qualified_at AT TIME ZONE 'UTC')and write the predicate in exactly that form. - Reason two is selectivity, and the fixed predicate does not remove it. One month is about 7.5% of the table; at roughly 50 rows per page the table is about 40 million pages, and 150 million rows scattered across it touch essentially every page. A plain index scan would pay random I/O for the whole table while a sequential scan reads it in order — the planner is right. Make the scan index-only instead:
(status, qualified_at, rights_holder_id)covers every column the query touches, so the plan reads index pages alone. That holds only while the visibility map is current, which makes it a dependency on autovacuum keeping up with continuous inserts. - Reason three is structural.
status = 'counted'is most of the table and is not selective by itself, but a partial indexWHERE status = 'counted'keeps reversed and pending rows out of the index entirely. At two billion rows the real fix is range partitioning byqualified_atmonth: the close reads one partition, the planner prunes the rest, and the partition boundary lines up withcounted_in_periodso a period close and a partition are the same object. - Name the BRIN trap before someone reaches for it. BRIN is the obvious cheap index for an append-ordered table and it is tight on the arrival timestamp — but
qualified_atis when the threshold was crossed, andoffline_syncrows land days later, so physical order correlates with arrival rather than withqualified_at. Checkpg_stats.correlationfor the column instead of assuming; poor correlation makes every block range a candidate and the index worthless. - Settle it with EXPLAIN (ANALYZE, BUFFERS) rather than argument, and look at estimated versus actual rows for the range predicate first. A plan that goes wrong at this size usually went wrong at the estimate, and a wrong estimate has a different fix from a wrong index.
- Sequence the work by cost and risk: rewrite the predicate first because it is free and takes no lock, measure, then build the covering index with CREATE INDEX CONCURRENTLY, and only then consider partitioning, which is a migration of the whole table rather than an addition to it.
Follow-up
- The close also needs plays that qualified in August but arrived in September. Which of your changes still works, and which partition does a late row live in?
- The same aggregate is now wanted per territory as well. Do you widen the covering index or accept the scan?
- How would you prove the scan really is index-only in production rather than on a freshly vacuumed copy?
Add a dedupe constraint to a live plays table
Hard · online migration · lock queueing · backfill · unique constraints
qualified_play holds about two billion rows, takes continuous inserts, and was created without the dedupe_key column the design calls for. Add the column, backfill it deterministically from session_id and content_version_id, and make it unique and NOT NULL without taking the table offline. Duplicate plays already exist and money was paid against them, and corrections are written as reversing rows that share the same logical key. Give the ordered plan, name the lock each step takes, and say what the uniqueness is actually on.
Approach
- Deploy the write path before the backfill. Ship the code that populates
dedupe_keyon new rows first so the backfill covers a finite set — everything older than that deploy — instead of chasing a table that keeps growing new null rows behind it. - Add the column as nullable with no default. A nullable column with no default has never rewritten the table; since PostgreSQL 11 a constant default does not either, but a volatile default still does. The statement takes ACCESS EXCLUSIVE briefly, and the real hazard is the queue: the ALTER waits behind a long analytics scan of this table, and every query arriving after it waits behind the ALTER. Take it with
SET lock_timeout = '2s'and retry, so a failure to get the lock is a no-op rather than a stall of the read path. - Backfill in bounded batches over the primary key range, committing per batch. Each UPDATE writes a new row version, so the backfill produces roughly a table's worth of dead tuples before autovacuum reclaims them; throttle to the rate autovacuum can keep up with and watch the dead-tuple count rather than wall-clock progress. Never hold one long transaction over the whole backfill — it pins the transaction horizon and blocks cleanup everywhere.
- Decide what the uniqueness is on before building anything. A reversing row carries the same logical key as the row it reverses, so a plain unique on
dedupe_keywould make corrections impossible. Either make the index partial — unique ondedupe_keyWHERE status IN ('pending','counted')— or fold a revision into the stored key. Say which, and say what a re-count after a reversal then looks like under your choice. - Resolve the pre-existing duplicates with a window function rather than a DELETE:
ROW_NUMBER() OVER (PARTITION BY dedupe_key ORDER BY qualified_at, id)identifies every row past the first, and each gets a reversing row. Deleting them would erase rows a payout already ran against; the ledger has to show the correction, not lose the evidence. - Build with
CREATE UNIQUE INDEX CONCURRENTLY, which takes SHARE UPDATE EXCLUSIVE and does not block reads or writes, at the cost of two table passes and waiting on open transactions. It fails if any duplicate remains, and a failed build leaves an index withindisvalid = falsethat must be dropped before retrying. Promote it withALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING INDEX, a brief ACCESS EXCLUSIVE that does not rebuild. For NOT NULL, addCHECK (dedupe_key IS NOT NULL) NOT VALID, runVALIDATE CONSTRAINTunder SHARE UPDATE EXCLUSIVE, thenSET NOT NULL, which from PostgreSQL 12 uses the validated check instead of scanning the table again.
Worked solution (40 min)
- Build a scaled copy with a known duplicate population and a writer process inserting continuously throughout.
- Hold a long SELECT open, then run the ADD COLUMN with
lock_timeoutset, and observe what queues behind what. - Backfill in batches, recording dead tuples and table size after each batch.
- Enumerate duplicates with ROW_NUMBER and write reversing rows for every row past the first — leaving one duplicate in place deliberately.
- Run CREATE UNIQUE INDEX CONCURRENTLY, observe the failure, and inspect
pg_index.indisvalidfor the leftover. - Drop the invalid index, reverse the last duplicate, rebuild, promote to a constraint, and set NOT NULL through the NOT VALID check path.
Expected result: No writer statement blocks for longer than the lock_timeout at any step. The first concurrent build fails and leaves an index with indisvalid = false that must be dropped rather than retried over. After the duplicates are reversed the build succeeds, promotion adds the constraint without a rebuild, and NOT NULL is set without a second full scan.
Verify it yourself
pg_index.indisvalidis true for the final index and no invalid index remains behind.- The writer process logs no errors and no statement waiting longer than the configured lock_timeout.
- Every pre-existing duplicate has a reversing row, and the count of rows in status
pendingorcountedperdedupe_keyis exactly 1. - The SET NOT NULL step finishes in far less time than a full scan of the table takes, confirming it used the validated check.
Follow-up
- The backfill is half complete when a bug in the key derivation is found. What is the recovery, and does it need another full pass?
- A duplicate surfaces after the constraint is live and the period is already closed. Which table changes and which does not?
- How do you prove the key is deterministic enough that a client regenerating it after a crash produces the identical string?
How would you implement a rate limiter for an API?
Medium · system design and architecture
Approach
- Fix the scope first: who calls this, how often, and what they do when it fails.
- Name the failure you are designing for, then the recovery path.
- State the consistency you need, and where you are willing to be stale.
Follow-up
- How does this behave when that dependency is down for an hour?
- What would you drop to keep the system up under load?
How would you design a system to support whether an author can sell a book or not?
Medium · system design and architecture
Approach
- Name the read and write paths separately; they rarely have the same bottleneck.
- State the consistency you need, and where you are willing to be stale.
- Fix the scope first: who calls this, how often, and what they do when it fails.
Follow-up
- What breaks first when traffic grows ten times?
- What would you drop to keep the system up under load?
Turn at-least-once heartbeats into exactly-once qualified plays
Medium · idempotency · partitioning · backpressure · sessionisation
Heartbeats arrive at 100,000 per second carrying (session_id, seq, played_ms, position_ms, client_ts). Delivery is at-least-once and out of order, a crashed player resumes under the same session_id, and an offline client can deliver days of buffered events in one burst. qualified_play carries dedupe_key UNIQUE, status ENUM('pending','counted','reversed') and counted_in_period, and money is paid against it. Give the stream partition key, how watch time is accumulated, the sessionisation rule including sessions that never report an end, the dedupe key, the period-close policy, and the backpressure for bulk replay.
Approach
- Partition by session_id and say why the alternative loses. The dedupe grain is the session, so partitioning by session_id makes the seen-sequence state partition-local and gives per-session ordering from per-partition ordering alone; client-generated UUIDs hash evenly, so no partition goes hot. Partitioning by account_id colocates an account's sessions and buys nothing, because no rule in this pipeline spans sessions.
- Order and deduplicate on (session_id, seq), never on client_ts. Client clocks are skewed and untrusted, so a timestamp is a data field rather than an ordering key; the per-session sequence number is the only monotone thing the client produces. Keep the seen-sequence set per session with a TTL past the idle gap so the state does not grow with lifetime session count.
- Accumulate watch time as MAX(played_ms) over the session rather than a sum of position deltas. Seeking backwards makes deltas negative and seeking forwards makes them enormous, so delta accounting turns a viewer scrubbing a title into a negative or multi-hour watch. MAX is also idempotent under duplicate and out-of-order delivery, which is exactly the property at-least-once transport demands, and it is the reason the naive version passes every test written from a happy-path trace.
- Sessionise as gaps-and-islands with an explicit idle gap, and close the sessions nobody closed. Order by (session_id, seq), open a new island when the gap since the previous heartbeat exceeds three heartbeat intervals plus margin, and run a reaper that sets ended_at = last_heartbeat_at and end_reason = 'unknown' for sessions past the gap, because a client that simply stopped reporting will never send an end. The query is O(n log n) dominated by the sort unless the store already clusters by (session_id, ts).
- Derive dedupe_key deterministically from session_id, content_version_id and the qualification-threshold version, so a crashed player regenerating it collides on the unique index. Insert with ON CONFLICT DO NOTHING and never update in place; a correction is a reversing row so the period close stays reproducible from the table alone. Hold the period open for at least the maximum offline licence life, then close, and let anything later land in the next period with counted_in_period set to that close.
- Give bulk replay its own path. A days-long offline sync is not proportional to current concurrency and cannot be capacity-planned from it, so route it to a separate topic with its own consumer group and lower priority, keeping operational alerting on startup time and rebuffer ratio out from behind a replay. Rate-limit per device at ingest with 429 and Retry-After, and require a durable client-side queue with jittered retry, because these events are payable and dropping them loses a rights holder money.
Worked solution (40 min)
- Choose the partition key and state the ordering and dedupe guarantee it gives you, plus the one it does not.
- Write the gaps-and-islands query with the idle-gap constant, and add the reaper that closes never-ended sessions.
- Define dedupe_key's inputs and argue it survives client retry, a player crash and a bulk offline sync.
- Write the period-close rule and the shape of a reversing row for an arrival after close.
- Size the replay path: separate topic, per-device limit, and the consumer-lag signal you alert on.
Expected result: session_id as the partition key with dedupe state kept local and TTLed, ordering by sequence rather than timestamp, watch time as MAX(played_ms), a gaps-and-islands rule with an idle gap of three heartbeat intervals plus a reaper for sessions with no end, a deterministic dedupe_key inserted with ON CONFLICT DO NOTHING, a close held at least as long as the maximum offline licence with reversing-row corrections, and a separate lower-priority replay path with per-device rate limiting.
Verify it yourself
- Replaying an entire session's heartbeats a second time changes no qualified_play row and no watch-time total.
- A session containing backward seeks produces a watch time no greater than its wall-clock duration.
- An offline sync arriving after period close produces a row in the next period, and the earlier close still reproduces its original total.
- A crashed player resuming with the same session_id yields exactly one qualified_play for that content version.
Follow-up
- A device ships with a bug that resets seq to zero every 65,536 events. What breaks, and how do you repair the affected days?
- Finance needs a period closed on the third working day, but offline licences last 30 days. Reconcile those two requirements concretely.
- How would you detect that a client is inflating played_ms, and what would you do with the sessions you flag?
Session service leaks two gigabytes a day, restarts nightly
Medium · memory leak · retention · ttl eviction
The playback session service holds about 400,000 concurrent sessions per instance and is restarted nightly because RSS grows roughly 2 GB a day; instances OOM within three days if left alone. Old-generation pauses lengthen through the day while throughput is unaffected until the end. Sessions are created at start, renewed by 10-second heartbeats, and removed on an explicit stop; playback_session.ended_at is null for every session whose client simply stopped reporting. Give the ordered checklist and name the leak.
Approach
- Split the growth before choosing a tool: plot RSS minus heap used. A rising heap live set is a reachability leak; RSS rising against a flat heap sends you to off-heap buffers, native TLS state, thread stacks or a pool that never shrinks.
- Read GC logs next. In steady state the live set after each old collection should be flat. A monotonically rising post-collection floor is the leak signature; lengthening pauses with a flat floor is fragmentation or sizing, which is a different fix.
- Take two heap dumps hours apart and diff the dominator tree rather than the class histogram. The histogram names what grew; the dominator names what is holding it, which is the only actionable answer.
- Expect a map keyed by session id holding per-session state, plus a scheduled renewal task per entry. Cross-check its size against the count of sessions with status 'active' whose last_heartbeat_at is older than the lease TTL. A client that stops reporting never sends a stop, so a removal path conditioned on an explicit end never runs for it, and the entry count tracks exactly that population.
- Fix by evicting on the same TTL that governs the lease rather than on an explicit end: time-based expiry plus a sweeper, with the scheduled task cancelled on eviction, since an orphaned timer keeps its captured state alive by itself and will survive a fix that only clears the map.
- Restate the invariant so it does not regress: in-process session state must not outlive the lease it mirrors. The lease store already expires these entries; the in-process copy diverged from it.
Follow-up
- How do you bound the map so that exhaustion becomes a rejected session start instead of an OOM?
- Eviction now also removes sessions whose client is briefly offline - what does the viewer experience, and how does that interact with the lease TTL?
- Which metric would have caught this at 5% of heap rather than at OOM?
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
The estimate and the day you knew
A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.
Describe a time you went above and beyond at work.
Medium · behavioural and engineering judgement
Approach
- Give the blast radius: what could have broken, and what you measured.
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
Follow-up
- How did you know your change caused the improvement?
- What did you decide not to do, and why?
Describe a time you had a difficult colleague and how you handled it to find common ground.
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
Follow-up
- What did you decide not to do, and why?
- How did you know your change caused the improvement?
Argue against pre-issuing licences to survive a live spike
Hard · entitlement · drm · load spike · design review
A scheduled live event concentrates an audience into a 60-second window, driving a 10x licence-service spike with no ramp. A proposal on your team: pre-issue licences at manifest fetch and cache the entitlement decision for the event's duration. Every dashboard improves. Describe a design you argued against where the cost was invisible in metrics. State the property the proposal removed, the alternative you offered with its own cost, how you made the risk concrete, and what happened when the decision went the other way.
Approach
- The probe is whether you can name the property being deleted instead of arguing taste. State it exactly: manifests and segments carry no user identity and are cached so they can be shared, which leaves the licence request as the only per-user, uncacheable point on the path. Moving authorisation to manifest fetch moves it onto a cacheable document, and a cacheable document is a shared one.
- Make the consequence a scenario with a number. A viewer whose payment fails mid-event keeps playing until the licence policy expiry, which is fixed at issue and cannot be shortened; with a three-hour event and a three-hour policy, the exposure is the whole event per account, with nothing left to revoke against.
- Split the two changes the proposal bundles, because they are not equally bad. Caching the entitlement lookup for 60 seconds keeps a per-request check and bounds the error to 60 seconds of staleness. Pre-issuing the licence removes the check and hands out a key with a long policy life. Attack the second and concede the first - conceding it is usually what gets the rest of the argument heard.
- Price the alternative rather than only objecting. This is the one load spike in the system set by a clock rather than by demand, so it is the one you can capacity-plan: pre-warm from the event schedule, shard by key ID, and keep the policy expiry short enough that key rotation re-checks entitlement during the event. The costs are idle capacity bought before the event and a higher steady-state request rate from rotation.
- Make it reviewable instead of a debate. Write the exposure as one sentence with a figure in it, in a decision record with an owner and a date, so the team is choosing the risk rather than drifting into it.
- Say what you did after losing. A strong answer instruments the thing it argued against - compare entitlement_snapshot on issued grants against live subscription state and count the issued-but-unentitled - so the next conversation has evidence rather than a rerun.
Follow-up
- The event ships with pre-issued licences. What do you measure during it so the post-mortem is not opinion again?
- Specify the 60-second entitlement cache: what is the key, what is the staleness bound, and which decision must never read it?
- How short can the licence policy expiry go before key rotation becomes the new spike?