Interview guides · Alexander Thamm
Alexander Thamm Software Engineer Interview Questions & Guide 2026
Debugging practice for Alexander Thamm software engineer candidates: reproduce it, narrow it, then change one thing per run.
The 60-second brief
As a Software Engineer at Alexander Thamm, you are joining a team dedicated to driving innovation through data science, AI, and advanced analytics. This role is pivotal in transforming complex business challenges into scalable, robust software solutions that empower organizations to make data-driven decisions. You will work at the intersection of engineering excellence and strategic consulting, ensuring that the technology stack not only functions flawlessly but also delivers tangible business value.
Allocate preparation against your weakest link rather than your favourite topic. A strong algorithm habit usually comes with weak out-loud explanation of tradeoffs, and years of shipping usually come with rusty from-scratch implementation under a clock.
Focus areas
- Choose indexes from the query's access path
- Bound every outbound call with a timeout
- Make every write idempotent under retry
Overview
The work at Alexander Thamm is characterized by both technical depth and a client-facing mindset. You will contribute to the development of sophisticated platforms and custom applications that leverage machine learning and data pipelines. Because Alexander Thamm operates as a consultancy-driven data specialist, this role offers the unique opportunity to see the direct impact of your code on real-world industry problems, making it a highly influential position within the firm.
Where to focus your preparation
Treat the sections below as preparation areas and confirm the format with your recruiter.
Preparation focus
No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.
What to demonstrate
- Breadth across SQL, experimentation and product reasoning
- Ability to state assumptions before choosing a method
How to prepare
- Drill the practice exercises below and time yourself
- Prepare three quantified stories about decisions you drove
Where candidates lose points
Letting a slow dependency consume unbounded concurrency
The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.
Running a schema change as though the lock lasts as long as the statement
In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.
Assuming fixed-width integer arithmetic cannot overflow
In languages with fixed-width integers, including C, C++, Java, Go and Rust, computing a midpoint as (lo + hi) / 2 overflows once the sum passes the type's maximum, so write lo + (hi - lo) / 2 instead. Say which language you are in: arbitrary-precision integers, as in Python or Ruby, remove this specific hazard and none of the others.
Treating a network call as though it were a local function call
A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.
Questions & practice
Find overlapping job attempts and peak concurrency from lease records
Medium · sweep line · intervals · leases
A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.
Approach
- Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
- For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
- For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
- Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
- State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
- Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.
Follow-up
- A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
- Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
- How would you compute both answers incrementally as records arrive rather than in a daily batch?
Canonicalise a request body into a stable idempotency fingerprint
Medium · parsing · canonicalisation · hashing
idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.
Approach
- Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
- Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
- Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
- Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
- Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
- Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.
Follow-up
- A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
- Where does the fingerprint get computed relative to request decompression and the body-size limit?
- The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?
Identify the heaviest tenants in a five-minute window under memory pressure
Medium · top-k · heavy hitters · streaming
The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.
Approach
- Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
- Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
- State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
- Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
- Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
- Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.
Worked solution (25 min)
- Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
- Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
- Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
- Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.
Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.
Verify it yourself
- The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
- The heap result matches a full sort of the tenant sums on a sample.
- The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).
Follow-up
- The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
- Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
- You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.
Write the update path that detects a concurrent edit
Medium · optimistic concurrency · transactional outbox · audit history
resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.
Approach
- One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
- Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
- State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
- Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
- Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
- Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.
Worked solution (25 min)
- Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
- Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
- Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
- Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
- Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.
Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.
Verify it yourself
- COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
- Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
- Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.
Follow-up
- A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
- Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
- Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?
Explain why the owner filter ignores the listing index
Medium · composite index · query plans · sort elimination
The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.
Approach
- Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
- For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
- Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
- Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
- Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
- Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.
Follow-up
- 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
- A dashboard runs this for 40 owners in one page load. What changes about the design?
- How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?
Choose what to break when replication lag reaches forty seconds
Hard · replication lag · read-after-write · consistency vs availability · load shedding
Reads are served from two replicas: 14k requests/second, about 85% absorbed by cache, so roughly 2.1k reads/second reach the database. Writes go to the primary at 1.2k/second. A tenant's backfill drives replication lag from under 100 ms to 40 seconds and it is still climbing. Sessions that have just written are pinned to the primary. Decide, endpoint class by endpoint class, whether to serve stale, fail, or route to the primary, and justify each choice with the load it adds to the primary. Then state what you would have built beforehand.
Approach
- Establish blast radius before cause, because mitigation and diagnosis have different deadlines. The decisive arithmetic is what happens if the database reads move to the primary: 2.1k reads/second on top of 1.2k writes/second roughly triples its operation count, on the node already absorbing the backfill that caused this. Reads and writes are not equal in cost, so treat that as an argument against a blanket move rather than as a capacity model - but it is enough to rule out routing everything to the primary.
- Classify endpoints by what staleness costs, not by how important they feel. Reads whose staleness is invisible - listings, search, counters - stay on the replica and return the watermark so the client can tell. Reads that immediately follow that same session's write keep their primary pin, which is a small bounded slice of traffic rather than the whole 2.1k/second. Reads that feed a decision with a side effect - authorisation, quota, the read half of a read-modify-write - must not be stale at all, because a 40-second-old permission row is the stale-permission failure wearing a different costume; those go to the primary or fail.
- Shed instead of queueing. If the must-be-fresh class alone exceeds the primary's headroom, refuse its lowest-value slice with 503 and a retry-after. A request queued behind a saturated primary holds a connection for a client that has already given up, and the retry storm that follows is what turns degradation into an outage. Bound the connection pool per role so the read fallback cannot consume the write path's connections - that bulkhead is the single decision that determines whether writes survive the next ten minutes.
- Attack the cause in parallel, since it is the one thing that can be stopped. The backfill is the load generator. A backfill that reads replication lag as its throttle signal and pauses above a threshold would have made this a non-event, with batch sizes small enough that each batch's write volume is a fraction of what a replica can apply per second. That is most of the answer to what should have existed beforehand.
- Name the mechanism you would prefer over session pinning. Capture the write position at commit and require the read path to be at or past it: compare the primary's pg_current_wal_lsn() at commit time against the replica's pg_last_wal_replay_lsn(), and fall back to the primary only for the specific request that is ahead of the replica. Session pinning is the cheap approximation and it over-pins - every read in the window goes to the primary whether or not it needed to, which is a share of the cost being paid right now.
Worked solution (35 min)
- List the endpoints in three buckets - staleness invisible, staleness visible to the writer only, staleness unsafe - and attach the share of the 2.1k reads/second each bucket carries.
- Compute the primary's operation count under each routing option and mark which options are arithmetically available.
- Write the pin rule and its window, then the shed rule: which slice, what status code, what retry-after.
- Write the backfill's throttle predicate against a measured lag value, including its pause threshold and resume condition.
Expected result: Staleness-invisible reads stay on the replica and return a watermark. Reads following the session's own write keep a bounded primary pin. Reads that gate a side effect go to the primary or fail, with the lowest-value slice shed as 503 plus retry-after if headroom runs out. Connection pools are bounded per role so the read fallback cannot starve writes. The backfill is throttled on measured replication lag. The durable fix is a commit-position token compared against the replica's replay position, replacing blanket session pinning.
Verify it yourself
- The traffic routed to the primary under the plan is stated as a rate, not as a category.
- No element of the plan increases the write path's connection usage.
- The backfill throttle references a measured lag value and has an explicit resume condition.
- Every endpoint left on a replica returns something with which a client can detect staleness.
Follow-up
- Lag returns to normal in nine minutes. Which mitigation do you remove first, and which one stays permanently?
- A user reports their change did not save, and the write committed. Trace the path that produces that report and name the signal that would have shown it before the report arrived.
- The replica is 40 seconds behind but otherwise healthy. Do you take it out of rotation? What does that do to the other replica's lag?
Design the bulk write endpoint a migration script retries blindly
Medium · batch api · partial failure · idempotency
A customer's migration script pushes 2 million resources through POST /v1/resources:batch, up to 500 items per call, and retries any call that errors or times out. Within a call some items fail validation, some collide with rows that already exist, and some succeed. Specify the request and response shape, whether a batch is atomic or per-item, how idempotency works for the call and for each item, the status code for a mixed outcome, the size and item-count limits with their error codes, and exactly what the script does after a timeout mid-batch.
Approach
- Choose atomicity deliberately and price it. All-or-nothing means one transaction holding locks for the whole batch on a primary already absorbing about 1.2k writes/second, which bounds batch size by lock duration, and it turns one bad row into 499 rejections the script must resend. Per-item partial success is the right default for a migration, and the contract's job is then to make a partial outcome impossible to miss.
- Require a client-supplied id on every item and echo it in every result. Deriving a per-item key from the array index breaks the first time the script resends a batch with the failures removed: the indices shift, previously-succeeded items acquire new keys, and they are created a second time.
- Key the effects at two levels. The call's Idempotency-Key covers an exact resend of the same bytes; per-item keys of (tenant_id, client_item_id) make a partially-applied batch safe to resend whole. Resending an identical batch must reproduce the same per-item results, not 500 conflicts the script has to interpret.
- Answer a mixed outcome with one status plus per-item detail: 200, or 207 borrowed from WebDAV if you prefer it - document whichever you pick - carrying an array of client_item_id, per-item status, and either resource_id or an error code from the same taxonomy the single-item endpoint uses. Reserve 4xx for the request as a whole: unparseable body, too many items, payload over the limit (413). Put a failed count at the top level so that even a script checking only the cheapest thing cannot conclude success while rows were dropped.
- Cap the request before doing any of it: item count, total bytes, and concurrent batches per tenant, since one tenant's migration otherwise consumes write capacity everyone shares. Anything that cannot finish inside the request deadline belongs on job_run behind a 202, not in a synchronous call that will time out halfway.
- Script behaviour after a timeout: the outcome is unknown and no results were received, so resend the identical batch with the same keys and read the results. Never resend 'only the items I have no result for' - a timeout yields no results at all, and that rule silently means resend everything anyway.
Follow-up
- At 500 items per call, what is the wall-clock time for 2 million rows, and what pacing do you publish so the migration does not become an incident?
- One item in every batch fails with the same code. How does the script discover that without a human reading logs?
Edge instances grow 400 MB per hour until the nightly restart
Hard · memory leak · heap analysis · metric cardinality · gc
Edge API instances start at 700 MB resident and grow about 400 MB/hour; a nightly rolling restart has hidden it for weeks. Growth continues unchanged when request rate halves overnight, p99 degrades in the last hours before an instance is recycled, and heap used immediately after a forced full GC rises monotonically. The service holds no product state. Name the discriminating measurement that separates the plausible causes, give the most likely cause, and give the fix and how you would verify it.
Approach
- Separate resident memory from live heap first, because they fail differently. Resident size can grow from fragmentation, native buffers or thread stacks while the heap is flat; heap used after a full GC rising monotonically is the measurement that says objects are reachable and not being released. You already have it, so this is retention, not fragmentation, and that closes off half the candidate list.
- Use the rate's independence from traffic as the discriminator. Growth that continues at half the request rate rules out per-request objects that are merely slow to collect and points at a structure that grows with distinct values observed rather than with call volume. Write the candidates that have that property: a metrics registry keyed on a high-cardinality label, an unevicted cache, an interner, a per-key lock map.
- Take two heap snapshots an hour apart and diff by retained size, reading the dominator tree, not by allocation count or instance count. Expect one root holding a map with millions of entries, then follow the reference chain to the code that inserts and never removes. Allocation profilers point at churn, which is the wrong signal here.
- The candidate that fits this service is an observability label carrying an identifier, such as a request path recorded before templating so that /v1/resources/48213 becomes its own metric series. That grows with distinct ids seen, is independent of rate, and explains the late p99 degradation, since GC cost rises with the size of the live set.
- Fix by bounding cardinality at the source: template the path to /v1/resources/{id} before it becomes a label, move tenant id from a label to a log field or an exemplar, and cap the registry with a bounded map that evicts. Add a cardinality ceiling that fails loudly in a lower environment rather than growing quietly in production.
- Verify with a soak rather than a restart. Hold one instance out of the nightly recycle for 48 hours with the fix and compare post-GC heap and series count against an unfixed control taking the same traffic.
Follow-up
- Post-GC heap is now flat but resident size still creeps. What are you looking at, and does it matter?
- How would you have detected this before an OOM, given the nightly restart masked the trend?
- That label is what makes one dashboard useful. How do you keep the dashboard and lose the leak?
Suggested practice checklist
Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.
Day 1 — Diagnostic, scored before you study anything
- Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
- Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
- Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.
Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.
Day 2 — Largest gap: find the boundary rather than the subject
- Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
- Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
- Re-attempt one of them from blank four hours later with nothing open.
Deliverable: A sub-skill map with the two blocking sub-skills circled.
Day 3 — Drill the blocking sub-skill by repeating the shape
- Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
- State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
- Have someone else read your one-sentence rule and find the precondition you left out.
Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.
Day 4 — Second gap, plus maintenance on the strongest area
- Run the same sub-skill decomposition on the second-largest gap in half the time.
- Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
- Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.
Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.
Day 5 — The gap that is not a skill
- Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
- Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
- Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.
Deliverable: Two recordings with a counted reduction in time-to-first-question.
Day 6 — Retest under day-one conditions
- Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
- For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
- Write down which single block you would still lose the offer on.
Deliverable: A second scored rubric placed beside the first, with one named remaining risk.
Day 7 — Full loop under interview conditions
- Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
- Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
- Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.
Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.
'We rewrote it in a better framework' is not a story
A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.
What motivates you to work on data-centric software projects?
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Give the blast radius: what could have broken, and what you measured.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- What did you decide not to do, and why?
- What would you do differently if you ran that again?
How do you handle complex technical challenges in a team environment?
Medium · behavioural and engineering judgement
Approach
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
- Close with what you would do differently, concretely.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Tell me about your previous experience.
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Can you describe a time you had to explain a technical concept to a non-technical stakeholder?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Close with what you would do differently, concretely.
- State the situation in two sentences and spend the rest on the reasoning.
Follow-up
- What did you decide not to do, and why?
- How did you know your change caused the improvement?