cswork / libraryOpen practice workspace

Interview guides · AssetMark

AssetMark Software Engineer Interview Questions & Guide 2026

Timed coding practice for AssetMark software engineer candidates: full attempts under a clock, then scored against a rubric.

The 60-second brief

As a Software Engineer at AssetMark, you are at the intersection of sophisticated financial technology and scalable platform architecture. You play a critical role in maintaining and evolving the infrastructure that powers investment solutions for financial advisors and their clients. Your work directly impacts the reliability, performance, and security of the systems that manage significant capital, requiring a blend of technical precision and business-oriented problem-solving.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

  • Evolve APIs without breaking pinned SDK clients
  • Scope every query and cache key by tenant
  • Make every write idempotent under client retries

Overview

This role is highly collaborative, often bridging the gap between high-level architectural goals and the daily needs of operational teams. Whether you are working on cloud-native infrastructure, platform management, or core software development, you will contribute to a culture that values stability and innovation. You will be expected to navigate complex systems, troubleshoot effectively, and communicate technical trade-offs to stakeholders, making this an ideal environment for engineers who enjoy high-impact, mission-critical work.

The AssetMark interview process

AssetMark candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Deep-Dives

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

  • In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
  • In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
  • Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

  • Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
  • Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
  • For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Engagement with Team

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

  • Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
  • Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
  • Whether you can be redirected in the middle of an answer without restarting it from the beginning
  • Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

  • Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
  • Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
  • Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Where candidates lose points

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

One shared connection pool for every tenant and every query class

A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.

Issuing one query per row of a result set

Fetch related rows in a single batched query keyed by the ids you already hold, or join them into the original query. A per-row round trip multiplies network latency by the row count, and it looks perfectly fine against the ten rows in your development database.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Questions & practice

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

  1. Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
  2. Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
  3. Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
  4. Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
  5. Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
  6. Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Worked solution (35 min)

  1. Reproduce the shape locally: generate 2 million events for one synthetic tenant over 5 days, fold them, then inject three defects, namely 0.2% of events bucketed by ingested_at, a handful of duplicate idempotency_key values whose retries cross midnight, and a block of events ingested after the seal.
  2. Write the watermark-bounded query first and record how much of the gap it explains on its own.
  3. Bisect by day, then SKU, then hour, recording the probe count and the rows each probe touches.
  4. For each located cell, write down the predicted signature before querying it, then check whether the data matches the prediction.
  5. Time a full re-fold of the 2 million rows and extrapolate to 90 million.

Expected result: The watermark-bounded query accounts for the post-seal block entirely and removes it from the investigation. The ingested_at bucketing appears as adjacent hours off by equal and opposite amounts. The midnight-crossing duplicates appear as one (tenant_id, idempotency_key) present in two ingested_day partitions. Bisection reaches each cell in fewer than 70 probes.

Verify it yourself

  • The three injected defects sum to the original gap with no unexplained remainder.
  • Each probe's row count is bounded by one day-partition for one tenant, not by the period.
  • Removing the source_max_ingested_at bound from the first query makes a gap appear, and grow, while ingest runs, even after every defect is removed.

Follow-up

  • The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
  • How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
  • Ingest is still running while you investigate. What makes your two numbers comparable at all?

Order a job dependency graph and find its critical path

Medium · topological-sort · dag-longest-path · cycle-detection · critical-path

A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.

Approach

  1. Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
  2. Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
  3. Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order: earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E).
  4. Second pass in reverse topological order for latest_finish, then slack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain. slack[v] = 0 is exactly the statement that some longest path runs through v; equivalently, the longest path through v has length T - slack[v].
  5. The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by min(d, T - L_avoid(v)), where L_avoid(v) is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, while min(10, 15 - 14) predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan is max(T - d, L_avoid(v)).
  6. Compute L_avoid(v) the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, since L_avoid(v) only ever matters through that max: set duration[v] := 0, recompute the makespan as T0(v) = max(T - duration[v], L_avoid(v)), and the gain is min(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.

Follow-up

  • Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
  • Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
  • Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

  1. Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
  2. Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
  3. Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
  4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
  5. If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Follow-up

  • Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
  • The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
  • How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Rebuild an hourly rollup with deduplication and late-arrival accounting

Hard · window functions · deduplication · event time vs ingest time · watermarks

From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.

Approach

  1. Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed: row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs.
  2. Bucket on occurred_at and nothing else, and pin the timezone explicitly. date_trunc('hour', timestamptz) truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argument date_trunc('hour', occurred_at, 'UTC') on PostgreSQL 16 or later, or date_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC' before that. Filter environment = 'production' explicitly, since metering covers three environments and billing covers one.
  3. Aggregate to the grain with sum(quantity), count(*) and max(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include.
  4. Compute the late share inside the dedup-and-aggregate step as a conditional aggregate, sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows: sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back.
  5. Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
  6. Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.

Worked solution (40 min)

  1. Generate a day of events for one tenant across three workspaces and two SKUs, inject 2% duplicates sharing an idempotency_key with a later ingested_at, and inject 1% whose ingested_at is six hours past occurred_at.
  2. Write the dedup CTE alone and confirm its row count drops by exactly the injected duplicate count.
  3. Aggregate and compare quantity_sum against a ground truth computed outside SQL from the generator's own records.
  4. Add the two window and filtered aggregates, and confirm the final hour's running total equals the ungrouped day total for each (workspace, sku).
  5. Run the whole query twice over static data and diff the two outputs.
  6. Re-run with the session TimeZone set to a non-UTC zone and confirm hour_start does not move.

Expected result: One query: a dedup CTE using row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id); aggregation by (workspace_id, sku, hour_start) producing quantity_sum, event_count and max(ingested_at) plus a filtered late-quantity aggregate; a running total as a window over the aggregated rows; and an explicit statement that the partition-scoped daily unique index cannot catch a duplicate that crosses the ingest-day boundary or arrives in a later replay.

Verify it yourself

  • Row count after dedup equals raw count minus injected duplicates, exactly.
  • The running total at the final hour equals the ungrouped day total for that (workspace, sku).
  • Two consecutive runs over static data return byte-identical rows.
  • Every hour_start is hour-aligned in UTC and unchanged by the session's TimeZone setting.

Follow-up

  • A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
  • Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
  • Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

  1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
  2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
  3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
  4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
  5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

  • Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
  • Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
  • The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Explain the core concepts of Object-Oriented Programming (OOP).

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the requirement backwards to the design.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Can you walk through a scenario where you had to debug a complex issue in a production environment?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Say what you would check first and why it is the highest-information step.
  3. Work from the requirement backwards to the design.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What are the pros and cons of using specific design patterns in a large-scale system?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Metering ingest that survives a six-hour producer replay

Hard · idempotency · dedup horizon · partitioning · watermarks

metering-ingest consumes usage events at-least-once - 250M/day, 10-40k/second at peak - and folds them into usage_rollup_hourly keyed (tenant_id, workspace_id, sku, hour_start). usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). A producer outage ends in a six-hour replay that re-sends events already ingested, some of whose originals crossed midnight. Design the consumer: partitioning, where the acknowledgement sits relative to the commit, the deduplication horizon and its storage cost, and how the rollup watermark advances. Nothing may be double-counted and nothing may be silently dropped.

Approach

  1. Choose the acknowledgement position deliberately and name what each choice costs. Acknowledging after the fold commits makes the consumer at-least-once: a crash between the two replays the batch and produces duplicates, which are ordinary and absorbable. Acknowledging first makes it at-most-once: a crash between the two drops revenue with no error raised anywhere and no way to detect it later. Take at-least-once and design everything downstream to absorb duplicates.
  2. Put the dedup and the fold in one transaction so there is no window between them. Insert the batch into usage_event with ON CONFLICT DO NOTHING, take the rows actually inserted, and fold only those into usage_rollup_hourly with an upsert on (tenant_id, workspace_id, sku, hour_start) bucketed by occurred_at, not ingested_at. A batch of about 2,000 rows is one round trip and one index probe per event.
  3. Attack the partition-key flaw head on: the unique index includes ingested_day because a unique index on a partitioned table must contain the partition key, so the same (tenant_id, idempotency_key) re-sent after midnight is a different index entry and passes. Deduplicate instead against a store keyed (tenant_id, idempotency_key) with no date component, whose horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. At 14 days that is 250M x 14 = 3.5 billion keys, which is a dedicated key-value store, not a larger index on the same table. The alternative - partitioning usage_event on (tenant_id, occurred_day) so the natural key is stable - fixes dedup but loses pruning on ingest time and makes retention by dropping partitions awkward.
  4. Partition the consumer by hash of tenant_id so one tenant's replay stalls only its own partitions, and give replay traffic a separate lower-priority lane so live ingest keeps its latency. The cost is explicit: that tenant's watermark lags while the replay drains, and everything gated on the watermark waits for it.
  5. Define the watermark as a property of committed work, not of wall-clock time: per partition it is the largest occurred_at such that every event with a smaller occurred_at has committed, and the sealing decision uses the minimum across partitions. Record source_max_ingested_at on every rollup row so any number can prove what it did and did not include, and keep restatement legal only while status = 'open' - after sealed_at the value is frozen and a late event becomes an invoice adjustment instead.

Worked solution (40 min)

  1. Write the consumer loop in pseudocode with the acknowledgement after the commit, then annotate each line with what is lost or duplicated if the process dies exactly there.
  2. Size the dedup store: events/day x horizon_days keys, bytes per key including the tenant prefix, and the resulting memory or disk. Compare that cost against simply extending retention on the partitioned table and say why the latter does not fix the problem.
  3. Take one event ingested at 23:59:58 and replayed at 00:00:04 and work out its fate under (a) the partitioned unique index alone and (b) the separate dedup store.
  4. Write the per-partition watermark formula, then what the seal uses, then what a single stalled partition does to sealing.

Expected result: An at-least-once consumer acknowledging after the commit; deduplication on (tenant_id, idempotency_key) in a store whose horizon of at least 14 days exceeds max retry window plus supported replay length, sized at roughly 3.5 billion keys; tenant-hashed partitions with replay on a separate lane; a per-partition watermark with sealing gated on the minimum across partitions; source_max_ingested_at written on every rollup row.

Verify it yourself

  • The midnight-crossing retry must be rejected by whatever you chose. If the table's unique index is your only dedup, it is accepted and the invoice is wrong by exactly that event.
  • You should be able to name a distinct failure for acknowledge-before-commit and for acknowledge-after-commit. If both sound the same, the position was not actually chosen.
  • quantity_sum for any (tenant, workspace, sku, hour) must be reproducible by re-aggregating usage_event filtered to ingested_at <= source_max_ingested_at. If it is not reproducible, the rollup is an opinion rather than a projection.

Follow-up

  • The dedup store is lost entirely. What can you still guarantee, and how do you rebuild it from what remains?
  • A replay delivers events for an hour that is already sealed. Trace exactly what happens to them, row by row.
  • One partition is stuck on a poison message, so the minimum-across-partitions watermark never advances and no tenant can be sealed. What is your escape hatch and what does it cost in correctness?

Webhook workers leak until OOM and drop in-flight deliveries

Medium · memory leak · eviction · leases

webhook-delivery workers grow from 400 MB to a 2 GB limit over about 36 hours, are OOM-killed, restart, and repeat. Each restart abandons in-flight attempts, so webhook_delivery rows sit in in_flight until their leases expire and the backlog spikes. The live set measured after a forced full collection also grows. The fleet serves tens of thousands of subscriptions, several thousand of which have been failing for weeks. Give an ordered checklist, the measurement separating retention from fragmentation, and the fix.

Approach

  1. Separate the two failure shapes with one measurement: track resident set size against the live set after a forced full collection. A live set that climbs monotonically is retention; a flat live set under a rising RSS is fragmentation, off-heap or native allocation, or an allocator that never returns pages. The stated symptom puts this in the first category, which rules out allocator tuning as a fix.
  2. Characterise the curve rather than the total. Growth linear in uptime implies an unbounded structure keyed by something that keeps arriving; step growth implies buffering a large object. Correlate the slope against event rate and separately against the count of distinct subscriptions seen, because those two diverge and only one of them will fit.
  3. Diff two heap snapshots an hour apart by retained size grouped by dominant root, not by allocation count, which is dominated by short-lived objects and will point at the wrong thing.
  4. Expect a per-subscription map with no eviction: circuit-breaker or backoff state created on first failure and never removed, so the retained set grows with endpoints that have ever failed, and the several thousand permanently dead endpoints hold theirs forever.
  5. Fix in two places. Bound the in-memory structure with a size-capped LRU or a TTL keyed on last use, and move state that must survive a restart onto the subscription or webhook_delivery row, since the worker holding it in memory is exactly why a restart loses it.
  6. Repair the second-order damage separately, because it will outlive the leak: workers claim by compare-and-set with leased_until, so a bounded lease returns in_flight rows to pending on a known schedule, and a graceful shutdown releases leases instead of waiting them out.

Follow-up

  • The backlog spike after a restart is itself a thundering herd against customer endpoints. What stops the recovery from becoming a second incident?
  • Suppose the live set had been flat while RSS still climbed. Name two causes and the measurement that separates them.
  • How would you size the LRU, and what does a miss on an evicted circuit-breaker entry cost a customer whose endpoint is down?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Scope is read from what you decided

Team size, service count and tickets closed say very little. Seniority shows in the decision you owned: what you chose not to build, which constraint you traded away, whose objection you had to resolve before anything could move. A large project where you executed someone else's plan is a small story.

How do you approach learning new technologies or frameworks required for a project?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Name the disagreement and how you resolved it with evidence.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Can you describe a time you worked with a cross-functional team to deliver a project?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Why are you interested in working for AssetMark?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • How did you know your change caused the improvement?
  • What did you decide not to do, and why?