Interview guides · Advantage Solutions

Advantage Solutions Software Engineer Interview Questions & Guide 2026

Advantage Solutions software engineer prep: algorithm drills where every complexity claim has to name the assumption under it.

The 60-second brief

A Software Engineer at Advantage Solutions plays a pivotal role in bridging the gap between complex business requirements and scalable technical solutions. As a leader in sales and marketing solutions, Advantage Solutions relies on its engineering teams to build robust systems that support high-volume data processing, retail operations, and client-facing analytics. You will be responsible for designing, developing, and maintaining software that enables the company to operate efficiently across global markets.

Getting the code to run is the floor. What usually separates answers is the case checked without prompting: empty input, a single element, duplicate keys, or a value that overflows the integer type you chose.

Focus areas

Version an API you cannot force clients to upgrade
Bound credential lifetime by the engagement end date
Diagnose failures inside environments you cannot instrument

Overview

This role is inherently collaborative and dynamic. You will work closely with cross-functional teams, including product managers and operations experts, to translate business needs into clean, maintainable code. Whether you are building internal tools to optimize supply chain visibility or developing customer-facing platforms, your work directly impacts how Advantage Solutions delivers value to its clients. You can expect to operate in a high-stakes environment where technical precision and the ability to adapt to shifting priorities are highly valued.

The Advantage Solutions interview process

Advantage Solutions candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Conversation

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Deep Dive

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Experience Discussion

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Hands-On Assessment

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Code Discussion

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Forking the product per client instead of building a configuration or extension point

A fork is the cheapest possible answer for one client and the most expensive possible answer for the estate. Maintenance cost grows as O(clients x changes): a single security patch now needs one port per fork, each with its own conflicts, its own test run and its own release window, and the forks drift further apart with every port. The honest rule is that a required fork is a product gap, so it should either become a supported extension point with a stable interface, or be an explicit, time-boxed and separately priced artefact that is never expected to receive upstream fixes.

Long-lived shared credentials for client systems

One static key used by the connector runtime, a debugging script and three engineers cannot be attributed to a person in an audit, cannot be expired at engagement close, and cannot be rotated without breaking an unknown number of consumers at once. The expensive part of the eventual incident is not the leak; it is that rotation requires finding every holder under time pressure, and nothing recorded who they were. Issue per-principal, per-target, short-lived grants from a broker so rotation is a no-op, attribution is automatic, and closure is a TTL rather than a search.

Abandoning working code to chase the optimal solution

Get the straightforward version correct, state its complexity, and only then optimise, keeping the working version until the faster one passes the same cases. A correct quadratic solution with a stated path to linear beats a half-written optimal one that never ran.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Questions & practice

Find peak window utilisation and the first rejected request

Medium · sliding window · two pointers · rate limiting

One connector binding's requests against a client endpoint arrive as timestamps t[0..n-1] in milliseconds, sorted non-decreasing, n up to 1,000,000. The endpoint's budget is R requests per rolling window of W milliseconds, and a request at time t is admitted only if strictly fewer than R already-admitted requests fall in the half-open window (t-W, t]. Return (a) the maximum number of requests falling in any W-millisecond window, counting every request regardless of admission, and (b) the index of the first request the limiter would reject. Both parts in O(n) time.

Approach

Part (a) is a two-pointer scan: for each right index, advance left while t[right] - t[left] >= W, which leaves the window exactly (t[right]-W, t[right]], and track max(right-left+1). Left is monotone non-decreasing, so total work is O(n) amortised with O(1) extra space. The maximum over all windows of width W is attained at a window whose right edge is a request timestamp, so scanning right edges is sufficient.
Part (b) is not the same scan. A rejected request consumes no budget, so the window must count admitted requests only. Keep a FIFO deque of admitted timestamps: pop from the front while front <= t - W, admit if the deque holds fewer than R, and return the first index where it does not. O(n) total pushes and pops, O(R) memory.
Pin the boundary convention before writing either loop. With a half-open (t-W, t] window a request exactly W milliseconds after an earlier one does not see it; switching to a closed window changes the answer at every boundary and is where most disagreements with the client's own limiter come from.
Note what fixed buckets of width W buy and cost: O(1) memory, but they admit up to 2R inside a span shorter than W, R at the end of one bucket and R at the start of the next. If buckets are unavoidable, halve the bucket width or use a weighted estimate across two adjacent buckets.
If part (a) already exceeds R, the binding cannot run at this concurrency under any limiter shape. That is a scheduling problem, not a retry-policy problem, and no backoff tuning fixes it.

Follow-up

Three bindings and four workers share this one endpoint's quota. Where does the counter live, and what is the correct behaviour when that store is unreachable?
Add full-jitter backoff to the retry path and state precisely what it prevents that plain exponential backoff does not.
The client documents the limit as 'about 100 per minute'. How would you measure the real one without tripping it in their production?

Resolve effective-dated engagement versions for two million access checks

Hard · binary search · effective dating · complexity budget

The engagement table holds 500,000 effective-dated rows: (engagement_id, version, effective_from, effective_to which is NULL for the open version). A database exclusion constraint guarantees that per engagement_id the tstzrange(effective_from, effective_to) values never overlap, and the ranges are half-open. You must answer 2,000,000 access checks of the form (engagement_id, at_time) in one process with no database, returning the version open at that instant or none. Also report, per engagement, any coverage gap between adjacent versions. Give the complexity of the scan-per-check approach and say why it is not an option.

Approach

State the naive cost first: scanning the table per check is 2e6 x 5e5 = 1e12 range comparisons. Even at 1e9 simple comparisons per second that is roughly fifteen minutes of pure compare work, against a budget measured in seconds. It is correct, just unaffordable, and the fix is an index built once rather than a faster comparison.
Build a hash map engagement_id -> array of versions sorted by effective_from, once. O(V log V) to sort, O(V) memory. The build is amortised over 2e6 lookups, which is what makes it worth doing at all.
Answer each check by binary-searching for the last effective_from <= at_time, then confirming effective_to IS NULL OR at_time < effective_to. If the confirmation fails, the instant lies in a gap and the answer is none, not the neighbouring version. O(log v) per check, O(V log V + Q log v) total.
Name the precondition: predecessor search is only valid because the ranges per engagement do not overlap, which the exclusion constraint enforces with btree_gist for the equality operator. Without that guarantee you need an interval tree returning all stabbed intervals, at O(log n + k). Use the same half-open convention as the database, where at_time equal to effective_to belongs to the next version.
Find gaps with one O(V) pass over each sorted array comparing prev.effective_to against next.effective_from. The exclusion constraint forbids overlaps, not holes, so a hole is entirely possible and is where an access check fails closed for a reason nobody can see in the contract.
If the checks can be buffered, sort them by (engagement_id, at_time) and merge against the version arrays: O(Q log Q + V log V), usually a large constant-factor win over 2e6 random binary searches because the merge walks memory in order.

Follow-up

The open version has effective_to NULL. How do you keep both the sort and the comparison total without a sentinel that overflows your timestamp type?
A gap exists for one engagement and a check lands in it. What should the access path return, and what should it emit so someone fixes the amendment?
Amendments are being written while you answer checks. What read model keeps an answer self-consistent, and what does a check mean if the snapshot is a second stale?

Compute the minimal re-run set that preserves idempotency keys

Hard · interval complement · idempotency keys · watermarks

One connector binding must cover [span_start, span_end). You have all its connector_run rows: (idempotency_key, attempt_no, status, watermark_from, watermark_to), with half-open ranges, where idempotency_key is derived deterministically from the connector and the range, so re-running a range reproduces its key. Only status='succeeded' counts as covered. Produce the minimal re-run plan: every instant in the span ends up covered, any sub-interval some prior run already claimed is re-run at exactly that claim's range, and genuinely unclaimed sub-intervals become new claims cut at the binding's chunk size. Explain what merging ranges would cost.

Approach

Merge the succeeded ranges into a maximal disjoint union: sort by start, sweep extending the open segment while the next start is not beyond the current end. O(n log n) time, O(n) space, and this is the only status that contributes coverage.
Take the complement of that union inside [span_start, span_end) to get the uncovered gaps, in O(u) over the merged segments. Clip at both ends so the plan never reaches outside the span.
Sweep the non-succeeded claims (ambiguous, failed, abandoned) against the gap list with a two-pointer over both sorted lists. Any claim intersecting a gap is emitted at its original watermark_from and watermark_to, byte for byte, so the derived idempotency_key reproduces and the retry rejoins the same ledger row. O(n) after sorting.
Subtract the emitted claim ranges from the gaps; whatever survives was never claimed by any attempt and becomes fresh ranges cut at the chunk size. Only these get new keys. O(u + c).
Accept that a re-emitted claim may overlap time that is already succeeded. Range bookkeeping is not what makes the retry safe; the record-level dedupe key is, and re-reading a covered window costs one extra fetch against the client's quota, which is far cheaper than a key you can no longer reproduce.
Exclude anything still in status 'running' with a live lease. Re-claiming it races the in-flight attempt, doubles the quota spend against the client endpoint, and produces two ledger writers for one key. Total plan cost is O(n log n) time and O(n) space.

Worked solution (40 min)

Span [00:00, 06:00). Runs: K1 [00:00, 02:00) succeeded; K2 [02:00, 03:00) attempt 1 ambiguous and attempt 2 failed; K3 [03:00, 04:00) succeeded; K4 [04:00, 05:00) abandoned. Nothing claims [05:00, 06:00). Chunk size is one hour.
Merge succeeded ranges: [00:00, 02:00) and [03:00, 04:00). They do not touch, so the union has two segments.
Complement inside the span: gaps are [02:00, 03:00) and [04:00, 06:00).
Match claims to gaps: K2 intersects the first gap and is emitted at [02:00, 03:00); K4 intersects the second and is emitted at [04:00, 05:00). Both keep their original ranges and therefore their original keys.
Subtract the emitted claims from the gaps: [05:00, 06:00) remains, unclaimed by any attempt, and becomes a single new claim at the one-hour chunk size.
Compare against the merge shortcut: emitting the two gaps as [02:00, 03:00) and [04:00, 06:00) would give the second range a key that no prior attempt ever used, discarding K4's ledger history.

Expected result: A three-entry plan: retry K2 at [02:00, 03:00) with its existing key, retry K4 at [04:00, 05:00) with its existing key, and one new claim at [05:00, 06:00). Two of the three reuse a reproducible key; only the never-claimed hour gets a new one.

Verify it yourself

The union of the plan's ranges together with the succeeded union covers [00:00, 06:00) with no hole.
Every entry tagged as a retry matches an existing run's watermark_from and watermark_to exactly, so recomputing its idempotency_key yields the same string.
No emitted range is empty, and no emitted range extends outside the span.
Running the planner a second time against unchanged input produces the identical three entries, so the plan itself is idempotent.

Follow-up

An ambiguous run may have applied every record before the timeout. What do you read to settle it before re-running, and what is left if the client's schema offers no natural key?
Retention pruned the ledger rows for one of these keys. What does the re-run do now, and what bounds how long the ledger must be kept?
Two planners run concurrently against the same binding. What stops them emitting and executing the same plan twice?

Find watermark coverage gaps with a window function

Medium · window functions · gaps and islands · event tables

Each succeeded connector_run claims a half-open watermark range [watermark_from, watermark_to). Ranges for one connector can overlap, and a re-run can be nested entirely inside an earlier range. Write the query that returns, per connector_id, every interval not covered by any succeeded run, as (gap_from, gap_to), ordered by connector then time. State the frame clause you use and why the default one is wrong, and give the index that removes the sort.

Approach

Rule out lag(watermark_to) first and say why: ordered by start time, the immediately preceding row may end earlier than a row before it, so a nested or shorter range makes you report a gap that a wider earlier range already covers. The correct comparison is against the running maximum of every previous watermark_to, not the last one.
Compute max(watermark_to) OVER (PARTITION BY connector_id ORDER BY watermark_from, watermark_to ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). The ROWS frame is load-bearing: the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes the current row, so the running max always covers the current row's own end and the query reports no gaps ever.
Add watermark_to as a tiebreaker in the window ORDER BY so rows sharing a start instant are ordered deterministically under a ROWS frame, then emit a gap where the running max is non-null and watermark_from is strictly greater than it. The first row per connector has an empty frame and a NULL max; that is not a gap unless you define a floor such as the binding's creation time, which you should decide explicitly.
Filter status = 'succeeded' in the window's input, and argue the classification: an ambiguous run may or may not have applied its range, so counting it as covered can drop data silently. Gap detection should treat ambiguous as uncovered and leave resolution to the reconciliation path.
Offer the shorter alternative with its precondition: on PostgreSQL 14 and later, range_agg(tstzrange(watermark_from, watermark_to, '[)')) per connector builds a multirange, and the gaps are its complement against the window you care about. It handles adjacency and nesting without any frame subtlety, at the cost of requiring that version.
Index (connector_id, watermark_from, watermark_to) WHERE status = 'succeeded'. It supplies the window's required ordering directly, so the plan is a WindowAgg over an index scan with no Sort, and the work is O(n) over that connector's succeeded runs.

Follow-up

How do you distinguish a real gap from a period when the connector was deliberately paused?
The connector has run for two years and you only care about the last 30 days. What does adding that predicate do to the correctness of the running maximum?
How would you turn this query into an alert without it firing on the trailing edge, where the newest range is simply not written yet?

Soft-delete environments without breaking uniqueness or audit history

Easy · soft deletes · partial indexes · audit history

environment holds one row per provisioned deployment: environment_id, client_id, engagement_id, env_name, region, status, destroyed_at TIMESTAMPTZ NULL. Decommissioned environments stay readable for seven years, and env_name must be unique per client among live environments only, because a client may reuse a retired name. Design the constraint set and the history table, write the query that lists a client's live environments, and say what your design does about connector_run rows whose foreign key points at a destroyed environment.

Approach

Keep the tombstone as a nullable timestamp rather than a boolean: destroyed_at carries when as well as whether, and enforce live-only uniqueness with a partial index, CREATE UNIQUE INDEX env_live_name ON environment (client_id, env_name) WHERE destroyed_at IS NULL, which indexes only the live subset and so stays small.
Get the NULL semantics right before you pick a spelling. In a B-tree unique index two NULLs compare distinct by default, so plain UNIQUE (client_id, env_name, destroyed_at) enforces nothing at all among live rows — every live row is unique on the third column purely by virtue of being NULL. PostgreSQL 15's UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) does satisfy the requirement: the NULL live rows now compare equal and collide with each other, while retired rows carry distinct destroyed_at timestamps and stay distinct, so a name can be retired and reused any number of times.
Prefer the partial index anyway, and be able to say why rather than asserting it. It indexes only the live subset — tens of rows per client — instead of seven years of tombstones; it needs no version floor; and it keeps destroyed_at out of the key, so it cannot be defeated by two rows retired at the same instant. That last case is reachable: now() is transaction-start time, so retiring a name, recreating it and retiring it again inside one transaction writes two identical destroyed_at values and raises a duplicate-key error under the constraint form. clock_timestamp() avoids it if you take that route.
Put history in a separate append-only table written by an AFTER INSERT OR UPDATE OR DELETE trigger holding environment_id, changed_at, changed_by, op and the before and after row images. PostgreSQL has no system-versioned tables, so this is a trigger or a logical-decoding consumer, not a DDL flag, and the soft delete arrives at that trigger as an UPDATE rather than a DELETE.
State the foreign-key consequence rather than leaving it implicit: a soft delete leaves every referencing connector_run row resolvable, which is exactly what the audit requirement wants, but it means no read is protected by the database. Give the predicate one home, a view or an RLS-backed view that every caller uses, instead of repeating destroyed_at IS NULL across hundreds of call sites.
Size the retention: seven years of history plus tombstoned rows in the hot table. Partition the history table by month so expiry is a DETACH PARTITION rather than a DELETE that has to be vacuumed, and decide separately whether a residency-driven erasure request can be honoured against an append-only store at all.

Worked solution (20 min)

Write the DDL with destroyed_at TIMESTAMPTZ NULL plus the partial unique index on (client_id, env_name) WHERE destroyed_at IS NULL.
Insert two live rows sharing (client_id, env_name) and confirm the second is rejected; set destroyed_at on the first and confirm the insert now succeeds.
Build the same table a second time under UNIQUE NULLS NOT DISTINCT (client_id, env_name, destroyed_at) and confirm it passes the same two tests, so you are choosing between two working designs rather than rejecting a broken one.
Add environment_history (history_id, environment_id, changed_at, changed_by, op, row_before JSONB, row_after JSONB) and the AFTER trigger that populates it.
Write the live-listing query: SELECT environment_id, env_name, region, status FROM environment WHERE client_id = $1 AND destroyed_at IS NULL ORDER BY env_name.

Expected result: A partial unique index covering only live rows, a separate append-only history table with its own partitioned retention, and a listing query whose predicate matches the index predicate exactly so the planner can use it.

Verify it yourself

Run retire-and-recreate three times for the same name: three tombstoned rows and one live row coexist without violating the index.
On the NULLS NOT DISTINCT variant, retire and recreate the same name twice inside one transaction and watch it fail on duplicate destroyed_at, since now() does not advance within a transaction; the partial index has no such failure mode.
EXPLAIN the listing query and see the partial index; widen the predicate to destroyed_at IS NULL OR destroyed_at > now() and watch the index drop out, because that predicate no longer implies the index's.
Soft-delete a row and confirm the history trigger fired; a DELETE-only trigger records nothing, which is the common way this audit trail ends up empty.

Follow-up

What stops the next endpoint from forgetting the destroyed_at IS NULL predicate, and does that mechanism survive a developer writing raw SQL?
The history table stores row images as JSONB and the base table gains a column. What do old rows now mean, and how do you read them six years later?
A residency erasure request requires a genuine hard delete for one client. What breaks, and what do you keep?

Mint short-lived client credentials that expire with the engagement

Easy · credentials · ttl · audit · partitioning

The Access Broker mints credentials for client-owned systems. Its callers are connector jobs and operator sessions, roughly 400 mints a minute at peak against a 150 ms p99 budget. Every mint appends to credential_grant (grant_id, engagement_id, principal_id, target_system_id, scopes, issued_at, not_after, issued_by_request_id), and those rows stay queryable per engagement for a contracted retention period measured in years. Design the mint path and the audit store: how not_after is computed, what happens when the audit write fails, and how an auditor answers who held access to one client system last March.

Approach

Compute not_after server-side as min(requested_ttl, engagement.ends_on + access_grace_days - now) and refuse the mint when that is already non-positive. The requested TTL is a hint; the engagement window is a bound. Then subtract the verifier's clock-skew allowance: a validator that tolerates 60 seconds of skew accepts a token 60 seconds past not_after, so effective expiry is not_after plus that leeway and it has to land inside the grace window rather than just outside it.
Write the audit row in the same transaction that reserves grant_id, before the secret material leaves the process, and treat a failed audit write as a failed mint. A mint that failed is retryable; access that cannot be attributed to a named principal is permanent, and the asymmetry decides the ordering.
Make the mint idempotent on issued_by_request_id with a unique index, so a caller retrying after a timeout does not leave a second untracked credential alive. On 23505, read the row back. Then choose deliberately between a short replay window that returns the same material and a conflict that forces a fresh mint, and justify it: replay means the secret crosses the wire twice, conflict means a timeout costs the caller a new credential.
Range-partition credential_grant by issued_at monthly and index (engagement_id, issued_at DESC). The auditor's question then reads one or two partitions instead of the full history, and expiring a retention window becomes DETACH plus DROP - a metadata operation - rather than a delete of millions of rows that then has to be vacuumed.
Note that the cross-table rule cannot be a CHECK constraint, because a PostgreSQL CHECK cannot reference another table. Either denormalise the engagement's end into the grant row and CHECK against that local column, or enforce it in a trigger, and run a scheduled query that proves no live grant outlives its engagement.

Worked solution (20 min)

Implement the mint with not_after = min(requested, ends_on + grace - now) and enforce the same rule in the database with a trigger or a denormalised column.
Request a 24-hour TTL against an engagement ending in 90 minutes and assert the returned not_after.
Force a timeout on the first attempt, retry with the same issued_by_request_id, and count rows in credential_grant.
Create 24 monthly partitions, load 10 million grants, and time the per-engagement auditor query for one month with and without partitioning.

Expected result: The 24-hour request returns roughly 90 minutes minus the skew allowance, never the requested value, and the database rejects the row if the application computes it wrong. The retry yields exactly one row. The partitioned auditor query reads one partition and returns in milliseconds while the unpartitioned equivalent scans the whole table.

Verify it yourself

No row in credential_grant has not_after beyond its engagement's ends_on plus access_grace_days, asserted on a schedule rather than only at insert.
Every grant carries a non-null issued_by_request_id that appears in the calling service's request log.
Dropping a retention-expired partition touches no live rows and requires no vacuum of the parent.

Follow-up

The connector caches a credential for its full lifetime and the engagement is terminated ten minutes later. What is actually true about access, and for exactly how long?
An operator needs a week of access to a client system but the engagement ends in two days. What does the system do, and who is allowed to change that answer?
How would you prove to an auditor that no credential was ever minted against a closed engagement?

Design a run API where the outcome is genuinely unknown

Hard · idempotency · state-machines · ambiguity · at-least-once

connector_run holds run_id, connector_id, environment_id, idempotency_key, attempt_no, status, failure_class, watermark_from, watermark_to, records_read, records_applied, source_schema_version, started_at and finished_at; idempotency_key is derived from the connector and the watermark range, and (connector_id, idempotency_key, attempt_no) is unique. Workers claim a range, write to a client-owned target that offers no idempotency support of its own, then report back. A worker can die at any point, and a write can time out leaving the target's state unknown. Design the claim, report and settle endpoints and the status transitions a scheduler and an operator drive.

Approach

Derive the run's identity from the work rather than from the caller. POST /connectors/{id}/runs carrying a watermark range computes idempotency_key deterministically, so a scheduler retrying a claim receives the same logical run instead of a second one covering the same rows. Mint a new attempt_no only on an explicit retry of an attempt that has already settled as failed, which keeps the attempt history readable and stops a crashed worker from quietly opening a parallel attempt over the same range.
Give the claim a lease with an expiry rather than a lock. The worker heartbeats; an expired lease makes the run reclaimable. Without an expiry, a dead worker wedges the binding until someone notices. With a lease and no fencing, two workers can both believe they hold the range, so carry a monotonically increasing lease token and reject any settle whose token is stale — that is what stops a zombie's late report overwriting the live attempt's.
Make ambiguous a first-class status with its own transitions, separate from failed. A timeout, or a reset after the request bytes went out, leaves the target's state unknown: settling as failed and retrying duplicates, settling as succeeded and advancing the watermark loses data silently. The settle endpoint therefore accepts succeeded, failed with a class, or ambiguous with evidence, and only succeeded advances the watermark.
Keep the ledger write and the watermark advance in one local transaction, and have settle carry the dedupe keys actually applied. The target write and your commit cannot be made atomic without a distributed transaction no client will grant, so ambiguity is resolved by a reconciliation read: re-read the target by dedupe key over the claimed range and settle from what you find. Store the key's input fields and a derivation version alongside the hash, because the client will eventually rename or retype one of those fields and every previously computed key still has to be reproducible.
Define each caller's behaviour precisely. The scheduler re-claims expired leases and retries only definite failures whose class is retryable; it never retries an ambiguous run. The operator has an explicit resolve endpoint used after reconciliation, recording who decided and on what evidence. An ambiguous run that auto-retries is the design bug — it converts an unknown into a duplicate inside a system you do not own and cannot correct.
Close the loop with the counters. records_applied differs from records_read by the dedupe hit count, so a retry over a partially applied range should report a high hit count and little net change. That difference is the observable proving the dedupe key works: if a retry reports applied equal to read, the key is not matching and you are duplicating on the client's side right now.

Follow-up

The client's endpoint offers no read-back by your dedupe key. How do you settle an ambiguous run then?
A worker's lease expires while its write is still in flight and a second worker claims the range. What exactly prevents the double apply?
How long do you keep the dedupe ledger, and what happens when a record arrives after its ledger row was pruned?

Credential minting stalls estate-wide from one client's provider

Hard · partial outage · timeouts · bulkheads · queueing

At 14:05 the access broker's p99 rises from 28 ms to 9.4 s. Error rate is only 3 percent. Connector runs and operator sessions for every client begin failing, yet only one client's identity provider is degraded: it accepts TCP connections and never replies. The broker handles five mint calls per second, of which one targets that provider, through a shared outbound worker pool of 50. Handler service time, measured from the start of each call, is unchanged. Diagnose in order and give the changes that bound the blast radius.

Approach

Separate slow from queued before anything else, because they need opposite responses. Split the broker's latency into time waiting for an outbound slot and time executing the call. Unchanged service time alongside a 300-fold p99 means the whole regression is queueing, so the target of the investigation is a saturated resource, not slow code.
Split every metric by target_system_id next. One client's provider owns all the non-completing calls; everyone else's calls are fast once they start. Combined with step one this gives the shape immediately: one dependency's calls occupy the shared pool and every other client waits behind them.
Do the arithmetic and check it against the timeline. One call per second that never completes fills all 50 slots in 50 seconds, which matches an onset measured in under a minute rather than gradual decay. If the numbers had not fit, saturation would not be the whole explanation and you would keep looking.
Confirm at the socket, since a hang and a failure are indistinguishable in an error rate. Connections in ESTABLISHED with bytes sent and nothing received distinguish a silent peer from a refused connection or a reset. Note explicitly that TCP keepalive will not rescue you here: the Linux default sends its first probe after two hours, so only an application-level read timeout bounds this at all, and its absence is the root defect.
Fix by bounding, in this order: a connect timeout and a read timeout on every outbound call, both set below the caller's own latency budget; a per-target concurrency limit so a single provider can occupy at most a small share of the pool; a circuit breaker keyed by target_system_id that opens on consecutive timeouts and half-opens with one probe; and a retry budget with full jitter, because three retries against a hanging peer triple the occupancy rather than improving the odds.
Then reconsider the blast radius rather than only the incident. A fail-closed broker on every request path is a shared dependency whose outbound concurrency must be partitioned per client, and a refusal for one client's target must surface as a typed result the caller can classify, not as a timeout that has already consumed a slot to produce.

Follow-up

Choose the read timeout and justify it with evidence. What breaks for a provider that is legitimately slow, and how do you tell the two apart?
The breaker is open for that client. What does an in-flight connector run observe, and which failure_class should it record?
Could the broker have kept serving every other client with no code change at all? What would that configuration have cost you on an ordinary day?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

These questions focus on your background, communication style, and how…

Medium · behavioural and engineering judgement

These questions focus on your background, communication style, and how you handle professional interactions. Tell me about your previous work experience and the projects you are most proud of. How do you manage communication with stakeholders when project requirements change? Describe a situation where you had to explain a complex technical concept to a non-technical team member.

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Ship a connector on a deadline with named debt

Medium · delivery pressure · risk bounding · idempotency

Describe shipping something you knew was incomplete against a fixed external date: a client change window, a month-end, a contractual go-live. Name the specific shortcut, such as an ambiguous outcome classified as a plain failure, a watermark advanced outside the ledger transaction, or an unchunked backfill. State the bound you put around it, who accepted the risk and where that was recorded, what would have made you refuse to ship, and what actually happened to the debt afterwards. Do not describe debt you never wrote down.

Approach

Name the shortcut precisely enough that its failure mode follows from the description: no ambiguous state, so a timeout retried and duplicated; or the watermark committed in a different transaction from the ledger row, so a crash between them loses or replays a range.
Give the bound, which is what separates a judged risk from a gamble: a first run restricted to a reversible slice, a record cap, a kill switch, or a reconciliation read against the target scheduled before the next run.
Say who accepted the risk and in what artefact. 'The client knew' is credible only with a written record and a date.
State your refusal line and why this fell on the shipping side of it. Irreversibility is the usual test: damage you can undo ships, silent corruption inside a client's system does not.
Report what happened to the debt with a date, including the case where it is still live, and what it has cost since.

Follow-up

The shortcut fires in production. Walk through the first hour: what you look at, in what order, and what you tell the client.
Same deadline, but the shortcut risks writing duplicates into the client's system rather than into yours. Does your answer change?

Disagree in review about a dedupe key derivation

Medium · idempotency · code review · schema drift

A colleague's connector change computes its dedupe key as a hash over the whole source payload. You believe it must be derived from named, stable source fields with a stored derivation version, because the client renames and retypes fields without notice and every previously computed key must stay reproducible. Recount a code review disagreement of this shape that you actually had. Give the argument you made, the evidence that moved it, how long it stayed open, and what you did when the author was still unconvinced.

Approach

State the technical claim as a failure rather than a preference: hashing the whole payload means any upstream field addition changes every key, so the next run finds no ledger hit and re-applies records the client already has.
Say what evidence you produced. A replay of a real past payload change through both derivations, or a count of fields in the sample that have already changed shape, wins a review; seniority does not.
Describe the boundary you used between disagreeing and committing: reversible choices get one round and then the author decides, while a choice that corrupts persisted data is worth escalating. Say which this was and why.
Include the cost you accepted. A versioned derivation means storing the input field names and the derivation version alongside the hash, and keeping each old version's code path alive as long as its keys can still be looked up.
Close with what the review changed beyond the one diff: a test that pins the key across a renamed field, or a written rule about what may feed a dedupe key.

Follow-up

The client renames a field that feeds the key. Walk through what your derivation version does on the next run, and what happens to records already applied under the old key.
The author was senior to you and unconvinced, and the merge was needed that afternoon. What did you do?