cswork / libraryOpen practice workspace

Interview guides · Abt Global

Abt Global Software Engineer Interview Questions & Guide 2026

Abt Global software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

As a Software Engineer at Abt Global, you are not just writing code; you are building technical solutions that support the company’s mission to improve the quality of life for people worldwide. Abt Global operates at the intersection of research, consulting, and international development, meaning your work directly influences high-stakes projects in public health, social policy, and economic growth.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

  • Scope every query and cache key by tenant
  • Make every write idempotent under client retries
  • Bound blast radius with per-tenant concurrency limits

Overview

In this role, you will often find yourself bridging the gap between complex data requirements and functional software applications. Whether you are working on internal tools that streamline research or public-facing platforms that track field conditions, your contributions directly impact how the organization executes its global development agenda. You will work in a mission-driven environment where technical precision is expected to meet the unique challenges of diverse, often remote, operational landscapes.

The Abt Global interview process

Abt Global candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

  • Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
  • Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
  • Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

  • Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
  • Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
  • Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Interviews

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

  • Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
  • Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
  • Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

  • Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
  • Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
  • Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Team Conversations

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

  • Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
  • Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
  • Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
  • Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

  • Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
  • Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
  • Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Questions & practice

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

  1. Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
  2. The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
  3. Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
  4. Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
  5. Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
  6. State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

  • A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
  • The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
  • What makes a re-run over the same day produce byte-identical rollups?

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

  1. Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
  2. due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
  3. complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
  4. Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
  5. Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
  6. All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Worked solution (30 min)

  1. Define the four structures explicitly: queues: subscription_id -> deque[delivery], ready: min-heap of (next_attempt_at, subscription_id), inflight: set[subscription_id], breaker: min-heap of (half_open_at, subscription_id).
  2. Write down the invariant you will assert after every operation: a subscription appears in at most one of ready, inflight and breaker, never in two.
  3. Implement due, complete and insert, then simulate 200,000 subscriptions with Zipf-distributed queue depths totalling 20 million deliveries.
  4. Add one endpoint that always times out after 10 seconds and one that always answers in 20 ms, then measure the fast endpoint's throughput with and without the per-endpoint breaker.
  5. Instrument heap size across the run.

Expected result: Heap size stays at or below 200,000 regardless of the 20 million deliveries. The fast endpoint's throughput is unaffected by the dead one once the breaker trips. The dead endpoint's deliveries accumulate in their own deque and cost exactly one heap entry.

Verify it yourself

  • The asserted invariant never fires under concurrent workers.
  • Replaying the same delivery set twice produces identical per-subscription ordering, even though the interleaving across subscriptions differs.
  • Replacing full jitter with fixed backoff makes retry attempts cluster visibly in a histogram of attempt timestamps.

Follow-up

  • One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
  • A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
  • The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

  1. Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
  2. Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
  3. Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
  4. Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
  5. Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
  6. Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

  • The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
  • A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
  • How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

  1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
  2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
  3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
  4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
  5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

  • Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
  • Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
  • The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Find the join that inflates every invoice total

Medium · join fan-out · aggregation grain · sql correctness

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

Approach

  1. Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
  2. Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
  3. Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
  4. Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
  5. Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
  6. Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Worked solution (25 min)

  1. Create one invoice with three lines of 1000, 1000 and 500 minor units and two payment attempts, then run the original query.
  2. Confirm it reports 5000 and 6 rather than 2500 and 2.
  3. Apply sum(distinct l.amount_minor) and confirm the total becomes 1500, which is worse rather than better.
  4. Write the two-CTE version with a LEFT JOIN and coalesce, and confirm 2500 and 2.
  5. Add a second invoice with lines and no attempts and confirm it still appears.

Expected result: The original sum equals the true line total multiplied by the number of payment attempts and the count equals attempts times lines; the corrected query pre-aggregates each branch to invoice grain in separate CTEs and LEFT JOINs them with coalesce, with voided lines excluded inside the line CTE and the tenant predicate applied to both branches.

Verify it yourself

  • The money total is unchanged by adding or removing payment attempts.
  • An invoice with zero attempts appears with its total and a zero count.
  • Two lines with identical amounts both contribute to the total.
  • Voiding a line changes the money figure and leaves the attempt count alone.

Follow-up

  • Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
  • Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
  • The same report is needed per tenant per month. What index makes the line CTE cheap?

Per-tenant rate limiting that holds across one hundred pods

Medium · rate limiting · hot keys · atomicity · write skew

The same gateway enforces each tenant's plan rate limit in requests/second and its monthly quota. Traffic is heavily skewed: a handful of tenants exceed all others combined, while a small tenant's requests may all land on one of the 120 pods. Both limits must hold in aggregate rather than per pod, per credential or per region, and the check may add at most 2 ms at p99. Specify the algorithm, where the state lives, the exact operation performed per request, and the behaviour when the counter store is unreachable - separately for the rate limit and the quota.

Approach

  1. Do the arithmetic on per-pod buckets before discarding them, and be precise about which half fails. Giving each pod rate/N keeps the sustained aggregate correct - 120 buckets of 5 rps sum to the 600 rps limit - but a tenant whose traffic all lands on one pod is throttled at 5 rps, a hundred and twentieth of what it pays for. The over-admission comes from burst depth, not sustained rate: if each pod carries a full bucket depth B, the aggregate burst is 120B against a limit that intended to allow B.
  2. Move the rate limit to a shared counter accessed in exactly one atomic round trip - a token bucket or GCRA as a single server-side script returning allow/deny plus a retry-after - never a read followed by a write. Keep the key tenant-scoped and region-local; a globally shared counter costs a cross-region round trip that alone exceeds the 2 ms budget, so shard the global limit per region in proportion to observed regional share and accept that a tenant whose traffic shifts regions is briefly throttled until the shards rebalance.
  3. Cut round trips for the largest tenants with leased tokens: a pod leases 50 at a time and spends them locally, reducing counter round trips by 50x at the cost of bounded over-admission of pods x lease_size, which is 6,000 requests in the worst case here. Shrink the lease for tenants with small limits, where that bound would swamp the limit itself.
  4. Treat the quota as a different problem, not a longer window. It is a count that other in-flight requests are changing, so select-the-usage-then-insert is write skew: read-committed permits it and repeatable read permits it too, because snapshot isolation is exactly what allows two transactions to read a consistent count and both write. Enforce it inside one statement - an atomic increment whose returned value is compared, or UPDATE ... SET used = used + $1 WHERE used + $1 <= limit RETURNING - or with a constraint that makes the surplus write fail.
  5. Write the unavailability policy per limit rather than one policy for both. The rate limit fails open to a conservative local bucket, because it protects capacity and failing closed turns a counter blip into a total outage. The quota fails open for tenants whose last known usage was well below the cap and closed for those at or above it, since that is the only decision with revenue attached, and reconciles from the usage ledger on recovery - which is why the monthly counter must be re-seedable from that ledger rather than being the only copy.

Follow-up

  • One tenant is 40% of all traffic and its counter key lands on a single shard. What do you change, and what does that cost in accuracy?
  • Clients receive 429s and retry. What stops the well-behaved ones from synchronising into a herd?
  • A tenant reports being throttled below its limit. Which evidence do you produce, and does your design emit it today?

Ship three breaking-looking changes without breaking pinned SDKs

Hard · versioning · backward compatibility · deprecation

GET /v1/runs/{id} returns status from the set queued, leased, running, succeeded, failed, timed_out, cancelled. You must add the terminal state lost, add an optional billable_seconds field, and start rejecting name values over 64 characters that were previously accepted. Clients are generated SDKs pinned inside customer build pipelines you cannot redeploy or reliably contact, some with strict deserialisers. Classify each change as compatible or breaking and justify it mechanically, then give the rollout: the versioning scheme, the transform old clients receive, the telemetry, and the sunset process.

Approach

  1. Classify against a deserialiser rather than against intuition. A new enum value breaks a generated client that validates the enum or switches exhaustively; a new optional field breaks one generated with additionalProperties false; tightening validation breaks every caller already sending an 80-character name even though no field moved. All three are breaking for some real client, and that is the answer, not a technicality.
  2. Choose the versioning axis on maintenance cost. A path version forks the handler, and each subsequent change multiplies the fork. A date-pinned version header, defaulted per credential at first use and pinned thereafter, keeps one internal model and a chain of ordered response transforms, so the handler stays single and the transforms compose.
  3. Write the downgrade honestly. Older versions have no lost, so the transform maps it to the least-wrong existing value and the changelog says which; mapping it to failed asserts an outcome nobody observed, so pair it with a reason field the old version already exposes rather than inventing a field, and keep the mapping in one documented table instead of a conditional inside the serialiser.
  4. Sequence the validation tightening separately from the payload changes: first measure how many tenants send names over 64 characters, then warn for a stated window with a Deprecation header and a non-fatal warning code, then reject only for versions dated after the change, never retroactively for a pinned client.
  5. Deprecate from evidence, not from announcement: per-version, per-tenant, per-credential request counts from the gateway, a Sunset header carrying the date, direct contact for the handful of accounts that make up most of the tail, and short scheduled brownouts before the cutoff so the clients nobody can reach fail on your calendar rather than during their release.

Worked solution (40 min)

  1. Write the three-row classification table with the client-side mechanism that breaks in each case, not just a verdict.
  2. Pick the versioning scheme and write the header, the default-and-pin rule for a new credential, and where the transform chain sits relative to the handler.
  3. Write the lost mapping for pre-change versions as an explicit table entry plus the changelog sentence a customer reads.
  4. Write the validation timeline with four dated steps, and mark which one is visible to clients that never change their version.
  5. Write the deprecation plan: the telemetry query by version and tenant, the headers, the brownout schedule, and the fallback for the largest holdout.

Expected result: All three changes classified breaking with the mechanism named, a date-pinned version header with one internal model and ordered downgrade transforms, a documented lost mapping, validation tightened only for versions after the change, and a sunset driven by per-version traffic with brownouts.

Verify it yourself

  • No change is called additive without naming the client behaviour that makes it safe.
  • A client that never changes its version header sees no behaviour change from any of the three.
  • The plan names a measurement that tells you when sunset is safe.
  • The lost mapping is stated once, in a table, not implied by serialiser branches.

Follow-up

  • Which step of your rollout cannot be rolled back cleanly once a client has depended on it?
  • A tenant pinned to a sunset version breaks their build the morning after the cutoff. What in your plan already lets you resolve it within an hour?
  • Your transform chain is now six versions deep. What stops it becoming the fork you were avoiding?

One tenant's counter writes stall the whole connection pool

Medium · lock contention · connection pool · multi-tenancy

A change that made a per-tenant usage counter correct now produces site-wide latency whenever one large tenant writes: unrelated endpoints time out waiting for a connection while database CPU stays low and no statement is slow. The change wraps the counter update in a transaction that takes SELECT ... FOR UPDATE on one row, calls an external pricing service, then updates and commits. Give an ordered checklist, the arithmetic that bounds that tenant's write rate, and three repairs with the cost each one accepts.

Approach

  1. Separate waiting from working. Low database CPU alongside high application latency points at a queue, so instrument connection-acquisition wait separately from query execution time; that queue forms in the application and is invisible in database metrics, which is why the database looks healthy throughout.
  2. Confirm the lock rather than assuming it: sample waiting sessions and group by wait event, relation and tuple. Contention concentrated on one tuple belonging to one tenant is the signature; a deadlock would instead show the database aborting transactions after its detection timeout, which is not happening here.
  3. Do the arithmetic out loud. Throughput on a serialised row is one divided by the lock hold time, and the hold spans the external call, so a 20 ms pricing call caps that tenant near 50 writes per second no matter how many pods run. Every waiter also holds a pooled connection while it queues, so the shared pool drains and unrelated tenants fail at acquisition.
  4. Repair one: shrink the critical section to a single statement with the price resolved before the transaction opens. Cost is a stale price for the duration of one request and a second round trip; benefit is a hold time measured in the database's own execution time.
  5. Repairs two and three change where the contention lives rather than how long it is held. Sharding the counter into per-(tenant, bucket) rows and summing on read multiplies write throughput by the shard count, at the cost of an aggregate on every read and a shard count you must size against the largest tenant rather than the median. Accumulating in memory and flushing periodically removes the per-write round trip entirely, paid for with a bounded loss window on crash, which is acceptable for a rate limiter and not for a billing counter.
  6. Contain independently of which repair wins: a separate pool or per-tenant concurrency cap for this write class, a statement timeout low enough that a pathological query dies before it accumulates waiters, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks.

Follow-up

  • What would a genuine deadlock look like here, which two code paths would produce one, and how does the database's response differ from what you observed?
  • If a transaction-pooling proxy sits in front of the database, which of your three repairs changes behaviour, and what stops working that would have worked on a direct connection?
  • The counter also enforces a quota. Why is SELECT the count and then INSERT still wrong after you have fixed the contention?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

  • Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
  • Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
  • Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

  • Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
  • Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
  • Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

  • Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
  • Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
  • Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

  • Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
  • Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
  • Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

  • Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
  • Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
  • Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

  • Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
  • Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
  • Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

  • Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
  • Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
  • Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Outage answers are graded on instruments

Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.

Why do you want to work for Abt Global?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

How is your opinion after seeing the videos of the current situation in the field?

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. Give the blast radius: what could have broken, and what you measured.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Tell me about what you did at your previous job.

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Can you describe your experience with the specific tech stack listed in your project history?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Name the disagreement and how you resolved it with evidence.
  3. Close with what you would do differently, concretely.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?